I was an engineer at Whole Foods Marker 20 years ago. Any time we’d talk about cool network/security stuff, our manager would say “yes, but will it sell groceries?”
I ask this often at my work or feel like I have to respond with “But, we billed customers for X, let’s focus on that.”. It is a hard thing, but it takes humility to acknowledge that the trade-offs of technology decisions are decided by the business, not the merits of the tech itself. But, this can also be demoralizing.
This is true even in a tech company and so, presumably, for employees working on the core business in other domains. Everything that you build comes with an opportunity cost. If I want developers to add a feature to some product then those same developers are not working on anything else for a certain period. What is the cost of that? This is particularly problematic in security because the return is non-linear. If Windows / Android / iOS doesn’t include a customer-visible feature that the other two have then that directly translates to lost sales. If they are vulnerable to a particular kind of exploit that the other two aren’t then that doesn’t affect sales until enough people both suffer from the exploit and understand that it is something avoidable.
I found this to be interesting reading and made me reflect on my latest career move where I went from being a line SRE to instead becoming a staff DevOps engineer. With the change, my area of expertise is wider (which I enjoy) and I’ve got much more say over the direction we are going in - a factor of that is also related to the company size.
I suppose it depends on the company, time, and luck, and “YMMV” as always. However, my experience working in staff roles was quite miserable, and many of my friends had the same experience.
Your manager may report to the COO (or the CEO in smaller companies), but it may not mean anything for either of you. If executives see you as a cost center that steals money from the real business, you will have to fight tooth and nail to keep your department funded. You may not even win: at quite a few places I’ve seen, such internal departments were staffed mainly by inexperienced people who would leave for a better job as soon as they could find one. But when disaster happens, you will be blamed for everything.
I’m pretty sure there are companies that don’t mistreat their staff IT personnel, but no assumption is universal.
IME: the harder it is for execs to see that “person/group X does their job which directly leads to profit” the more of an uphill battle it is. Even a single hop can have a big effect: note the salary differences between skilled sales people and skilled engineers.
Can confirm. This is particularly challenging for “developer experience” or “productivity” teams, where all of the work is definitionally only an indirect contribution to the bottom line—even if an incredibly important and valuable one.
Gotta be able to sell everything you do. It’s hard when metrics are immaterial but in those specific areas, you have to be showing “oh, I save business line X this many person-hours daily/weekly/etc.” constantly in order to advance
As an idea that sounds good, but in practice no one knows how to even estimate that in a lot of categories of important tech investment for teams like mine. I have spent a non-trivial amount of time with both academic and industrial literature on the subject, and… yeah, nobody blows how to measure or even guesstimate this stuff in a way that I could remotely sign my name to in good conscience.
note the salary differences between skilled sales people and skilled engineers.
The latter usually have a higher salary or total compensation so I’m not sure if I understood your point. Maybe sales make more in down-market areas of the industry that don’t pay more than $100k for programmers if they can help it?
$100k for programmers exists in the companies that have effectively scaled up their sales pipeline. Most programmers work on some kind of B2B software (like the example in the article, internal billing for an electricity company), where customers don’t number in the millions, engineer salaries have five digits, and trust me, their package can’t touch the compensation of the skilled sales person who manages the expectations of a few very important customers.
I think a good discriminant for this might be software-as-plumbing vs. software-is-the-product. I suspect SaaS has driven down the costs a lot of glue type stuff like this.
I’ve had exactly the opposite experience. Being in staff roles has been the most enjoyable because we could work on things that had longer term payoffs. When I’ve been a line engineer we weren’t allowed to discuss anything unless it would increase revenue that quarter. The staff roles paid slightly less but not too much less.
I had a similar experience. I worked on a devops team at a small startup, and we did such a good job that when covid hit and cuts needed to be made, our department was first on the chopping block. I landed on my feet just fine, finding a job that paid 75% more (and have since received a promotion and a couple of substantial raises), but I was surprised to learn that management may keep a floundering product/dev org over an excellent supporting department (even though our department could’ve transitioned to dev and done a much better job).
It’s interesting to compare this view with companies – in my experience, a scaling-up startup – that have a “frontend” team and a “backend” team. I was on the frontend team, but had friends on the backend team.
The frontend team was pure line – we took tickets and turned them into features. New checkout UI, higher conversions, more money – that was the theory (it was a feature factory no doubt).
The backend team was split – they’d sometimes do data modeling and backend API work (though often us FE people were expected to do that), but they also did a lot of devops, infrastructure work, etc – so sort of a split between staff and line.
This company was weird and sort of dysfunctional IMO, but I wonder how much that plays out across the industry. I’ve heard that backend salaries can often be lower than frontend or fullstack ones – usually the explanation being “it’s easier” which is total bullshit of course. But I wonder if execs see backend work as “more staff-y” than the user-facing frontend work, and pay based on that.
I guess my answer to that would be: it’s still running on a server somewhere!
Interesting! I have seen the exact opposite: front-end developers get paid way less, because they’re “just” doing HTML and CSS and JavaScript, and those are easy. The backend teams are doing the “hardcore” algorithms, data structures, scalability, performance, and stability work.
(FWIW I don’t believe in this view of the world, React and CSS are way harder for me to understand than, like, B-trees and kernel tuning. But I do believe there is a weak correlation: went to college ~= backend, went to boot camp/self-taught ~= frontend, and that might explain some of the pay discrepancy.)
When talking about “staff” in this article, I do not mean the Staff software engineer role that is found at tech companyies after senior. That is a different usage of the term.
Talk about a buried lede for the kind of places this gets discussed.
Indeed, I expected this to be about senior roles - but I’m sort of glad for the confusion because I don’t think I’d have clicked otherwise. Turns out I found this topic more interesting. While not a software developer / engineer, it turns out I’ve hopped between “line” and “staff” a number of times and it’s helping to clarify some of the things I’ve been thinking about in my latest career progression.
Yes, the delineation is correct with some nuances.
A developer can be a ‘Cost center’ or a ‘Profit center’.
If a developer works for ‘Internal It’ – that means they are a ‘Cost center’.
If they work for a company that sells software, or software services – they are a ‘Profit Center’.
There are companies (usually large investment banks, online retailers, healthcare conglomerates, etc) that would claim that they are ‘just like a technology company’ – implying that software developers in there are ‘profit centers’… Because it is the technology that differentiates them from their competitors.
In investment banks, if a person working close to a ‘profit center’ – eg the trade desk/trader, then they will receive benefits of being a ‘profit center’.
But in the same org, working somewhere in regulatory/compliance, or middle or back office (that is much further from the trade desk) – another developer, may be with even higher skills – will be a ‘cost center’.
In healthcare (but with much lower bonuses) something similar happens.
Cost-center engineers are treated differently from salary, promotion perspective.
Profit-center engineers will make significantly more, but also, they have to switch companies/places to stay competitive, and, often, they ‘trail’ their business bosses and move with them…
Tried both, my experience was that staff IT was treated as “electricity bill” and was kept as low as it could be, being a cost center. Not so motivating nor inspirating environment to work in. Having deep understanding in a specific business domain also didn’t helped relying on that experience in further roles.
I can relate to a lot of this. I’ve started selling myself as a staff engineer for small to medium companies. I work primarily on internal tools that make the business more efficient. People hire me as a force-multiplier, not an incremental increase to their staff count.
The biggest thing I lean on is having previously been involved in the start to finish operations of an annual convention for several years. I have a basic understanding of almost every aspect of business. And I know how every annual convention is a 14-month-long process. (You start planning the next year 2 months before the current year is finished.)
I’m a bit surprised by the conclusion at the end that staff roles are under-discussed, though I agree when I give it some thought. When I was a student, I was taught that staff roles accounted for around 90% of software engineering jobs. Back in the ’90s, this was also a big part of the argument for open source: 90% of software is written in-house and is intrinsically F/OSS (the user of the software is the company that owns the copyright, whether they share it or not) and, since the software is not part of their core competency, releasing the code and collaborating with competitors is a sound economic decision.
I have no idea whether the 90% statistic is still correct and it’s something that I’ve heard far less in the last 15 years than before.
I think the general push to SaaS and ISVs/integrators as software providers has decreased the size and criticality of in-house dev teams for companies that don’t explicitly sell software (hosted or otherwise). Current management wisdom is very much, “build the thing that’s your core competency or competitive advantage, and buy everything else.”
By Adam’s framing, that makes a larger percentage of programmers “line” workers than before, because (say) the HRIS system you’re working on is the product you sell, not overhead that you have to build b/c nothing off-the-shelf will work. Likewise “professional services” teams, analytics and marketing devs inside SaaS vendors, etc.
The move to cloud providers and away from self-operated datacenters has just accelerated this, since systems administration and operations are being provided as a paid service by outside vendors, who in turn treat that as “line” work.
I wonder the extent to which that’s true. A lot more of in-house development is outsourced but the scale of the development has also increased by orders of magnitude. 30 years ago, a typical (non-tech) small-to-medium business might use a database for payroll and inventory control and a word processor for the secretary to type up letters, but certainly not a computer for every employee. They might have an in-house programmer / sysadmin (often the same person, long before DevOps was a buzzword) to do custom things like write a VB GUI for the database. They might outsource this to a contracting firm because there wasn’t enough work to justify a full-time employee (a lot of companies in the late ’90s realised that they had outsourced writing some critical bit of business software in VB4 to a company that had subsequently gone bankrupt and no there was no one who could understand it enough update it, even if they had the rights to the source code).
Today, that same company probably have completely computerised inventory, billing, and payroll systems, possibly with some direct computer control of certain phases of production. They will likely have a web-based storefront and associated infrastructure. They might be buying a load of it off the shelf from companies like Shopify, but the amount of customisation that they need that’s specific to their business is vastly more complicated than it was for the company 30 years ago and they’re likely to need an in-house programmer. Whether they have one or not is an open question.
I was an engineer at Whole Foods Marker 20 years ago. Any time we’d talk about cool network/security stuff, our manager would say “yes, but will it sell groceries?”
And sometimes the answer was no.
I ask this often at my work or feel like I have to respond with “But, we billed customers for X, let’s focus on that.”. It is a hard thing, but it takes humility to acknowledge that the trade-offs of technology decisions are decided by the business, not the merits of the tech itself. But, this can also be demoralizing.
This is true even in a tech company and so, presumably, for employees working on the core business in other domains. Everything that you build comes with an opportunity cost. If I want developers to add a feature to some product then those same developers are not working on anything else for a certain period. What is the cost of that? This is particularly problematic in security because the return is non-linear. If Windows / Android / iOS doesn’t include a customer-visible feature that the other two have then that directly translates to lost sales. If they are vulnerable to a particular kind of exploit that the other two aren’t then that doesn’t affect sales until enough people both suffer from the exploit and understand that it is something avoidable.
I found this to be interesting reading and made me reflect on my latest career move where I went from being a line SRE to instead becoming a staff DevOps engineer. With the change, my area of expertise is wider (which I enjoy) and I’ve got much more say over the direction we are going in - a factor of that is also related to the company size.
I suppose it depends on the company, time, and luck, and “YMMV” as always. However, my experience working in staff roles was quite miserable, and many of my friends had the same experience.
Your manager may report to the COO (or the CEO in smaller companies), but it may not mean anything for either of you. If executives see you as a cost center that steals money from the real business, you will have to fight tooth and nail to keep your department funded. You may not even win: at quite a few places I’ve seen, such internal departments were staffed mainly by inexperienced people who would leave for a better job as soon as they could find one. But when disaster happens, you will be blamed for everything.
I’m pretty sure there are companies that don’t mistreat their staff IT personnel, but no assumption is universal.
IME: the harder it is for execs to see that “person/group X does their job which directly leads to profit” the more of an uphill battle it is. Even a single hop can have a big effect: note the salary differences between skilled sales people and skilled engineers.
Can confirm. This is particularly challenging for “developer experience” or “productivity” teams, where all of the work is definitionally only an indirect contribution to the bottom line—even if an incredibly important and valuable one.
Gotta be able to sell everything you do. It’s hard when metrics are immaterial but in those specific areas, you have to be showing “oh, I save business line X this many person-hours daily/weekly/etc.” constantly in order to advance
As an idea that sounds good, but in practice no one knows how to even estimate that in a lot of categories of important tech investment for teams like mine. I have spent a non-trivial amount of time with both academic and industrial literature on the subject, and… yeah, nobody blows how to measure or even guesstimate this stuff in a way that I could remotely sign my name to in good conscience.
The latter usually have a higher salary or total compensation so I’m not sure if I understood your point. Maybe sales make more in down-market areas of the industry that don’t pay more than $100k for programmers if they can help it?
$100k for programmers exists in the companies that have effectively scaled up their sales pipeline. Most programmers work on some kind of B2B software (like the example in the article, internal billing for an electricity company), where customers don’t number in the millions, engineer salaries have five digits, and trust me, their package can’t touch the compensation of the skilled sales person who manages the expectations of a few very important customers.
I can confirm that I have never worked for companies where the sales people were paid less than the engineers. At least not to my knowledge.
In fact, in most companies I worked for, the person I reported to had a sales role.
I think a good discriminant for this might be software-as-plumbing vs. software-is-the-product. I suspect SaaS has driven down the costs a lot of glue type stuff like this.
I’ve had exactly the opposite experience. Being in staff roles has been the most enjoyable because we could work on things that had longer term payoffs. When I’ve been a line engineer we weren’t allowed to discuss anything unless it would increase revenue that quarter. The staff roles paid slightly less but not too much less.
I had a similar experience. I worked on a devops team at a small startup, and we did such a good job that when covid hit and cuts needed to be made, our department was first on the chopping block. I landed on my feet just fine, finding a job that paid 75% more (and have since received a promotion and a couple of substantial raises), but I was surprised to learn that management may keep a floundering product/dev org over an excellent supporting department (even though our department could’ve transitioned to dev and done a much better job).
It’s interesting to compare this view with companies – in my experience, a scaling-up startup – that have a “frontend” team and a “backend” team. I was on the frontend team, but had friends on the backend team.
The frontend team was pure line – we took tickets and turned them into features. New checkout UI, higher conversions, more money – that was the theory (it was a feature factory no doubt).
The backend team was split – they’d sometimes do data modeling and backend API work (though often us FE people were expected to do that), but they also did a lot of devops, infrastructure work, etc – so sort of a split between staff and line.
This company was weird and sort of dysfunctional IMO, but I wonder how much that plays out across the industry. I’ve heard that backend salaries can often be lower than frontend or fullstack ones – usually the explanation being “it’s easier” which is total bullshit of course. But I wonder if execs see backend work as “more staff-y” than the user-facing frontend work, and pay based on that.
I guess my answer to that would be: it’s still running on a server somewhere!
Interesting! I have seen the exact opposite: front-end developers get paid way less, because they’re “just” doing HTML and CSS and JavaScript, and those are easy. The backend teams are doing the “hardcore” algorithms, data structures, scalability, performance, and stability work.
(FWIW I don’t believe in this view of the world, React and CSS are way harder for me to understand than, like, B-trees and kernel tuning. But I do believe there is a weak correlation: went to college ~= backend, went to boot camp/self-taught ~= frontend, and that might explain some of the pay discrepancy.)
Talk about a buried lede for the kind of places this gets discussed.
Indeed, I expected this to be about senior roles - but I’m sort of glad for the confusion because I don’t think I’d have clicked otherwise. Turns out I found this topic more interesting. While not a software developer / engineer, it turns out I’ve hopped between “line” and “staff” a number of times and it’s helping to clarify some of the things I’ve been thinking about in my latest career progression.
Yes, the delineation is correct with some nuances.
A developer can be a ‘Cost center’ or a ‘Profit center’.
If a developer works for ‘Internal It’ – that means they are a ‘Cost center’. If they work for a company that sells software, or software services – they are a ‘Profit Center’.
There are companies (usually large investment banks, online retailers, healthcare conglomerates, etc) that would claim that they are ‘just like a technology company’ – implying that software developers in there are ‘profit centers’… Because it is the technology that differentiates them from their competitors.
In investment banks, if a person working close to a ‘profit center’ – eg the trade desk/trader, then they will receive benefits of being a ‘profit center’.
But in the same org, working somewhere in regulatory/compliance, or middle or back office (that is much further from the trade desk) – another developer, may be with even higher skills – will be a ‘cost center’.
In healthcare (but with much lower bonuses) something similar happens.
Cost-center engineers are treated differently from salary, promotion perspective.
Profit-center engineers will make significantly more, but also, they have to switch companies/places to stay competitive, and, often, they ‘trail’ their business bosses and move with them…
I have been in line jobs for 40 years and never knew about this.
Tried both, my experience was that staff IT was treated as “electricity bill” and was kept as low as it could be, being a cost center. Not so motivating nor inspirating environment to work in. Having deep understanding in a specific business domain also didn’t helped relying on that experience in further roles.
I have always found admin overhead departments to be painful to work in. no one really cares.
I can relate to a lot of this. I’ve started selling myself as a staff engineer for small to medium companies. I work primarily on internal tools that make the business more efficient. People hire me as a force-multiplier, not an incremental increase to their staff count.
The biggest thing I lean on is having previously been involved in the start to finish operations of an annual convention for several years. I have a basic understanding of almost every aspect of business. And I know how every annual convention is a 14-month-long process. (You start planning the next year 2 months before the current year is finished.)
I’m a bit surprised by the conclusion at the end that staff roles are under-discussed, though I agree when I give it some thought. When I was a student, I was taught that staff roles accounted for around 90% of software engineering jobs. Back in the ’90s, this was also a big part of the argument for open source: 90% of software is written in-house and is intrinsically F/OSS (the user of the software is the company that owns the copyright, whether they share it or not) and, since the software is not part of their core competency, releasing the code and collaborating with competitors is a sound economic decision.
I have no idea whether the 90% statistic is still correct and it’s something that I’ve heard far less in the last 15 years than before.
I think the general push to SaaS and ISVs/integrators as software providers has decreased the size and criticality of in-house dev teams for companies that don’t explicitly sell software (hosted or otherwise). Current management wisdom is very much, “build the thing that’s your core competency or competitive advantage, and buy everything else.”
By Adam’s framing, that makes a larger percentage of programmers “line” workers than before, because (say) the HRIS system you’re working on is the product you sell, not overhead that you have to build b/c nothing off-the-shelf will work. Likewise “professional services” teams, analytics and marketing devs inside SaaS vendors, etc.
The move to cloud providers and away from self-operated datacenters has just accelerated this, since systems administration and operations are being provided as a paid service by outside vendors, who in turn treat that as “line” work.
I wonder the extent to which that’s true. A lot more of in-house development is outsourced but the scale of the development has also increased by orders of magnitude. 30 years ago, a typical (non-tech) small-to-medium business might use a database for payroll and inventory control and a word processor for the secretary to type up letters, but certainly not a computer for every employee. They might have an in-house programmer / sysadmin (often the same person, long before DevOps was a buzzword) to do custom things like write a VB GUI for the database. They might outsource this to a contracting firm because there wasn’t enough work to justify a full-time employee (a lot of companies in the late ’90s realised that they had outsourced writing some critical bit of business software in VB4 to a company that had subsequently gone bankrupt and no there was no one who could understand it enough update it, even if they had the rights to the source code).
Today, that same company probably have completely computerised inventory, billing, and payroll systems, possibly with some direct computer control of certain phases of production. They will likely have a web-based storefront and associated infrastructure. They might be buying a load of it off the shelf from companies like Shopify, but the amount of customisation that they need that’s specific to their business is vastly more complicated than it was for the company 30 years ago and they’re likely to need an in-house programmer. Whether they have one or not is an open question.