1. 31
  1.  

  2. 13

    The dark not-so-secret of VC-funded startups is that they all have massive bills for cloud services like AWS and Heroku. I’ve once saw $500k per month pre-product. It’s insane.

    Part of the problem is that most CEOs and CTOs don’t know what these services are for. They’re great for legitimate startups (i.e. small companies that haven’t yet mastered demand prediction) and overflow capacity. They’re the only option when you have no idea what you’re building and what you’re going to need. They’re going to be a lot more expensive than bespoke hardware, though.

    The truth is, though, that often these costs don’t matter. If you’re the CEO, you have two options:

    • A: Hire 60 mediocre engineers ($15k/month each, fully loaded) run a $100k/month “Cloud” bill. Your total costs are $1M per month. What they build starts to break as soon as it’s taken beyond the specified level of capability.
    • B: Hire someone like me (who can recognize top talent) and let me build a team of 10 exceptional engineers. Give them lots of autonomy. Give them legit offices instead of open-plan bullpen space. You might be paying $200-250k/month fully-loaded for the team. Cloud bill is $10,000 per month. What we build exceeds the specified level of capability. So your costs at worst are $260k per month.

    Moving from A to B, you drop $650k on labor (salaries, office space, benefits) and $90k on AWS/Heroku/whatever. That seems like a major improvement. Even if we assume that these top-notch engineers won’t accept autonomy and real equity (not that 0.03 percent shit) in exchange for “low” salaries of $25k, and put their costs at $35k per month– although, in my experience, the best people are a lot less expensive than one might expect– they’re still a huge win. But this isn’t what CEOs choose. They prefer A over B, and I think that’s because B requires them to run a completely different kind of company, and one that’s too far out of their comfort zone. If you can’t get a legit 140 IQ to be your CTO (and, let’s be honest, we’re not the easiest people to manage) then you might see A as the easy/low-risk path. $100k/month in AWS bills is chump change in comparison to the perceived risk associated with the B-type company with its over-intelligent weirdos.

    The reality is that Silicon Valley implicitly expects that people focus on one thing. (Peter Thiel’s “one thing” management theory is an accurate statement of this.) Founders focus on raising the next round, and that’s it. Product managers focus on their products, and nothing else. Engineers are expected to work on the Jira tickets dropped on them each “sprint”, and not try to optimize things they weren’t asked to look at. So no one looks at these kinds of quality/performance/cost issues until things have gotten far out of control (at which point, the result is usually executive overreaction).

    1. 37

      Has anybody ever told you that you are a very modest person?

      1. 12

        I like reading MoC’s comments, but this was still funny. Michael, you probably want to work on that humblebrag a little :)

        Expanding a little on Michael’s comment: I suspect that VC’s see hiring a bunch of high-capability people as being inherently more risky than sticking a bunch of scut-work programmers in a room and throwing Jira tickets at them until a product falls out. AWS is the new “no one ever got fired for buying IBM” - it’s the safe (in career / PR terms) option.

        (Slightly wilder speculation follows.)

        Also of course, it’s perversely in VC’s interest to push start-up CEOs towards the higher expense options if that means they get to allocate more of the value of the company to themselves because it means startups need more capital, not less. In a world of unicorn breakouts, getting 10% more of the one unicorn in your portfolio massively outweighs the failures that die due to running out of cash because you pushed them towards the high-operating expenses business model.

        The last thing a VC needs is start-up CEOs realising that they can do without them & get to where they need to be with angel money alone.

        1. 2

          AWS is the new “no one ever got fired for buying IBM” - it’s the safe (in career / PR terms) option.

          AWS doesn’t necessarily correlate with hiring scut-work/Jira engineers. There are plenty of use cases where the AWS offerings are precisely what you should use– for example, if you’re starting out and have no idea what your load level is going to be, and can’t afford to hire full-time support people to manage hardware.

          I suspect that VC’s see hiring a bunch of high-capability people as being inherently more risky than sticking a bunch of scut-work programmers in a room and throwing Jira tickets at them until a product falls out.

          That’s part of it, but I also think that many of them are MBAs who don’t realize that the highly capable people exist. Or, more accurately, they assume that highly capable people will want to be businessmen like them, and that career software engineers are therefore mediocre by definition.

          The other factor is that most business people believe that if you hire 100 commodity workers, you’ll get 3-5 excellent ones, just based on the statistical draw. That’s actually true. There are plenty of very smart people with mediocre CVs who’ve just never had the opportunity to rise above commodity grunt work. You will get some good people, even with mediocre hiring processes. The problem is that they don’t want to stick around and carry the weight of all the bozos, so you’re going to find it hard to keep them.

          The old model of corporate management was that people joined a functional group (accounting, sales, engineering) and the best people (the 3-5 excellent people you were statistically likely to get, just by casting a 100-wide net) were supposed to be pulled into general management (i.e. executives) around age 40. Lawyers recognized that this was a raw deal (competing against other highly intelligent people) and hacked the system to their benefit: professionalization, enshrined in law. It failed for lawyers, and it fails for us. It doesn’t work in software engineering for a variety of reasons. One is that incompetent people do a huge amount of damage and are often negatively productive, so you’re forcing your top people to clean up messes. Another is that the people in general management are completely incapable of evaluating our work, which means politics plays a huge role. That’s uncorrelated with actual capability, so the 3-5 good ones recognize that the probability of them getting selected for advancement is… about 3-5 percent. They don’t like those odds, so they leave, and you get what Erik Dietrich calls “The Dead Sea Effect”. You’ve lost all your good people, so… congratulations, you’re now an Agile Scrum shop.

          Also of course, it’s perversely in VC’s interest to push start-up CEOs towards the higher expense options if that means they get to allocate more of the value of the company to themselves because it means startups need more capital, not less.

          It’s strange. On one hand, it’s true that VCs want to increase their financial control. On the other, you see a culture of co-funding that exists because the career incentive of the VC is not his portfolio but his reputation. This means that an arrangement that allows more people to “get in on” a round of funding is beneficial. You don’t make your career by building a strong portfolio that outperforms the market, because no one has a good sense of what “the market” is supposed to be for unregulated, beta-loaded private equities. You make your career by having been in on Google or Facebook.

          With either set of incentives, though, you do have a culture where VCs want to overfund the companies that they pick. Part of this, I think, is that the new game in town is to take advantage of emerging (and often transient) natural monopolies. That favors dumping a lot of money into each prospect, expecting it to move as fast as humanly possible, and not giving a shit about waste, product quality or even safety, or wrecked careers.

          The last thing a VC needs is start-up CEOs realising that they can do without them & get to where they need to be with angel money alone.

          There’s a lot of game theory involved. If you don’t take VC, and the VCs fund your competitors, then you can be screwed. VCs have a carrot that you can skip, but they also have a stick.

          In the short term, sales and marketing will make or break a business, and those are commodity functions that can be solved by throwing money and meat at a problem. This means that VC “rocket fuel” (or the lack thereof) can singularly determine which company succeeds and which fails. Now, technical excellence, culture, and all those things that we care about, all might matter in the long term… but if you get pushed out of existence by a VC-funded company with better marketing and distribution, then there is no long term.

        2. 5

          I don’t view honest self-perception as a sin. The world is full of terrible people who will take you down for any reason or no reason. You need internal strength to keep popping back up and that requires knowing exactly who you are and what you’re good at, and what you’re not. This is especially true of the corporate world, where the worst people tend to rise the fastest. It’s what most middle-class, hard working engineer types are never told about the corporate/VC career game: that it’s rigged against them, in favor of the worst that humanity has on offer, and if they don’t have a certain internal resilience about them, they’ll be crushed.

          I have no qualms about claiming to be better (morally and intellectually) than the people who run our crony-capitalistic system. I am better than they are. Why hide from what is true, just because it’s impolite to acknowledge the fact?

          Of course, there are plenty of things that I’m not great at. If I start claiming to be able to throw 100 mph fastballs, or that I’m a world-class musician, or that I’m in the same mathematical league as Ramanujan, then please call a doctor because I’ve clearly become delusional. And I’ve met plenty of people who are smarter than I am, and I’m humble enough to acknowledge that. What I’ve observed, in the corporate world, is that they don’t do well either. I’ve seen IMO gold medalists (who are smarter than I am and, therefore, also in the tribe of highly intelligent people) put on fucking PIPs at “Big 4” tech companies, and that’s completely unacceptable. (If he punches someone in the face or sexually harasses someone, then fire him because that’s never okay. If he’s missing “agile sprint deadlines”, well… the fact is that someone at that level of talent shouldn’t have “agile sprint deadlines” because someone at that level of talent ought to be working on a real project rather than “user story” bullshit.) That’s beyond a personal insult. It’s an act of tribal warfare, and I would like to fight back.

          If more people were like me and honest about what they were and what they were not, we’d have a better world.

          1. 6

            You’re not wrong, but it’s still the case that “Hey everyone, did you know my IQ is over 140?!” is r/iamverysmart grade self-aggrandisement & is pretty much guaranteed to rile any audience you care to name.

            1. 2

              Fair, especially given that there are high-IQ people who are functional idiots, and highly intelligent people who put up mediocre scores. (“Not a good test taker” is rare, but it does exist.) IQ, in this sense, tends to be a shorthand for a level of intelligence, but “good at taking tests” isn’t the intelligence that matters in the real world.

      2. 1

        Tangent:

        We’re in the process of moving from NSQ to Kafka

        Can you explain why? My company has a large NSQ footprint and we’re currently decommissioning some of our kafka nodes in favor of NSQ. I’m guessing you’re doing something clever with kafka partitioning strategies?

        1. 1

          I read through the DynamoDb problem and was surprised that they had to add special logging to figure out and disable hot keys like ‘user_id’. If they had setup structured logging, a quick python script over the log would have shown this issue.

          1. 4

            (I debugged this issue) This is roughly what we did. I guess it didn’t come across in the blog that way?

            1. 1

              From the blog

              So we dreamt up a simple hack to give us the data we needed: anytime we were throttled by DynamoDB, we logged the key

              Is there a reason you don’t want to log every single key?

              1. 1

                a) it’s sensitive data belonging to third parties, b) at the scale they are dealing with the logs would impose significant size and time penalties

                1. 1

                  Isn’t that unnecessary? You really only care about when there’s a problem, or else your logs will be noisy. If you log every time, it’ll be kind of hard to differentiate which keys are okay and which keys are a problem.