Threads for o5r


      First weekend (of 12) of my coaching class. Where I’ll learn to coach better, I think. Learn a bunch about psychology, one hopes.

    2. 0

      Esolangs that take the path of least resistance are so tired. An uninteresting, one-note joke with a docs page. Stack based, uses the most obvious elements as instructions, uses js for metaprogramming instead of using the structure of html itself to do interesting things.

      Requiring valid HTML as programs would be way more interesting, and make it a fun polyglot language. It would be interesting as well if HTML programming language was able to generate HTML, or able to self-modify even, allowing you to programmatically create webpages with html syntax.

      1. 2

        So like xslt?

      2. 2

        ikr, this looks like it was made as a joke by some idiot over the weekend as they tried to avoid folding laundry or something

        some good news though!

        HTML, the programming language does support self modification, in that you can call methods to modify the DOM of a program and then jump to the modified code via the <a> tag.

        some additional good news as well: the language is entirely pluggable so you can redesign the entire thing by mutating the html.meta.commands object, making an HTML5 compatible language, redoing semantics, whatever you’d like!

        and, in fact, HTML: the programming language, supports dynamic language modification in that, in HTML: the programming language, a program can actually modify the language itself during execution, since it can mutate html.meta.commands as well.

        many possibilities, but the language is still terrible tho, probably designed by an idiot

        1. 1

          certainly overlooked the self modification portion. <samp> does some of this I assume. Certainly can burn an insane amount of time making html.meta.commands accept valid html

        2. 1

          by “entirely pluggable” what do you mean? can i modify html.meta.commands from within HTML? I don’t need any way to access named objects.

          1. 2

            you can access named objects via the <cite> command and then properties with the <rp> command.

            here is an example of an HTML, the programming language, program that adds a new command to HTML, the programming language, and then executes that command:


            we add a new command, <sup>, which does the exponent of the second-to-top element on the stack to the power on the top of the stack, and leave the result on the stack

            the extension to HTML, the programming language is defined entirely within HTML, the programming language, itself

            due to the way the interpreter of HTML, the programming language, works, you can also implement arbitrary control flow if you’d like, by returning the next element/command to execute (rather than “undefined”, which is used to tell the runtime to execute the next element, if any)

      3. 1

        The mechanism for collectively deciding if something is interesting or not is already in place, and that is the voting system.

        I agree that they could have made even more out of the idea, though.

    3. 17

      I’ve been working on databases for some time, and you win an upvote.

      Sure, I can find some nits and have various opinionated color to things (learned indexes, for instance, are interesting but ultimately useless in practice under write load), but generally speaking you’ve done a great job simply highlighting (and, thankfully, not overexplaining or dismissing) many of the hard problems of databases. Double props for getting down to fsync() which, my god, don’t deal in filesystems, kids, because every part of the stack lies.

      Will be linking people to this when they’re learning as a teachable moment :)

      1. 3

        Double props for getting down to fsync() which, my god, don’t deal in filesystems, kids, because every part of the stack lies.

        I’m laughing with tears in my eyes because I’ve been there… and at the time (circa linux kernel 2.4.x) it seemed that the more complex your storage provisioning (ex. SAN) the less you could trust fsync.

        1. 1

          I think that’s still essentially valid, and storages didn’t get simpler over time either, as far as i’m aware

    4. 2

      Normally “how to pass the coding interview” stuff is trite but Robert Heaton consistently writes really great stuff. Just check some of his past stuff. Also his first tip is

      1. If you do Google the question beforehand, delete your tracks afterwards

      So you know it’s gonna be good.

      1. 3

        I mean. If you want to pass a code interview with me, and you need to look up stuff in order to solve the problem, tell me that you don’t know, let me see how you search, and it’s all peachy. It’s a very important skill, I think. I might be wrong. I don’t really care? Admitting you don’t know something is important too.

    5. 5

      I would like things to have many names, like I do. Especially software components. That way if I need to get very specific I can use the full name, and if I don’t because everybody got local context, I can use a nickname. I guess renaming things is about as complicated with software components as it is for people and landmarks but I have a hard time understanding why.

      Names matter only if people understand what you’re referring to, they don’t necessarily have to explain what they refer to. But it’s kinda nice when you can get that out of the name. Real world examples would include places of business names after the person running it. “Bob and Son’s Plumbing” is pretty good. Would be more confusing if it did not contain “Plumbing”. Locals probably refer to that outfit as “Bob and Son’s”, “Bob’s”, or maybe even “the plumber” if the town is really small. Conversely you could have something like idk, the “Bob and Son’s Message Queue Service”. Nobody would care unless you started having many message queues, I guess? Idk.

      Naming is weird and complicated and contextual, and I believe efforts to magic away the complexity by making them just opaque strings might be unwise.

      1. 3

        Software needs middle names so I can properly scold it.

    6. 3

      Blacksmithing all day Saturday. Some window restoration work, at some point. Probably gonna try to trap and relocate what I think is a raccoon pooping all over my shed/workshop

    7. 5

      I would like to research technology that does not require attention. I would also research technology that reuses old hardware instead of throwing it away. I would also research technology that makes running the software in your own home (as opposed as in a SaaS) feasible again, possibly piggy-backing on the “reusing old hardware” research.

      I would love for my mom to be able to (relatively) securely run a piece of software at home without having to understand the intricacies of setting it up. Something like how my smart tv runs, at most. I know that phones home, but like. You get the idea? Kinda? Idk.

      More local shit that doesn’t require your attention, more software that you own on hardware that you own, that will still run when my company goes under. Peaceful, community-building things, that help you be in the real world, not suck you into a virtual hellscape.

    8. 12

      Please name your mascot Nickelangelo.

    9. 1

      Probably installing a blower on my forge. If the motor ain’t blown, I might use an electric control for managing the speed of the motor instead of controlling air intake by means of blocking out the pipe with a sheet of metal, adjusted for desired flow. Probably cutting a sheet of metal into a square shape, putting holes in it, in order to finish putting the forge to usable shape. Maybe installing the chimney on the forge so that the smoke emitted by the burning coal doesn’t end up in my lungs. That maybe is more of a “maybe this week, maybe next week” kinda deal, I really don’t wanna breathe that shit. Maybe spending a bit of time using my grinder with a metal brush attachment to remove the rust coat from the forge, although I might keep that in, unless it’s actively detrimental.

    10. 4

      One more week of vacation, where I will definitely get books on Iceland in order to plan the honeymoon trip this summer. I might, maybe, work on getting a forge up and running, or buy an old one from someone within a reasonable distance from here. If I am very very very lucky, I might even hit heated steel with a hammer.

      1. 3

        If you can, I loved the freedom of renting a small camper van when visiting Iceland. We had a rough plan of the sites we wanted to see and with the van you can spend as much or as little time at each place.

        1. 2

          We rented a gorgeous little house by the sea, we’re there for 8 days. We want to do a few museums, go for a thermal bath, do the thing where you go down the volcano. There’s also this fishing trip thing we saw that I think would be cool to do, but as I list those things it already sounds kinda tight 😂 I mean, gotta keep some time for hiking within Reikjavík, and hiking outside of Reikjavík too. Gonna go by too fast.

    11. 35

      Networking is the place where I notice how tall modern stacks are getting the most.

      Debugging networking issues inside of Kubernetes feels like searching for a needle in a haystack. There are so, so many layers of proxies, sidecars, ingresses, hostnames, internal DNS resolvers, TLS re/encryption points, and protocols that tracking down issues can feel almost impossible.

      Even figuring out issues with local WiFi can be incredibly difficult. There are so many failure modes and many of them are opaque or very difficult to diagnose. The author here resorted to WireShark to figure out that 50% of their packets were re-transmissions.

      I wonder how many of these things are just inherent complexity that comes with different computers talking to each other and how many are just side effects of the way that networking/the internet developed over time.

      1. 4

        Considering most “container” orchestrators (at least the ones I’ve used) operate on virtual overlay networks and require a whole bunch of fuckery to get them to talk to each other, on top of whatever your observability-platform-of-the-week is, the complexity is both necessary and not. Container orchestration is a really bad way of handling dynamic service scaling IMO. For every small piece of the stack you need yet-another-container™️ which is both super inefficient (entire OS sans-kernel) and overcomplicated.

        1. 5

          I’m not wed to containers, but they often seem like the least bad thing (depending on the workload and requirements). The obvious alternative is managing your own hosts, but that has its own pretty significant tradeoffs.

          1. 4

            Containers themselves are fine for a lot of cases. The networking layer (and also the storage/IO layer) are a large source of complexity that, IMO, is not great. It’s really unfortunate we’re to the point where we’re cramming everything on top of Linux userspace.

          2. 2

            There’s a bunch of different options that have varying degrees of pain to their respective usage, and different systemic properties between each of them.

          3. 2

            For me the killer feature is really supervision and restarting containers that appear to be dead, with the supervision done by a distributed system that can migrate the containers.

    12. 10

      As a warning I’d like to point out that using awk and jq splitting on just , will subtly break if you have strings in your CSV files that contain that character as part of the string. Good stuff though!

      1. 2

        That spreadsheet programs decided to call what they do “CSV” has caused no end of confusion since the expansion of the acronym literally contradicts the format { unless you expand it “Commas (don’t actually) Separate (all possible) Values” ;) }. I would not be shocked to hear that internally they wished they could’ve used “EQCSV” but 1970s era DOS 8.3 filename limits motivated “CSV” instead.

        The best approach here is to have a “translator program” convert escape-quoted-CSV into values “actually separated” by delimiter bytes: or its C port are one example. As part of a pipeline you likely get parallel speed up. Arbitrary data fields can be done by Anyway, then you can use ideas from the article without as much worry.

        Once confident ASCII newline only terminates records, if you have big files & enough space (temporarily anyway) for a copy you can also then segment by “nearest ASCII newline to 1/N” and parallelize processing on it.

        1. 3

          I have taken to using ASCII characters 28-31 to build any “CSV” files I create. It eliminates any delimiter collision and, being ASCII, is nicely handled by most tools.

          1. 2

            It is historically odd that people don’t use those characters more. It’s not like they’re in the 128-255 range and can’t be used or something. They’re right there, universally available, and designed for specifically this purpose. The only problem with them is they can’t encode nested records/arbitrary data, but no one wants that anyway.

            1. 1

              Another downside is if you’re producing a CSV for someone else, you’ll have to explain to them such delimiters exist and how to make use of them if they’re opening the file in Excel.

              1. 2

                I guess the answer to the immediate historical question is that Excel supports CSV, so CSV is popular. But the deeper question remains: Why did Excel use CSV instead of the actual delimiters that were designed for exactly this purpose? Then again Excel was clearly, with all due respect, designed by amateurs, who for example don’t know how leap years work, so it wouldn’t be out of character for them to just not be aware of the ASCII separator fields either.

                1. 3

                  Why did Excel use CSV instead of the actual delimiters that were designed for exactly this purpose?

                  Because those delimiters aren’t on keyboards. Spreadsheet programs before Excel used CSV too.

                  Then again Excel was clearly, with all due respect, designed by amateurs, who for example don’t know how leap years work, so it wouldn’t be out of character for them to just not be aware of the ASCII separator fields either.

                  Excel’s leap year bug is intentional because it’s supposed to be 100% backwards compatible with Lotus, and that includes having the exact same logic bugs.

                  1. 1

                    Because those delimiters aren’t on keyboards. Spreadsheet programs before Excel used CSV too.

                    This is precisely the problem. If I output a CSV using these symbols in one of my projects, then I also have to provide handy copy/paste symbols in the README and hope someone comes back to find that instead of giving up as soon as it looks funky when they open it. These are great for my personal use, but I don’t consider them appropriate for general consumption if I want my code to be used by many.

    13. 4

      I wish that the fediverse had a PGP-like trust system, where I can specify the trust of a given party to my fediverse instance, and then validate the trust of randoms relative to the explicit trust of parties I do trust. Preferably have the possibility to apply some ranking based on hop distance between me and the rando, and trust level (maybe decrementing as you further remove yourself from the rando). This would/could have a nice effect of reinforcing networks where you’re likely to have “actual contact” with the other parties, which is really what I want in a social network: stuff from people I know, or that the people I know can vouch for.

      You can have some priorization of content based on “stuff I like”, a “recommendation engine”-sort, if you will, however you decide to implement that, but to me it would be more practical and desireable to have the social priorization first.

      1. 2

        I think Urbit’s ID model goes a long way laying the ground work for this type of thing.

        I think the federated approach will never work for reasons others describe here (even email and the web broadly are failures that primarily lead to centralized systems).

        To really solve this requires fixing problems earlier in the stack:

        1. 12

          Unfortunately, Urbit has moldbug’s neofeudalism at its core, baked in the design of the protocol and language. And artificial choices like 2**32 systems (or whatever it’s nomenclature), choices of language to obfuscate ideas, and leadership - all of these show me the original designer’s “ideals” are inherent in that design.

          I’ll pass on that.

          1. 1

            It’s worth a deeper look imo.

            I don’t align with the politics of the founder, but the reasons for the system design are independent of that (and I think correct).

            Smart people tend to prematurely undervalue things when they dismiss them for unrelated reasons - I think that’s largely the case here.

            1. 8

              I was careful in how I said my response.

              If it was just because the founder was present, it’s one thing. He’s no longer there. However, the ideals of neo-feudalism apply at all levels of Urbit, specifically around “land ownership”, “disowning users on your land”, and the like.

              The system forces an hierarchy where one shouldn’t necessarily exist. Instead, it instead forces it on everyone, in the way feudalism did so in history. That inherent design choice is what I wholly reject.

              And the language all inherent of Urbit also serves to cover and distract from these core choices. And along with distracting, it also does a good job in making sure that ideas in that system are effectively land-locked in understanding their way of things, without a good translation.

              As a corollary, Lobsters also has a feudalist-like invitation system. However, one above you cannot “disown” you or otherwise control you (unlike urbit), destroying your account. And I’d think that @jcs and other sysops here would also frown pretty greatly if I started selling invites here.

              (Edit: as an aside, Mastodon and the fediverse is different. Sure, we’re running on someone else’s server. And they can boot us. However, I can move elsewhere, no longer under the influence of admins I don’t like. Or I can make my own. There’s no way to make your own “urbit” - it doesn’t federate, and it’s owned by someone who can deplatform you for no reason.)

              1. 2

                The land metaphor doesn’t matter - it’s the IDs that enable moderation to actually be possible at the user level and it’s the mild scarcity of these IDs (4billion initially) and cheap, but non-zero cost that prevent the spam problems that cause things to recentralize.

                Federated systems are worse about this - a handful of servers end up being actually feudalistic and capriciously enforce rules (see: It will always be a handful of servers because these systems don’t solve the root problems that cause recentralization (spam, linux sysadmin complexity, true p2p). You end up in a worst of both worlds situation: a crappier experience than good centralized systems, but with even worse security. It doesn’t solve any of the problems it sets out to at scale due to incentives that lead to recentralization.

                On Urbit there’s no distinction between user and ‘server’ so this doesn’t happen. The hierarchy only serves to route traffic updates to prevent version mismatch problems that plague federation (they’re more like ISP routers) as well as the ability to do public key lookups for setting up p2p connections between users. You could also just run urbits outside the hierarchy entirely if you wanted to for some reason and there’s a large number of traffic routing nodes, so there will be a lot of options along with the ability for users to push back (akin to web users pushing back on ISP routing).

                The language/OS design is about solving complexity problems that lead to recentralization (which are hard to solve) that’s why separating the kernel from the OS it’s running on is important (and having it be a functional event log is important) - everything stems from that core idea.

        2. 2

          I’d like to turn that on its head: email is a resounding proof that federation works. Same goes for the web. True, email and the web at large have largely coalesced into a handful of ginormous players. That being said, you still can send email to those even if you’re outside that oligopoly under very specific conditions. Within that oligopoly, it mostly works. I think for email and the web the problem is more the ease of access (or lack thereof) for the layperson. It has not been a commercial focus to make it easy for Everyone To Host Their Own Crap because I don’t think there is a whole lot of money to be made in it (relative to the costs of supporting Everybody).

          1. 4

            It “technically” works, but it failed to achieve its goals (of the 90s cypherpunks anyway). My argument is that fixing the underlying system design could fix the incentives that lead everything to centralization, but it won’t happen via federation and it (likely) won’t happen with the existing tools.

          2. 3

            email is a resounding proof that federation works.

            I can’t even apply for a hCaptcha accessibility cookie using Yandex because I need to “use a real email address”. Handing the unstoppable deluge of spam email addresses (both servers and compromised accounts) is an entire industry. Gmail drops inbound and outbound mail effectively randomly. Email is an abject failure, which is why in developing countries most communication is done over centralised social media, be it WhatsApp, Facebook or their local thing. We only use email because it was good enough as the only option, and reliance on it ballooned.

    14. 5

      I had briefly falled down a similar rabbit hole a while back with one of those Planck ortho-linear keyboards, but I returned to a regular keyboard layout for the following reasons:

      • I occasionally play video games, and need the odd key here and there
      • I have a hybrid work environment, sometimes from home, sometimes from the office, so I would need to either always carry my keyboard around, or have two.
      • This also makes it impossible (or at least confusing) to use the built-in laptop keyboard.

      But if it works for the OP, I’m all for it. Ergonomics are super important in this line of work.

      1. 3

        I have the Planck EZ and more or less the same problems, but about 4 months in:

        -I have made custom layouts for games -I sometimes work away from my home office and at those times I do carry the Plack EZ along with all the wiring -In extremis I can always fall back to the computer’s builtin keyboard and it’s not all that jarring (for me, anyway).

        Also 4 months in I have observed that I am using the keyboard wrong and/or the columnar layout is not helping me much. My fingers travel a lot anyway. I think I must be using it “wrong”

        1. 1

          Yeah, it’s not insurmountable, but I think I underplayed how much I play video games, and it’s not feasible to program a new layer every time you find a new game with slightly different keybindings, I find.

          As much as I like the whole QMK open firmware project (and it’s related projects), it’s not exactly a rapid process to change things around.

          1. 2

            True. I don’t play a whole lot on the computer, I’m more of a console person (and will often prefer a controller even on computer). When I do play on the computer, games tend to have similar bindings by “genre”, more or less. If I wanted to play more using my Planck, I probably would have layers by genres.

      2. 3

        i have been alternating between a staggered-qwerty (laptop keyboard) and ortho-colemak (the ferricy), and i am comfortable with both now! i have been able to consistently hit ~90 WPM on both layouts, it takes me a few minutes of typing to “switch” my brain over from one to the other.

        1. 2

          Nice overview. I’ve been rocking 42 keys for nearly a decade now and I’d never go back, but I really only have one layer I use regularly.

          One thing I’m curious about that the article didn’t mention: how long did it take you to get proficiency in this layout? (For me it took about 3 weeks to get fast on the Ergodox, and once I had that proficiency, bringing it down to 42 keys on the Atreus only took 2 weeks, but from what I hear about other people switching to the Atreus, 3-4 weeks is common.)

          1. 2

            glad you liked it. big fan of the unibody-split design of the atreus.

            the descent to 34 was gradual. i started out with a Lotus58, plucked out a few keys until i got to 48, 36 and finally 34. All in all, it took me around 3 months to go from a 60% to a 35%. That being said, I am not as fast as I was on staggered-qwerty yet. I am currently hovering at about 90 WPM on the ferricy, whereas I could hit upwards of 130 on qwerty. going from 36 to 34 was particularly tricky, every key is load-bearing at that point.

      3. 2

        I’ve been using non-standard layouts for 15+ years, and a mix of ortholinear and normal staggered keyboards for 5+ years. I can switch layouts mid-sentence and go between staggered and ortho layouts in a breeze as well (the only awkward part is to have two keebs on the same table), the entire typing should be in your muscle memory and not in your head. It can be done, without issue.

        And keyboards like these have nearly nothing to do with ergonomics. Keyboards are awkward and stupid to use for humans. :)

        1. 1

          And keyboards like these have nearly nothing to do with ergonomics. Keyboards are awkward and stupid to use for humans. :)

          Do you think any writing/typing implement is ergonomic?

          1. 1

            The closest are probably Maltron, Kinesis Advantage, Dactyl and friends. And while I love using dvorak I don’t have any delusions that my layout of choice would improve ergonomics in any way (beyond placebo, which is powerful in itself).

      4. 2

        I switched to ortholinear about 4-ish months ago. I swap back to a standard TKL board for gaming, though that’s partially because I have a tented split keyboard. At first I had a little trouble switching back and forth between ortholinear and staggered, but after the first maybe 2 weeks I don’t have much trouble switching back and forth.

      5. 2

        Ergonomics are super important in this line of work.

        Agreed. And it’s great that there are so many keyboard options because it seems everyone needs something different. I love the Planck, despite its flaws. After trying a few different styles I settled on the Planck because I have small hands and the less distance my fingers have to travel the better.

    15. 19

      Most companies are not using cloud as a replacement for colo. RDS, SQS, S3, managed elasticsearch, etc are really really valuable and difficult to replicate on your own. Of course the cloud vendors want to lock you in to these services and then overcharge you for the basics, just like some grocery stores lure you in with cheap specialty foods and then overcharge for bread and milk. It doesn’t mean it’s a bad deal though.

      1. 19

        RDS and S3 are standouts in part because the lock-in is operational, not architectural.

        You can develop against vanilla PostgreSQL, deploy on RDS, then change your mind – or at least threaten AWS with a change at contract renewal time – and switch to’s managed Postgres. (Or any of the other excellent hosted offerings.) Or go “on-prem”, “edge”, etc. (I.e., run your own servers.)

        S3 was a moat but the API + domain model are now available from your choice of vendors, including Minio if you want to roll your own.

        I’m far more suspicious of applications that make heavy use of SQS, DynamoDB, etc. without having a really strong proof they need that scale and all the distsys pain it brings. You can get a long way on Celery (or your choice of “worker” tools) running batch jobs from your monolith against a “queue” table in Postgres. IME most projects/companies fail long before they outgrow the “COSP” threshold.

        For cost management, disaster recovery and business continuity, and the ability to work + test your systems offline, I think minimal cloud provider API surface in your application is a Good Thing. That + “don’t create a ton of microservices” (also good advice in most cases) usually implies: monolith + one big backend database + very select service extractions for e.g. PII that shouldn’t sit in the main DB.

        1. 4

          I think you nailed it here:

          the lock-in is operational, not architectural.

        2. 1

          You can develop against vanilla PostgreSQL, deploy on RDS, then change your mind – or at least threaten AWS with a change at contract renewal time – and switch to’s managed Postgres.

          How does this work with security, though?’s managed Postgres is going to be open to the internet, presumably, whereas in AWS I can control (and log) network access as I see fit.

          1. 3

   postgres is very much not open to the internet unless you need that for some reason.

          2. 2

            I think Fly has a pretty good story here, actually:

            But really, any managed DB vendor is going to have better network controls than “just use pg_hba.conf”. Most even offer AWS VPC bridging.

            1. 1

              Thanks for the link. I was maybe thinking of Supabase when I wrote the comment. Like if the business is providing managed databases but no compute then doesn’t the database basically have to be open to the internet so the backend servers can reach it? Eg talking to Supabase from Vercel or Netlify? Or can something clever be done with eg Wireguard to secure it all?

              1. 2

                There are a few approaches that services like this take. Sometimes they provide access over a VPN (e.g. through Wireguard, this is what managed Postgres does if you connect from a service and how you connect to private RDS databases from outside AWS), and sometimes they do just have a database listening on an Internet IP/port (maybe secured by some IP whitelisting, usually secured by TLS, and definitely secured by username/password authentication; this is what DigitalOcean managed databases, Supabase direct connections, and public RDS databases do)

      2. 3

        I guess it goes without saying that if you

        • need 99,99+% uptime and want to sueblame somebody big otherwise
        • need a distributed database for a ton of access that “Just works”
        • want a “familiar” stack where you can just slap some specific product of the three letter company as a requirement in the job description

        … then go to the big cloud providers and pay your premium (be aware of the network and database per-operation fees), you already made up your mind.

        But I’d bet that are maybe 1% of the customers.

        1. 9

          need 99,99+% uptime and want to sueblame somebody big otherwise

          I haven’t checked in a while, but I’ve never seen a cloud service actually meet this 99.99+% uptime. I don’t think any of them are very transparent about their historical outages anymore as they realized they weren’t having good uptime performance.

          I checked a few years ago for $WORK, when some boss type wanted to move to the cloud, I compared out all the data I could gather from the various cloud providers and we handily beat them in uptime and total cost across time. I think I went back 5-ish years at the time, though I can’t seem to find that spreadsheet at the moment.

          I agree there are valid reasons to move, but I would never blindly recommend switching dedicated stable compute to the cloud. Bursty compute however is a perfect fit for the cloud, and easy to recommend.

          1. 1

            I’m always worried about comparisons in uptime to someone’s single company to big clouds. AWS will have both more issues and more varied ones, but they’ll be often limited in scope. It’s hard to compare it to a smaller local setup without a list of specific risks and expected time to recovery. At an extreme, the box under my desk at home had 100% uptime in the last few years, but I wouldn’t make decisions based on that.

            1. 3

              I agree a single companies uptime comparison vs cloud providers isn’t very useful to outsiders, but it can be useful in that single companies decision making. That’s why we did the comparison.

        2. 15

          need 99,99+% uptime and want to sueblame somebody big otherwise

          More importantly, don’t want to pay for in-house expertise to manage the systems when it is not part of their core competency. For smaller companies, they often need 10% of a very qualified sysadmin. They can either hire a full-time one for 10x the price of what they actually need, or outsource to a cloud provider and, even if the markup is 100%, be paying 80% less.

          need a distributed database for a ton of access that “Just works”

          The ‘Just works’ bit is far more important here than the ‘distributed’ or ‘ton of accesses’ part, because it translates to not having to pay an administrator.

          want a “familiar” stack where you can just slap some specific product of the three letter company as a requirement in the job description

          Again, this is a cost-saving thing. It’s much easier to hire in-house talent or to outsource a particular project to a (small or large) company if the infrastructure that they’re building on is generic and not something weird and bespoke that the developers would need to learn about.

          In a huge number of cases, the cost of the infrastructure (cloud or on-prem) is tiny in comparison to the cost of the people to manage it. Using the cloud lets the provider amortise the cost of this over millions of customers and pass on a big chunk of that saving to you.

          Buying a big server has a few drawbacks. If any hardware component fails, then you need to RMA that part, which means you need either an expensive support contract or you need someone on staff who is competent to identify the faulty component and send it back. If a cloud server fails, then your VM is restarted on another machine. If you are using PaaS offerings then someone else is responsible for building a platform that handles hardware failure and you don’t even notice.

          If you want a separate test and production version, then you need at least two of those big servers, whereas with even IaaS offerings it’s trivial to spin up a clone of the production server for a test deployment on a different vnet and if you’re using PaaS then it’s even easier, and the number of test instances can easily scale with the number of developers in both cases.

          TL;DR: If you think the cost of the hardware is important then either you’re thinking about massive deployments or you completely misunderstand the economics of this kind of thing.

          1. 14

            In my experience the companies I have worked for tend to end up at least doubling their spend when moving from dedicated to cloud for little added benefit and almost the exact same maintenance burden, in one case a company I worked for they went from £3,200/year spend on a managed 24-core/112GB RAM dedicated box with 1 hour SLA on having a tech at the datacenter make changes/do maintenance/etc to ~£1,400/month spend on far less resource except now they now had to handle the server changes/maintenance in house on top of managing the cloud infra which actually required hiring someone new to handle.

            For my own company we rent two dedicated boxes (16-core/64GB RAM each) at a total cost of £108/mo which provides more than enough capacity, and in the past six years has had five nines uptime while costing a fraction of what it would have to go with cloud.

            1. 1

              now had to handle the server changes/maintenance in house

              I’m not sure I understand. What server maintenance are you doing for a cloud based servers that’s comparable to the dedicated one?

              with 1 hour SLA on having a tech at the datacenter make changes/do maintenance/etc

              That’s 1h SLA to having someone look at the issue, not for a working replacement, correct?

              1. 1

                I’m not sure I understand. What server maintenance are you doing for a cloud based servers that’s comparable to the dedicated one?

                It was more running updates, kernel patches and such. With the managed setup the hosting provider acted as ops and took responsibility for ensuring updates didn’t break production, they were our sysops. There were a few cases when we were being bottlenecked by various hardware and requested it replaced. Every so often I got a call from the datacenter’s ops team to confirm server access was legitimate, or to inform me the server had some unusual activity on and they had investigated over night.

                That’s 1h SLA to having someone look at the issue, not for a working replacement, correct?

                Typically it was an instant phone call to someone in the datacenter who would either remote into the box, or walk over to it and deal with it in the rack, the SLA was on getting hold of someone on the floor to talk with who could remote in and diagnose what was wrong live. No call centre, no account handler, no middle men, a direct line to an experienced sysops engineer; that’s pretty rare nowadays.

          2. 13

            A couple of nits, directly:

            More importantly, don’t want to pay for in-house expertise to manage the systems when it is not part of their core competency.

            I would argue that managing systems is a core part of developer competency, and I’m tired of people acting like it’s not–especially when those people seem to frequently employed by companies whose business models depend on the meme of systems administration being some black art that can only be successfully trusted to the morlocks lurking in big data centers.

            Using the cloud lets the provider amortise the cost of this over millions of customers and pass on a big chunk of that saving to you.

            This is manifestly not what’s happening, though, as we’re seeing. The savings are being passed on to the shareholders–and if they aren’t, we should all be shorting MSFT and AMZN!

            If you want a separate test and production version, then you need at least two of those big servers

            Or, you know, you host both things on the same box under different VMs, or under different vhosts. This has been a problem with a well-known solution since the late 90s (though sadly not reliably applied).

            you completely misunderstand the economics of this kind of thing.


            • We’ve seen figures in this very thread of at least a 2x price increase using cloud providers.
            • The option typically doesn’t exist to not have a sysadmin–we just hire “devops” people now, who are okay sysadmins who also tend to spend most of their time functioning as embedded salespeople for the vendor of their preferred stack. We’re out a six-figure salary regardless.
            • If your team opts not to have a sysadmin (!), a bare metal or rented dedi is a lot easier to understand and maintain since it basically looks like a developer machine–just beefier and with actual paying customers on it.

            I submit that perhaps we aren’t the only ones who misunderstand the economics. :)


            To be clear, there are some things like S3 that I just cannot be arsed to host. Hosted Postgres is nice when you don’t want to bother setting up metrics and automatic backups–but then again, I’m pretty sure that if somebody wrote a good script for provisioning that or a runbook then the problem would go away. It’s also totally fine to keep a beefy machine for most things and then spin off certain loads/aspects to cloud hosting if that’s your kink.

            Remember, there was a time when the most sensible thing was to send your punchcards and batch jobs down to the IBM service bureau, because it was more economical. These things go in cycles.

            1. 9

              Addendum, reading back over this:

              The more I think about this, the bigger issue is probably that if you run your own infra there’s the requirement that there be some continuity of ownership and knowledge–and that is difficult in an industry right now where average tenure is something like less than two years for startups.

              1. 4

                Most of my career so far has been, essentially, cleaning up somebody else’s historical mistakes by paving over them with my soon-to-be historical mistakes. An endemic part of the problem is always that very specific and arcane parts of the system are forgotten, or stop being understood, as the flow of brains does its thing. I used to be in camp “rewrite”, a decade ago. I’m now firmly in the camp “nooooooooo, fix it, please don’t do this to me, please please please fix it”

              2. 3

                I’m honestly dumbstruck by how obvious this is once it’s pointed out explicitly.

                Even when I started out 15+ years back, I had the distinct impression that traditional “ops” roles tended to have far higher average tenures than developer roles.

            2. 4

              I would argue that managing systems is a core part of developer competency

              I am not talking about developers, I am talking about companies. Most big cloud customers are not software companies, they are companies that have some in-house infrastructure that is a cost centre for their business: it is a necessary cost for them to make money, but it is not the thing that they make money from. They may employ some developers, but managing infrastructure and writing code are different (though somewhat overlapping) skill sets. Importantly, developers are not always the best administrators and, even when they are, time that they spend managing infrastructure is time that they are not spending adding features or fixing bugs in their code.

              For a lot of these companies, they outsource the development as well, so the folks that wrote the code are contractors who are there for a few months and are then gone. An FTE sysadmin is a much higher cost.

              This is manifestly not what’s happening, though, as we’re seeing. The savings are being passed on to the shareholders–and if they aren’t, we should all be shorting MSFT and AMZN!

              That doesn’t follow. If it costs 100 times as much to manage 1000 machines as it does to manage one, then a company that passes on half of the saving to their customers will still be raking in cash. The amount that it costs to maintain a datacenter of a few tens of thousands of machines with a homogeneous set of services running in large deployments across them is vastly less that the cost of each customer maintaining their own share of that infrastructure.

              We’ve seen figures in this very thread of at least a 2x price increase using cloud providers.

              The numbers I’ve seen there are comparing hardware cost to hardware cost, which ignores the bit that’s actually expensive. They’re also talking about IaaS, which does not get most of the savings. And they’re talking about companies with steady-state loads, which is where IaaS does the worst. Renting a 64-core server is probably more expensive than buying one (a cloud vendor will pay less for it by buying in bulk, but that’s not a huge difference, and they want to make a profit). The benefit that you should get from IaaS is that you can move between a 2-core server and a 64-core server with a single click (or script) so that you can scale up for bursts. If you are a shop with a trickle of sales across the year and 100 times as many on cyber monday, for example, then you might need a 64-core system for 2 days a year and be happy with a 2-core machine the rest of the time. Comparing buying and renting a 64-core machine for the entire year is missing the point.

          3. 2

            Not just small companies. Larger companies often have terrible tech ops. Moving to ops as a service can be a way to fix that, though there is the danger that your existing ops people and processes will infect what you do in the cloud and either prevent you from getting the advantages or even make it worse than what you had.

        3. 5

          Interesting, it didn’t occur to me that only 1% of customers would want good uptime they’re not responsible for, a reliable database, and an easy to match watch-word for hiring.

          1. 4

            I’ve got 99,99 SLA one some tiny box at some irrelevant hoster in germany, with a downtime of 1 hour in 10 years when the whole box died (was up then again in 1hour on another system). So you could say I’ve got my 99,9% without any failover.

            If that’s possible for a normal company with only some KVM + guaranteed CPU, RAM and bandwidth, you may not need the big(tm) cloud for that same hardware.

            1. 4

              I have seen far more (and longer) outages caused by messing up with cloud systems than by hardware failure.

              Some examples I have personally seen:

              • Autoscaling policies based on CPU load / memory causing outages when load patterns shift
              • Brief but frequent “elevated error rates” caused by insufficient wait periods on scale-in events
              • Network speed degradation in AWS causing application outages
              • Cron-triggered script to terminate/delete un-tagged resources (to ensure people were tagging things for cost control purposes) ran during an outage of the AWS tagging service. All resources were reported as un-tagged and 30% of instances were terminated before it killed the instance it was running on.
              1. 1

                also I didn’t mention it, but I’ve got a 24/7 hotline in case my system is down, won’t pay anything as long as it’s not my fault (then I’m billed for every 15 minutes), and I did use it at one sunday when the network latency spiked

        4. 2

          need 99,99+% uptime and want to sueblame somebody big otherwise

          Many companies and even just clubs and stuff had that kind of uptime long before cloud providers even were a thing and if you look at guarantees from cloud providers you will generally not find more guarantees than what most companies provide. While cloud providers have more staff they also have way more complexity than smaller companies, bringing their own kinds of outages and every now and then you hit limitations of managed services, need to upgrade cause they decided to change something, which can be less planable than in your own company. And good luck if you hit some bug based on the particulars on how you use the service and going through layers of support lines, unless you are really big - big enough to easily do most stuff in-house.

      3. 2

        Elastic Search I set up ten years ago on physical machines and was fairly trivial. I think early on that was one of their main selling points. We even helped a very big bank to set it up on their infrastructure. When we came over to discuss any remaining topics they were done and built their own orchestration around it. Fun fact they built basically their own Nomad/Kubernetes and I think it was largely shell script (not completely sure though!). I don’t know how it is these days though.

        S3 is pretty easy to replace and low maintenance with things like minio and seaweedfs.

        And also, if you ever run any serious setup where you (think you) need the cloud you will certainly end up troubleshooting issues on the managed services, but only after scraping together enough evidence that it’s their issues. Even more fun when you have to go through their partners first. So you need people that are both experts in the field, but also experts with the particular cloud providers. So, in any capacity where you think you might actually need cloud providers you certainly need people that could easily set things up on their own. And that is why you can make a ton of money DevOps jobs, if you like doing that. There’s always need.

        But even if you happen to never run into any of these problems. You usually need experts for technologies you use, way before your standard server setup is even close to limit you somehow. And usually it’s not a clear cut how much they need to know. So they will certainly know how to run these technologies. Again, that’s if you don’t run into any issues with your cloud provider’s setup and that at some point will happen, even with Amazon and Google. After all they also run physical hardware, have tons of management infrastructure that also can have bugs, have situations that their monitoring doesn’t detect.

        The biggest thing is that you can blame them, but then you need to be able to proof it, which can be really hard at times, especially if you don’t know their setup.

        I think there is a lot of “right sounding” things said about cloud computing, that also typically aren’t inherently wrong, but still at best only apply to the practical reality to a certain degrees and cloud providers would be stupid not to make statements based on that and people wanting to get DevOps jobs, do consulting, sell books do the same. I think it’s rarely intentional though. It’s just easy to make a generic true-ish statement to justify what we do. But that goes into psychology.

      4. 2

        That’s the thing. There are a small number of companies whose domain/problem space is such that they can 100% avoid lock-in by treating cloud instances strictly as VMs and running all their own services, but as your needs grow that can be SUPER hard to maintain without a sizable investment in engineering which not every company is willing to make.

        Maybe they should? But they aren’t.

    16. 32

      And that’s why the abuses of techno-optimism from the ruling classes are creating a new wave of luddism inside and outside the tech industry.

      The argument from OP is not new: the conflict of humans vs machines has been a major trope of 20th century philosophy, literature and art, especially after the brutality of nazi-fascism in Europe. Actually, it’s the whole premise of entire fields of study, political institutions and organizations.

      Obviously, this stuff is not taught to engineers, that are trained to implement acritically anything that is requested from them. Just sprinkle some ethics-washing on top and they will believe they are the good guys.

      It’s always fun (not really) when techbros discover they are perceived as the “bad guys” outside their bubble. They get mad at people writing “no programmers” or “no cryptobros” on dating apps or “if you work in tech, everybody hates you. Just leave” on the walls of a gentrified neighborhood.

      1. 4

        Obviously, this stuff is not taught to engineers

        Depends on the schools, in Québec (maybe in the rest of Canada, I don’t remember) we are required to take a course on ethics in engineering. I also had course on sociology (also geared towards technology and engineering), but I don’t know if it’s required outside of Polytechnique of Montréal.

        1. 4

          This kind of courses are taught throughout the world, for what I know, but they are very very shallow compared to the responsibility and a power that a software engineer has. Also they tend to reinforce an idea of ethics that supports the status quo and usually draws the line at “95% of what is being done with technology is totally ok, the remaining 5% must be eradicated and please don’t put AI in your weapons”. I don’t know the one you took, but all the syllabi of the courses I’ve seen are wildly insufficient.

        2. 4

          Canada uses the word “engineer” very differently from USA. Here is it a regulated term with requirements to be one (including ethics training). In USA it can describe almost any practical job, but in this context often means “someone paid to write code”.

        3. 3

          Hi, I hail from Quebec too, and I’ve been practising software development for the past decade and a half. Can’t legally call myself an engineer, only went through college. Most of the people I have worked with over a decade and a half are not legally allowed to call themselves engineers. So “this stuff is not taught to engineers” is not true, from a very technical standpoint, but the reality on the ground is that indeed, the practitioners are not taught that stuff.

          1. 1

            That’s a pretty good point, almost all my colleagues went to the same engineering school, so I tend to forget that not all software developer went to engineering school.

      2. 2

        They get mad at people writing “no programmers” or “no cryptobros” on dating apps or “if you work in tech, everybody hates you. Just leave” on the walls of a gentrified neighborhood.

        They sure do love the engineers’ salaries when it comes to supporting a family or paying taxes for their community programs, though. Damnedest thing.

        1. 2

          Flaunting money is possibly even more repelling than being into cryptos.

    17. 3

      Finishing up work where I’m currently employed, preparing for my new place of employment. That’s gonna be most of it, really. Not-work-wise, snow’s been melting a lot, so probably a good deal of cleaning up on the yard.

    18. 1

      There’s a decent snow storm supposedly headed our way tonight lasting through much of tomorrow so we’ll hunker down. I will be planting seeds for the garden, as well as build the temporary indoors greenhouse that’ll hold the plants until spring. We have huge maple trees on the property and my wife’s father might come by on sunday to see if we can pull out some sap to make syrup this spring. Oh. Using the snowblower a lot too.

    19. 1

      I might put together a very simple note system that fits with my immediate flow, having seen a few people post about that this week. More or less, i only type notes from my computer, so it should be easy to write and consume from there, where I also do my work. Also, infrequently, I want to refer to my notes using my phone, so l will have a very minimal web interface on top. Probably slap a text search on top. No tags, no categories, l will lean on the search functionality for that. Initially I thought of using codesearch for that, and maybe rank the search results by frequency of search terms or something. Idk. I want it to be really simple to use, and have zero frills.

      Otherwise I’m feeling a little down and overwhelmed and professionally bored and I’m probably going to use Final Fantasy XIV as a temporary cure for that.

    20. 1
      • Continuing to turn lengths of slightly rotten timber into boxes/bags of cut up wood drying out in the shed
      • Mowing the lawn
      • Maintenance on the BMW Z4 ahead of travelling in it on Sunday/next week
      • Running a 10km event. Haven’t really trained since I last ran it in September so not expecting great things, but I’m going to do it regardless.
      1. 2

        Mowing lawn, what a notion. Here the lawn is under about a foot of snow!