1. 29

    When an error in your code base can take down millions of users who depend upon it for vital work you should

    1. Have good CI
    2. Have extensive tests
    3. Make small changes at a time
    4. Have at least one set of extra eyes looking at your changes
    1. 13
      1. Make use of language features that push you towards correctness, for example static typing.
      1. 7

        I find it shocking how many people love “dynamic languages”

        1. 5

          I don’t. There’s a lot of neat tricks you can do at runtime in these systems that would require 10x more work to do at build time, because our build tools are awful and far too difficult to work with. Problem is that we only have the build-time understanding of things while we’re actually programming.

          Don’t get me wrong, I disagree with taking this side of the trade-off and I don’t think it’s worth it. But I also realise this is basically a value judgement. I have a lot of experience and would expect people to give my opinions weight, but I can’t prove it, and other rational people who are definitely no dumber than me feel the opposite, and I have to give their opinions weight too.

          If our tooling was better (including the languages themselves), a lot of the frustrations that lead people to build wacky stuff that only really works in loose languages would go away.

          1. 5

            I don’t, because I used to be one of those people. Strong type systems are great if the type system can express the properties that I want to enforce. They’re an impediment otherwise. Most of the popular statically typed languages only let me express fairly trivial properties. To give a simple example: how many mainstream languages let me express, in the type system, the idea that I give a function a pointer to an object and it may not mutate any object that it reaches at an arbitrary depth of indirection from that pointer, but it can mutate other objects?

            Static dispatch also often makes some optimisations and even features difficult. For example, in Cocoa there is an idiom called Key-Value Coding, which provides a uniform way of accessing properties of object trees, independent of how they are stored. The generic code in NSObject can use reflection to allow these to read and write instance variables or call methods. More interestingly, this is coupled with a pattern called Key-Value Observing, where you can register for notifications of changes before and after they take place on a given object. NSObject can implement this by method swizzling, which is possible only because of dynamic dispatch.

            If your language has a rich structural and algebraic type system then you can do a lot of these things and still get the benefits of a static type checking.

            1.  

              Regarding your example, honestly I am not 100% sure that I grasp what you are saying.

              In something like C++ you can define a constant object and then explicitly define mutating parts of it. But I don’t think that quite covers it.

              I have enjoyed some use of Haskell a few years back and was able to grasp at least some of it. But it gets complicated very fast.

              But usually I am using languages such as c# and typescript. The former is getting a lot of nice features and the latter has managed to model a lot of JavaScript behaviour.

              But I have no problem admitting that type systems are restrictive in their expressibility. But usually I can work within it without too many issues. I would love to see the features of Haskell and idris, and others become widely available - but the current languages don’t seem interested in that wider adoption.

              1.  

                Regarding your example, honestly I am not 100% sure that I grasp what you are saying.

                In something like C++ you can define a constant object and then explicitly define mutating parts of it. But I don’t think that quite covers it.

                I don’t want an immutable object, I want an immutable view of an object graph. In C++ (ignoring the fact that you can cast it away) a const pointer or reference to an object can give you an immutable view of a single object, but if I give you a const std::vector<Foo*>&, then you are protected from modifying the elements by the fact that the object provides const overloads of operator[] and friends that return const references, but the programmer of std::vector had to do that. If I create a struct Foo { Bar *b ; ... } and pass you a const Foo* then you can mutate the Bar that you can reach via the b field. I don’t have anything in the type system that lets me exclude interior mutability.

                This is something that languages like Pony and Verona support via viewpoint adaptation: if you have a capability that does not allow mutation then any capability that you load via it will also lack mutation ability.

                But usually I am using languages such as c# and typescript. The former is getting a lot of nice features and the latter has managed to model a lot of JavaScript behaviour.

                Typescript is a dynamic language, with some optional progressive typing, but it tries really hard to pretend to be a statically typed language with type inference and an algebraic and structural type system. If more static languages were like that then I think there would be far fewer fans of dynamic languages. For what it’s worth, we’re aiming to make the programmer experience for Verona very close to TypeScript (though with AoT compilation and with a static type system that does enough of the nice things that TypeScript does that it feels like a dynamically typed language).

                1.  

                  I really like the sounds of Verona.

            2.  

              We have somewhat believable evidence that CI, testing, small increments, and review helps with defect reduction (sure, that’s not the same thing as defect consequence reduction, but maybe a good enough proxy?)

              I have yet to see believable evidence that static languages do the same. Real evidence, not just “I feel my defects go down” – because I feel that too, but I know I’m a bad judge of such things.

              1.  

                There are a few articles to this effect about migrations from JavaScript to TypeScript. If memory serves they’re tracking the number of runtime errors in production, or bugs discovered, or something else tangible.

                1.  

                  That sounds like the sort of setup that’d be plagued by confounders, and perhaps in particular selection bias. That said, I’d be happy to follow any more explicit references you have to that type of article. It used to be an issue close to my heart!

                  1.  

                    I remember this one popping up on Reddit once or twice.

                    AirBNB claimed that 38% of their postmortem-analysed bugs would have been avoidable with TypeScript/static typing.

              2.  

                Shrug, so don’t use them. They’re not for everyone or every use case. Nobody’s got a gun to your head. I find it baffling how many people like liquorice.

                1.  

                  Don’t worry, I don’t. I can still dislike the thing.

            3. 6

              And if you have millions of users, you also have millions of user’s data. Allowing unilateral code changes isn’t being a good steward of that data, either from a reliability or security perspective.

            1. 12

              I believe in mandatory reviews, but I usually communicate what I expect from my reviewer when I request a review. Likewise as a reviewer I ask what’s expected of me when I don’t already know. I don’t think mandatory reviews create a culture of carelessly tossing reviews over the wall, and if you stop doing mandatory reviews I don’t think people will magically try harder.

              One policy doesn’t make or break a culture of developing quality code.

              1. 12

                I worry that a standard process gets in the way of building a flexible culture for collaboration.

                This gets to the heart of my problems with the piece, I think.

                Lack of process does not scale: in an ideal world everyone would be disciplined in their approach but as the size of an organisation increases you need to agree on what “disciplined” looks like.

                Even for small projects, I have on days and off days and I appreciate technology that can prompt me towards the better approach. The way I often phrase this in discussions about tooling is: it should be harder not to use it.

                That comes with a corollary: you should still be able to not use it! No process is sacred or perfect for every situation and it’s important to be flexible. If you’re bypassing something regularly, maybe it’s time to think about why it’s not working. A good process is one that makes your life easier.

                Maybe this is what the author is trying to get to, and I’m missing the point. I’m certainly not arguing that enforcing PR review is the only way to do things.

                But I’m wary of arguing that something is all bad just because we might want to do something different sometimes.

                1.  

                  That comes with a corollary: you should still be able to not use it!

                  This is fine in many situations. The problems crop up when developers do not fully understand their commitments. These should always be explicit, but a significant number of organizations work with significantly complicated customers and industries that understanding the domain becomes as a big a challenge as the project. You also have the human elements of being hired into a role that’s different from the one presented, organization changes, etc.

                  The rules and process in an effective organization, where “effective” means meeting customer requirements, needs to have some level of enforcement. If it was purely size then the option to skip it would be more acceptable, but for many situations it isn’t. I’m ignoring a large set of organizations that operate with risk by unknowingly violating regulations, or have just hit a point where their legal department will finally force them to snap to a process.

                  Quality of life is an important consideration for developers and operators. In the current tech market, this group tends to be well positioned to dictate a lot about what a good process means from their perspective. When it comes to a drive towards process is optional, especially as short-hand for developers will opt out of the processes that they feel like, then it starts to challenge our level of professionalism as an industry.

                  1.  

                    This is very true! Some process should not be skipped - and we’re back to expecting people to use good discipline and judgement, which doesn’t always work.

                    Perhaps in such cases that should read: “you should be able to skip it if you can prove you have a damn good reason for doing so”?

                    1.  

                      Treating the skippage of very important steps as an incident and holding a plain post-mortem is a good start. So less about the skipper proving themselves, more about exploring the circumstances that lead to skippage.

                      1.  

                        Perhaps in such cases that should read: “you should be able to skip it if you can prove you have a damn good reason for doing so”?

                        If you act with good intent and break a regulatory requirement, you have still broken a regulatory requirement. Depending on circumstances this is still a serious issue and not infrequently investigation finds that it was avoidable. It is much better to pause and collaborate with others to overcome the impasse even when you are uncomfortable because the consequences are not always immediate, or direct.

                    2.  

                      I agree. Processes like mandatory code review is important in so far as they document your beliefs and knowledge about what was best at the time the decision was made. Beliefs and knowledge change with evidence and circumstances, and so will your processes.

                      The option to duck out of It – if the situation so demands – is also important to me. I think the only road to long term success is to hire smart people and give them freedom under responsibility.

                      Of course, you could argue that doesn’t scale, and I think that’s not a problem with the principle, but simply a consequence of the fact that human creative work does not scale, period. You can’t construct a well-functioning 500 person product development organisation. In order for it to work at all, you’d have to extinguish many of the things that make creative teamwork powerful.

                      Of course, the alternative might be better: put together the 500 people into small mini-orgaanisations that are as independent as possible, and essentially run a little company inside the company (driving stuff from user need to sunset), but with goals determined by larger coherent strategic initiatives.

                  1. 13

                    We use vegeta at work and it’s pretty great.

                    1. 3

                      Importantly, it does not suffer from coordinated omission, which almost every other load generator does. Especially the ones you hack together on your own. Properly generating load is hard.

                      Coordinated omission is when the load generator backs off on the pressure when the target is overloaded. It happens naturally since you can’t keep an infinite amount of outgoing requests active at once. You have to compensate for this effect.

                      Coordinated omission sounds like something that will just result in overestimated performance numbers, but it’s worse than that. It can easily reverse figures, making performance improvements look like things got worse.

                      1. 1

                        It seems like the Coordinated Omission problem is more or less a euphemism for backpressure. Backpressure is a cross-cutting concern, though. You can see it if any of your systems in the chain slow down, from the load generator, through the web server, appserver, etc. A bottleneck in any area makes the downline systems look “bettter” under reduced load.

                        The remedy for backpressure awareness is that, as you add more concurrency you watch that the graph for throughput and requests/second increase in the exact same manner as the increase in VUs. I don’t think this is a tool problem–just an area to have awareness. For example, as I ramp up load, if I’m using a step pattern, bandwidth and hits/second should follow the step pattern. If they don’t, the test is invalid the moment they deviate.

                        The worst tools, in my experience, are the ones that “throw load” when real users would be “stuck.” Those tools are far more dangerous and have worse failure modes. For example, if the page returned happens to have a 200 status code, with a giant message saying “Critical System Failure” a tool that ignores backpressure and slings load might show you improved performance once the error page appears as it loads fast!

                        1. 2

                          Good points! With your awareness of the problems involved, and willingness to declare a test invalid when the actual load doesn’t match the attempted load, I’m not too worried about you drawing the wrong conclusions – no matter what tool you use.

                          The problem you mention in your last paragraph is a problem whether or not your tool corrects for constant attempted load. Any tool ignores backpressure when the “backpressure mechanism” is a very fast 200 OK.

                    1. 1

                      I’m in my sixth month of paternal leave, so all weeks are sort of the same yet imperceptibly different as my son grows, explores the world and learns new tricks.

                      It’s also a bit mind-numbing following him around – what with him not being able to speak so he just points and grunts. When he sleeps I’ve been working on statistically modeling football to beat some co-workers in a silly prediction pool. So far, I’m not, and my latest models have me at a 9 % chance or so of winning, out of 12 participants, so…having better luck with the dad thing so far.

                      (Not that it’s as easy to quantify.)

                      1. 2

                        There’s one thing that isn’t mentioned often. It’s a fundamental difference that gives rise to many subsequent differences: we build software, and the building blocks we use are software. Our field is inherently recursive. We stick together smaller pieces of software to make bigger pieces of software.

                        A construction engineer does not build houses out of many little houses, themselves built out tiny houses. We do that. It makes things complex in a different way than otherwise.

                        1. 2

                          This is not what the question asked, but since details of the question reveal some gaps in knowledge about load testing, I’ll take a moment to address one issue:

                          Knowledge of how to properly measure and test load is way more important than choice of tooling. Gil Tene talks about this in an approachable way. A good start might be the YouTube video that got me down the rabbit hole a long time ago: https://m.youtube.com/watch?v=lJ8ydIuPFeU

                          1. 1

                            Really good content in that, in general!

                            I have some thoughts on the Coordinated Omission problem, but I moved them to your comment above.

                          1. 17

                            I’ve used ab, siege, jmeter and wrk in the past. I think wrk was the best (for quick stuff and heavy load) unless you actually want to set elaborate test setups of complete workflows incl login, doing stuff, logout, etc.pp - then jmeter.

                            NB: I’ve not done web development full time for many years, so maybe there’s new stuff around.

                            NB2: Always measure from a second host, a good stress testing tool will destroy the perf of the host it is running on.

                            1. 2

                              Do you know if wrk have fixed their coordinated omission problem or should one still use wrk2 to get correct measurements?

                              1.  

                                I’ve always used the (unmaintained-looking) wrk2 instead of wrk, precisely because of C.O. Nothing I could see in the wrk readme suggested they merged the C.O. work back in..

                                1. 1

                                  No, as I said it’s been a while and I don’t remember this term or type of problem, sorry. It worked for me at the time.

                              1. 5

                                This is a good post, but it’s somewhat missing in clarity of thought – I suspect because it tries to cast failure analysis in a framework of counterfactuals and causality.

                                I prefer to instead view failure as a system that for some reason went outside the constraints placed on it. It can do this for primarily three reasons: a sub-system went out of constraints (disk full), a control mechanism was insufficient (disk space freed too slowly) or a control mechanism was missing in the first place (disk space not automatically freed.) Of course, the reasons for control mechanism failure is often failure of control mechanisms higher up in the system hierarchy.

                                I’m not yet very good at describing this view, which is known as the System theoretic accident model. It also emphasises how we shouldn’t stop at a fixable cause, but rather explore the causal links as far as economy allows us, because the more leveraged fixes tend to be deeper. (As an example in my company, a simple hardware failure led us to realise we were missing an entire department in our organisation, that we would need to prevent failures like this one – and many, many others – in the future.)

                                I strongly recommend the CAST handbook, which is available as a free PDF and was my intro to this way of looking at things. It presents many of the same things as this article, except in my opinion with a much more clear conceptual framework. It also goes much deeper.

                                1. 3

                                  Hey thanks for that. Next time I’ll be sure to write a full-length monograph so I can hit every aspect of a topic.

                                  (OK, I’ll set aside the sarcasm. For now.)

                                  Look, when writing a blog, one always has to pick one thing to discuss in one post. Safety and system failure is a vast topic. Most readers, and from what I’ve seen most people in our industry, don’t know about any of it. Instead, they are shown examples of behavior in their companies. The typical company today still believes in human error (e.g. HBO Max throwing an intern under the bus last week), root cause analysis, and situational awareness. Absent external influences, people duplicate the practices they see.

                                  My post was written to address a single, very specific pattern of behavior I see as unhelpful.

                                1. 14

                                  REPLs are nice but they work well only for reasonably isolated code with few dependencies.

                                  Which is exactly the reason to have all your code broken up in this way. Even if you don’t use REPL.

                                  It’s hard to set up a complex object to pass into a function.

                                  Avoid complex objects then. Everything wrapped in bespoke objects with methods that each don’t even need all of the object’s state is one of the most prominent things that’s wrong about OO.

                                  1. 18

                                    “Avoid complex objects” is a good rule of thumb to keep code clean, but if you’re modeling a complex real-world concept, the complexity needs to be reflected in your model somewhere or it will be an incorrect representation.

                                    1. 13

                                      Ah-hah! A friend of mine once put it very nicely when, in a fit of rage caused by trying to refactor a 4,000-line drive, he just yelled fsck it, this DMA controller here is super murky so of course my driver for it is going to be super murky, sillicon don’t care I gotta write clean code.

                                      Not all things admit a “beautiful” model. A set of tools that only works with things that admit a beautiful model makes for great blog posts but lots and lots of frustration when they meet the messy real world of here are the 14 steps you have to perform in order to get a byte off this bus, yes the writes at steps 6, 7 and 10 are at aliased, unaligned memory locations, why do you ask?

                                      1. 1

                                        But isn’t this what abstraction… does? We have an ugly inconsistent machine that doesn’t solve our problem very elegantly (hardware of some sort) so we sandwich in other, abstract machines of higher and higher level of abstraction until the thing we want to express almost falls right out of just asking the question of the highest level?

                                        (And each level abstract machine does mask over a few of the most glaring problems of the machine just underneath it, so it’s low-dependency and trivial to test and reason about.)

                                        1. 2

                                          When the underlying hardware is an ugly, inconsistent machine, the first layer of abstract machines will have to mimic its model, you don’t really have a choice about that. If it consists of 12 non-consecutive memory-mapped registers that can be written independently, but not concurrently, then you’ll probably have 12 pointers and a mutex (at best – sometimes you’ll need 12…) whether you like it or not, because ultimately data will have to be written in memory. If getting a byte off a bus requires you to do fifteen ugly steps, then your code will have to do them.

                                          Now, in addition to that, you can sometimes wrap this is a unified, ellegant interface. That won’t save the first layer of code but it will at least allow third parties to use it without having to deal with its peculiarities.

                                          If you poke around Linux’ driver tree (I’m mentioning it just because it’s probably the one that it’s easiest to find docs about, it’s not necessarily my favourite), you’ll find plenty of cases where the driver authors had to go to incredible lengths, and sometimes do a bunch of horrendous hacks, in order to get the driver to conform to the unified interface while still keeping the hardware happy.

                                          But even that doesn’t always work. Abstractions inevitably leak, for example – you can’t augment the “unified” interface to account for every hardware quirk in existence, otherwise you end up with an interface so large that nobody implements it correctly. That’s true outside the world of device drivers, too, I’ve seen plenty of codebases where people really wanted ellegant, unified interfaces, and they ended up with 120-function interfaces to “middleware” interfaces that plugged to the real interfaces of the underlying implementation. Most programmers, including yours truly, just aren’t smart enough to navigate the innards of that kind of a behemoth, which is why it looks great in the docs but it’s a bug-ridden piece of junk if you try to run it.

                                          Speaking strictly for device drivers, you also can’t pile up abstractions forever, at some point you have to stop at the unified API everyone expects. E.g. if you pick ten devices from the same family, you could probably come up with a higher-level, more ellegant abstraction than the one their device drivers implement on a given platform, but it would be completely useless because that’s not the interface the next layer in the kernel expects. Sometimes it’s literally impossible to change the next layer (e.g. on Windows, unless you’re Microsoft), but even if it’s technically possible, that’s a whole other can of worms.

                                    2. 3

                                      To clarify, I do still think the described tooling would be useful.

                                    1. 3

                                      I think the “AI” point was dismissed way too quickly. We’re already seeing hugely impressive programs (in image recognition and elsewhere) based on incomplete input–output pairs, i.e. they do the right thing even when presented with an input it has not seen before. Why not use this for code, once we can?

                                      I know these algorithms are fallible, but perfection is not the target. Beating the human is.

                                      Edit: to br clear, I’m also not saying this will give better results than conventional programming, only that it will be so much cheaper to produce so that a big number of businesses will accept the inaccuracies.

                                      1. 3

                                        I’m missing Lehman’s laws of software evolution, at least the first two:

                                        (1974) “Continuing Change” — a system must be continually adapted or it becomes progressively less satisfactor.

                                        (1974) “Increasing Complexity” — as a system evolves, its complexity increases unless work is done to maintain or reduce it.

                                        1. 1

                                          re: conway’s law

                                          i think you can work around this to some extent:

                                          • have more components than people (an order of magnitude or so?)
                                          • avoid “utils” and other catch-all dumping grounds
                                          • have your components be searchable

                                          you’re still going to get libraries/tools/etc. that follow org structure, but you can get a lot outside of that structure too, and that’ll be more reusable

                                          1. 2

                                            Why work around it? Isn’t the purpose of Conway’s Law to accept the inevitable rather than fight it?

                                            FWIW I’ve worked in an environment that follows your suggestions and it still followed Conway’s Law.

                                            1. 1

                                              Yes. This goes beyond software, too. The way to exploit Conway’s law is to shape the organisation after the system you desire to build. This implies things like smaller, cross-functional teams* with greater responsibility (in order to get less coupling between system components). That way you maximise communication efficiency.

                                              * Lean people would advocate that the team should be co-located too. The idealist in me still clings to the Microsoft research that showed org chart distance mattered orders of magnitude more than physical distance.

                                          1. 1

                                            I find that naming things is hard when I don’t (yet) understand what I’m building - but easy when I do.

                                            Does this mean that domain understanding is hard and the difficulty with naming things is a side effect?

                                            1. 2

                                              That’s one way to interpret the evidence. Another is that the difficulty of naming things doesn’t change, only your perception of it. In other words, when you know the domain well, you’re good at fooling yourself that you’ve come up with great names, when in fact they may be no better than before.

                                              Not suggesting anything either way, just dropping in with a reminder of how not receiving a signal is not evidence of there being no signal.

                                            1. 3

                                              I get the feeling that this is a really good way to describe to crowds of non-computer people what it is I do all day and why it’s so hard. Sure, there are many other aspects of the job, but this is one that’s not talked about a lot in the mainstream, yet should be easy enough to understand. And the ideas of hair-splitting generalise well to some other aspects of the craft.

                                              1. 5

                                                This is exactly what used to be known as modularity. Module boundaries can absolutely cross technologies.

                                                For the classic reads on the subject, check out David Parnas’ papers on modularity and designing for extension and contraction (the original names for these things.)

                                                1. 2

                                                  “Designing Software for Extension and Contraction” is awesome, I’ve read it!

                                                1. 2

                                                  This is a pet peeve of mine, but any prediction/forecast like this is useless without sufficient detail to make it falsifiable. What does one mean by “the year of the ARM workstation”? What market share? In which countries? What does “TypeScript broadly seen as an essential tool” mean? What evidence would one need to know that turned out to be false?

                                                  Vague predictions without sufficient detail to be falsifiable tends to boil down to various fancy ways of saying “popular technology will keep being popular”. By putting it in concrete terms you force yourself to be honest.

                                                  Additionally it’s a huge difference between 95 % and 55 % confidence. I’d recommend anyone forecasting tech here attach a confidence level to your prediction. That way you can be scored and we can get an aggregate crustacean prediction skill score too!

                                                  1. 5

                                                    It seems like they tried to coordinate the size reduction from the top, which is hard for many reasons, not least that size problems – like performance problems – can be characterised as death by a thousand cuts.

                                                    I’ve read about an alternative approach to deal with that problem, which is to find out how much an oversize app costs (which they did) and then translate that down to an optimal price point of $/kB.

                                                    Let every team sell size reductions on their little slice of functionality at that price, and you have everyone incentivised to reduce size in whichever way they reasonably can without accidentally incurring greater costs than just shipping the oversize application would result in. The localised decision making will make much better choices on what to optimise than any global control could. Teams that build large functionality will have better opportunities to reduce their size by a fraction than teams building tiny features. Just as it should be!

                                                    1. 1

                                                      Interesting approach. I wonder: is this much different from running perf stat to find out the number of instructions executed (and other cache-related metrics, I suppose) by a running process?

                                                      1. 2

                                                        If you’re using consistent hardware then yeah, that kinda works (you may end up sharing caches with other running processes…).

                                                        In a series of runs on different cloud VMs, though, while instructions will probably (at a guess, untested) be fairly consistent, cache misses will be less consistent:

                                                        Cache sizes may be inconsistent across VM instances.

                                                        GitHub Actions says it uses Standard_DS2_v2 Azure instances, and if you look at Azure docs (https://docs.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series?toc=/azure/virtual-machines/linux/toc.json&bc=/azure/virtual-machines/linux/breadcrumb/toc.json) it seems this involves 4 different CPU models, across 4 generations of Intel generations, with differing L2 and L3 cache sizes.

                                                        You’re sharing hardware with other VMs, which can impact the processor’s caches, in particular L3.

                                                        Found this empirical research where they tried to measure cache sizes on cloud instances over time, and it’s noisy, especially for L3: http://www.cs.cornell.edu/courses/cs5412/2018sp/recitation_slides/cs5412-cache-weijia.pdf (slide 22). Seems like if it was just about CPU models you wouldn’t see that level of variation.

                                                        (I’ll update the article with this info at some point).

                                                        1. 1

                                                          Oh, this is very Interesting and relevant to a future project of mine. Thank you for doing the deep digging. I didn’t expect such a detailed response!

                                                      1. 5

                                                        I write scripts both in Python and Bash/ZSH but I feel like Perl 5 would be the sweet spot between those two languages for sys admin stuff.

                                                        I don’t know if Perl 6 serves that spot as well, but it may be an interesting choice for sys admins if it does.

                                                        1. 6

                                                          I think Perl 6 serves that spot well in many senses except one: it doesn’t come preinstalled with virtually every real operating system.

                                                          1. 6

                                                            Perl 6 has been renamed to Raku: https://www.raku.org/