Threads for jonahx

  1. 4

    Does anyone have examples of the kind of Toyota-worshipping devops discussion the post is complaining about?

    I’ve read a lot of material about agile methodologies in software, and aside from books and articles that are specifically focused on the historical roots of agile practices, I don’t think I’ve ever seen one that lingers on the specifics of Toyota’s processes for long enough to warrant a reaction this strong. Usually Toyota gets a paragraph or two at most before the author moves on, and it’s often just a sentence or two.

    1. 7

      You will find much of buried in corporate training material. To this day I hate Toyota cars for no reason other than that their stupid logo was practically on the first slide of every agile course I had no choice but to attend gladly attended of my own volition between 2009 and 2014 or so.

      1. 2

        Ah, that explains it. My only exposure to corporate training material about agile processes has been in the form of writing it as part of dev onboarding documentation.

      2. 1

        I hear it mentioned in many of the devops books, articles, blogs, videos I’ve consumed. I can’t give you an exhaustive list but it’s mentioned for sure in The DevOps Handbook, and (I think) Jez Humble’s other books, which are probably the most popular in the field. Here’s another blogger who I’ve heard talk about it in multiple articles/reviews.

        Toyota love in devops is definitely a real phenomenon.

      1. 6

        I mean, what human can interpret *++argv?!

        I now believe the reason is fairly simple: in essence, OpenBSD’s style squeezes more code onto the screen.

        I don’t see how one can come to such a conclusion based on the echo source code shown as an example: there are blank lines separating logically related lines of code, the variable name (nflag) could have easily been called just n or perhaps nf (n is traditionally used for size/length/count), etc. Overall, I found the code quite easy to follow and there is definitely much denser code out there where you will hardly see a blank line.

        But I think the first quote hints at the real reason (IMO, anyway): the example is easy to follow because it doesn’t shy away from idioms that pack a lot of meaning into very little code.

        1. 1

          J/APL, and their inspired golfing languages come to mind when thinking of the upper limits of terseness.

          1. 9

            I was researching array languages recently and came across this interesting quote in the README of an open-source K implementation:

            A note on the unusual style of C code: It attempts to replicate the style of Arthur Whitney. A striking original example is contained in file There are 2 versions of the buddy memory allocation system. The first is in 11 lines written by Whitney. The second is in well documented traditional C (almost 750 lines).

            The example is, indeed, striking.

            I MZ[31]={1};Z I *MM[31];mi(){MZ[7]=MZ[13]=MZ[19]=MZ[25]=2;DO(30,MZ[i+1]+=MZ[i]*2)}
            Z mmr(n,i){if(i<18)i=18;R err(2,n),tmp((MZ[i]+2)<<2),1;} /* Dan MZ[i+1]? */
            C *mab(m)unsigned m;{I *p,*r,i=2,n=m;for(n=(n+3)>>4;n;n>>=1)++i;
             do{if(p=MM[i])R MM[i]=(I*)*p,(C*)p;for(n=i;n<30;)if(p=MM[++n]){
              if(mc()>=i)continue;} while(mmr(m,i));}
            I *ma(m){R(I*)mab(m<<2);}
            mf(p)I *p;{I i=p[-1];*p=(I)MM[i],MM[i]=p;}
            mb(p,n)I *p;{I i=31,j;for(n-=2,++p;i--;)if(j=MZ[i],j<=n)n-=j,*p=i,mf(p+1),p+=j;}
            mc(){R 0;}
            I *mz(){Z I b[31];I *p;DO(31,for(b[i]=0,p=MM[i];p;p=(I*)*p)++b[i])R b;}

            Although the rest of the code – the Whitney-inspired code – is equally baffling to my eyes.

            I suppose that, in order to invent or implement K, you have to place a pretty high value on terseness.

            1. 3

              This is a rather famous block of code, but each time I see it I feel further away from any precise conclusions about it. It seems clear that the authors of J/K/etc. are comfortable with this style and our ability or inability to read it is pretty irrelevant since we’re not likely to find ourselves maintaining it or patching it.

              The whole APL family sets up different priorities than the rest of computing. For instance they like “idioms” more than functional abstraction, because apparently it’s both difficult to name some of the idioms in a useful way, and when you do, the name tends to be larger than the idiom itself. Also, at least J (but I believe most APL family languages) makes liberal use of “special code” that notices certain idioms and has optimized code to handle those cases.

              I have come to see it as a completely different coding “civilization,” and I won’t pass judgement on it because I have seen how much work it would take to reorient myself to it, and I just won’t ever find the time to do so.

              1. 1

                Most programming languages have idioms. And apl does not lack functional abstractions. Special code seems rather similar to the open-coded algorithms based on simple graphical pattern matching which can be found in most compilers.

            2. 3

              I came in to say….

              Gradually I started to realise that not only could I cope with OpenBSD’s terse code style, but it was actually easier for me to read it than code I’d written myself.

              If you want the best teacher for this lesson, learn APL or J or K or BQN, and keep learning until that code no longer seems dense, but simply “just the right amount to express what I want”.

          1. 9

            This idea is very compelling.

            The one question I had that wasn’t answered in the video is: If the network is cleanly abstracted away, and not even visible in the code, how is network error handling achieved?

            1. 1

              How are errors usually handled in clojure?

              I’d imagine that (p/server foo) returns the equivalent of (using a rust like syntax) Result<type_of(foo), NetworkError>. At least that’s how I’d design this if I was designing this.

            1. 5

              That’s nice and all, but ultimately gives me an even bigger impression that if I want to stay away from a service playing Man-In-The-Middle (cloudflare), I’m more or less a roadkill for those guys.

              1. 18

                Honestly, that’s how it’s always been. You have bigger pipes than the DDOSer or you pound sand and fall over. There’s no good way around it.

                Welcome to the modern internet, hoping you won’t get DDOSed is not a scalable strategy to prevent DDOS attacks. I personally only really use cloudflare defensively, because my site had been knocked over entirely with DDOS attacks and I don’t want to deal with that headache again.

                1. 4

                  hoping you won’t get DDOSed is not a scalable strategy

                  It worked so far, mostly, at least for years ;) But I also didn’t launch anything public really, and I’m at the point of changing that, which gives me nightmares. (And I really, really dread having one service taking the (TLS) keys to my kingdom..)

                  P.S. Yes, you’re right..

                  1. 2

                    I personally only really use cloudflare defensively

                    Could you elaborate on how you use them?

                    We looked into them at my company, and I even had a call with their sales guys, but it was impossible to get useful information out of them. They basically wanted 6k / mo, or else just use the free tier…. and everything was business jargon.

                    1. 9

                      Sure. I am a person that likes to write about tech on their blog. I am also not a white man that is attracted to women, which unfortunately makes me a target for harassment, hate and apparently (as in this is something I have directly observed through my personal experience in this hellplane) getting biblical amounts of traffic sent at me with little warning. I use Cloudflare defensively because they are not likely going to take Cloudflare down with the amount of traffic that the kids that apparently decide that me being honest about observable facts about how I experience reality means that my site turns into a competition of how much traffic can booters and DDoS for hire services can throw at it.

                      By using Cloudflare, which sits between the public internet and my blog server, the attackers will target cloudflare, not my blog server. This makes their attempted attacks do nothing. Combined with the ridiculous levels of engineering I have done to make my blog as fast as possible, I am fairly confident that my site can weather most DDoS attacks that will get thrown my way. The other part of this puzzle is to avoid writing things that are too inflammatory so that I am not an enticing target, but that is different and is more of a deterrence strategy than an outright technical thing.

                      Cloudflare is part of my strategy to protect my website against random DDoS attacks. I really wish I didn’t have to use them, but overall I think that the net good outweighs the complications that I take doing this kind of stuff. Definitely worth the $20 a month I pay for the service, if only for the peace of mind.

                      1. 1

                        Thanks for the reply. That is a shitty situation. If I want to setup similar protection, are there specific technical options you use? Addons? Configuration? Etc…

                        Or does just the standard service automatically give you the DDoS protection?

                        1. 3

                          I can check, but other than telling it “holy crap cache everything as much as you can” I’m pretty sure it’s fairly vanilla.

                  2. 5

                    Keep in mind that cloudflare is not the only game in town. Sure, it’s all effectively MitM, but there are other valid options from not-as-evil companies. Or as reezer mentioned, you can rely on your provider to some extent.

                    1. 6

                      Do you have any good sources talking about Cloudflare’s evilness? The amount of power they have is obviously absurd but I haven’t found much info about them actually abusing it.

                      1. 7

                        IIRC the loudest voices RE cloudflare object to their free speech absolutism & handling of abuse complaints - e.g. when they received complaints about far-right sites selling pro-insurrection merchandise, nazi memorabilia, etc they A) kept serving them at a profit, and B) took personal details including address from the complainant and passed it on to the site owners.

                        While free speech absolutism has those for and against it, I’m sure you can imagine why someone might object to having their home address passed to a white nationalist group as “an activist against white nationalism with an asian name lives at this address”.

                        1. 3

                          For me this is where they reached the comic book villain level of evil Since then I’m actively encouraging people to read it and use some other service instead.

                      2. 3

                        A lot can already be done with a haproxy<=>varnish sandwich¹ on a €30 dedicated machine.² But this not cool anymore, the new coolness is man-in-the-middle with vendor-lock-in-freemium and unreliability.

                        What can I say, I’m an old man, now… :)

                        ². I dont want to advertise for a German provider which starts by “Hetz” and finishes by “ner” or a the semi-cheap branch of a french provider which name asks “so, you start?” There are many others, like one formerly known as “dedibox”, etc…

                        1. 12

                          You seem to dismiss the new MitM vendors, but anything you deploy to your dedicated machine has two basic issues:

                          1. You can get DoS-ed by anyone with a similar pipe to yours. It makes no difference what you deploy on your machine - if your network interface is flooded with empty packets, that’s all you’ll see. So anyone with a €30 dedicated machine can kill your service.

                          2. (mentioned in the post) - If you’re a popular DoS target, your service provider may not like to be constantly flooded by your traffic and will drop you. This has happened to many people already and there’s nothing you can do about it, unless you explicitly pay them for being your DoS shield.

                          1. 4

                            You’re right. At the end if they want to DDoS, they will succeed unless you use these special companies (~= Cloudflare, Akamai, Fastly, …)

                            But in the case of the author, network was not the limiting factor, he got properly DoSed (not DDoS) because his web app had slow calls (= high CPU usage), and didn’t limit incoming connections. HAProxy (= limiting connection) and Varnish (= caching page) will go a long way before the network becomes a limit. And when the network starts becoming a limit these €30 dedicated machine providers (at least Hetzner, OVH/SoYouStart, Scaleway, …) have DDoS protections.

                            In the case of the author, HAProxy and Varnish would have done a lot already.

                        2. 1

                          Most bigger providers, be it Hetzner, OVH or others give you quite good DDOS protection for free.

                          Also with the numbers shown in the article one really shouldn’t underestimate what resources are available for relatively cheap these days.

                          And using Cloudflare can also mean that these “How X nearly caused the internet to go down” and similar turning downtime into marketing articles on the Cloudflare blog probably affect you, whereas you are fine otherwise. Don’t get me wrong, they are sometimes pretty informative, yet it’s something that one should consider.

                          So it’s all a bit context dependent. Depending on what service you are hosting attacks might also look differently and individual solutions might or might not work. DOS can often be achieved in more effective ways than just hitting the landing page, that usually is quite static in first place many times. And if you have something that rapidly fills up databases or causes compute resources to be used recovering might be a lot harder. Of course dumb attack are the most frequent ones, but usually also the ones having the smallest effect.

                          In short what I wanna say is that if you blindly use cloudflare (or any others) that form or that websocket thingy might still knock your service out, with a lot less resources on the attackers side.

                          1. 4

                            Hetzner’s DDoS protection is not very effective. Sure, it’s best than other providers such as Contabo (another German provider) where their “DDoS protection” is a free nullroute, but nothing’s compared to OVH’s protection. And then, they (at least happened before, not sure nowadays) didn’t protect any traffic coming from inside their network - so have fun dealing with OVH boxes DDoSing other OVH boxes.

                            1. 1

                              I’d argue that dropping traffic early from your provider is a lot easier to deal than traffic from random places on the internet. Don’t they even have a firewall thingy, so you could block that even before touching any local packet filter.

                            2. 3

                              They don’t give you DDOS protection, they just don’t boot you for having higher traffic (as they did ~10 years ago, trust me). And they are so friendly to nullroute that traffic. It does not help with the load on your side.

                              1. 1

                                Again, depending on the kind of attack and the service the same is true for CloudFlare, etc. Sure, if your problem is someone GETing / then pretty much using a CDN will fix your problem. I just don’t think the sentiment that it’s CloudFlare or not is the right one. It’s not a 0 and 1 thing. Especially not for the particular case discussed here.

                          1. 7

                            All I had to do was submit a GDPR data request, wait a few days for the cloud to think and then I got a 3 gigabyte zip file full of everything I’ve ever tweeted. Cool!

                            In case anyone else is looking to get a copy of their Twitter data you don’t need to submit a GDPR data request. You can request a data export right from your profile:

                            I do this occasionally so I have a copy of my own tweets that’s easy to ripgrep through to find stuff I liked in the past (as I don’t believe there’s a way to search your likes on Twitter).

                            1. 2

                              I would like to do the same thing but out of iPhone Messages app, and without a 3rd party app.

                            1. 40

                              I tried out this language while it was in early development, writing some of the standard library (hash::crc* and unix::tty::*) to test the language. I wrote about this experience, in a somewhat haphazard way. (Note, that blog post is outdated and not all my opinions are the same. I’ll be trying to take a second look at Hare in the coming days.)

                              In general, I feel like Hare just ends up being a Zig without comptime, or a Go without interfaces, generics, GC, or runtime. I really hate to say this about a project where they authors have put in such a huge amount of effort over the past year or so, but I just don’t see its niche – the lack of generics mean I’d always use Zig or Rust instead of Hare or C. It really looks like Drew looked at Zig, said “too bloated”, and set out to create his own version.

                              Another thing I find strange: why are you choosing to not support Windows and macOS? Especially since, you know, one of C’s good points is that there’s a compiler for every platform and architecture combination on earth?

                              That said, this language is still in its infancy, so maybe as time goes and the language finds more users we’ll see more use-cases for Hare.

                              In any case: good luck, Drew! Cheers!

                              1. 10

                                why are you choosing to not support Windows and macOS?

                                DdV’s answer on HN:

                                We don’t want to help non-FOSS OSes.

                                (Paraphrasing a lot, obvs.)

                                My personal 2c:

                                Some of the nastier weirdnesses in Go are because Go supports Windows and Windows is profoundly un-xNix-like. Supporting Windows distorted Go severely.

                                1. 13

                                  Some of the nastier weirdnesses in Go are because Go supports Windows and Windows is profoundly un-xNix-like. Supporting Windows distorted Go severely.

                                  I think that’s the consequence of not planning for Windows support in the first place. Rust’s standard library was built without the assumption of an underlying Unix-like system, and it provides good abstractions as a result.

                                  1. 5

                                    Amos talks about that here: Go’s file APIs assume a Unix filesystem. Windows support was kludged in later.

                                  2. 5

                                    Windows and Mac/iOS don’t need help from new languages; it’s rather the other way around. Getting people to try a new language is pretty hard, let alone getting them to build real software in it. If the language deliberately won’t let them target three of the most widely used operating systems, I’d say it’s shooting itself in the foot, if not in the head.

                                    (There are other seemingly perverse decisions too. 80-character lines and 8-character indentation? Manual allocation with no cleanup beyond a general-purpose “defer” statement? I must not be the target audience for this language, is the nicest response I have.)

                                    1. 2

                                      Just for clarity, it’s not my argument. I was just trying to précis DdV’s.

                                      I am not sure I agree, but then again…

                                      I am not sure that I see the need for yet another C-replacement. Weren’t Limbo, D, Go, & Rust all attempts at this?

                                      But that aside: there are a lot of OSes out there that are profoundly un-Unix-like. Windows is actually quite close, compared to, say, Oberon or classic MacOS or Z/OS or OpenVMS or Netware or OS/2 or iTron or OpenGenera or [cont’d p94].

                                      There is a lot of diversity out there that gets ignored if it doesn’t have millions of users.

                                      Confining oneself to just OSes in the same immediate family seems reasonable and prudent to me.

                                  3. 10

                                    My understanding is that the lack of generics and comptime is exactly the differentiating factor here – the project aims at simplicity, and generics/compile time evaluations are enormous cost centers in terms of complexity.

                                    1. 20

                                      You could say that generics and macros are complex, relative to the functionality they offer.

                                      But I would put comptime in a different category – it’s reducing complexity by providing a single, more powerful mechanism. Without something like comptime, IMO static languages lose significant productivity / power compared to a dynamic language.

                                      You might be thinking about things from the tooling perspective, in which case both features are complex (and probably comptime even more because it’s creating impossible/undecidable problems). But in terms of the language I’d say that there is a big difference between the two.

                                      I think a language like Hare will end up pushing that complexity out to the tooling. I guess it’s like Go where they have go generate and relatively verbose code.

                                      1. 3

                                        Yup, agree that zig-style seamless comptime might be a great user-facing complexity reducer.

                                        1. 16

                                          I’m not being Zig-specific when I say that, by definition, comptime cannot introduce user-facing complexity. Unlike other attributes, comptime only exists during a specific phase of compiler execution; it’s not present during runtime. Like a static type declaration, comptime creates a second program executed by the compiler, and this second program does inform the first program’s runtime, but it is handled entirely by the compiler. Unlike a static type declaration, the user uses exactly the same expression language for comptime and runtime.

                                          If we think of metaprogramming as inherent complexity, rather than incidental complexity, then an optimizing compiler already performs compile-time execution of input programs. What comptime offers is not additional complexity, but additional control over complexity which is already present.

                                          To put all of this in a non-Zig context, languages like Python allow for arbitrary code execution during module loading, including compile-time metaprogramming. Some folks argue that this introduces complexity. But the complexity of the Python runtime is there regardless of whether modules get an extra code-execution phase; the extra phase provides expressive power for users, not new complexity.

                                          1. 8

                                            Yeah, but I feel like this isn’t what people usually mean when they say some feature “increases complexity.”

                                            I think they mean something like: Now I must know more to navigate this world. There will be, on average, a wider array of common usage patterns that I will have to understand. You can say that the complexity was already there anyway, but if, in practice, is was usually hidden, and now it’s not, doesn’t that matter?

                                            then an optimizing compiler already performs compile-time execution of input programs.

                                            As a concrete example, I don’t have to know about a new keyword or what it means when the optimizing compiler does its thing.

                                            1. 2

                                              A case can be made that this definition of complexity is a “good thing” to improve code quality / “matters”:

                                              Similar arguments can be used for undefined behavior (UB) as it changes how you navigate a language’s world. But for many programmers, it can be usually hidden by code seemingly working in practice (i.e. not hitting race conditions, not hitting unreachable paths for common input, updating compilers, etc.). I’d argue that this still matters (enough to introduce tooling like UBSan, ASan, and TSan at least).

                                              The UB is already there, both for correct and incorrect programs. Providing tools to interact with it (i.e. __builtin_unreachable -> comptime) as well as explicit ways to do what you want correctly (i.e. __builtin_add_overflow -> comptime specific lang constructs interacted with using normal code e.g. for vs inline for) would still be described as “increases complexity” under this model which is unfortunate.

                                              1. 1

                                                The UB is already there, both for correct and incorrect programs.

                                                Unless one is purposefully using a specific compiler (or set thereof), that actually defines the behaviour the standard didn’t, then the program is incorrect. That it just happens to generate correct object code with this particular version of that particular compiler on those particular platforms is just dumb luck.

                                                Thus, I’d argue that tools like MSan, ASan, and UBSan don’t introduce any complexity at all. The just revealed the complexity of UB that was already there, and they do so reliably enough that they actually relieve me of some of the mental burden I previously had to shoulder.

                                            2. 5

                                              languages like Python allow for arbitrary code execution during module loading, including compile-time metaprogramming.

                                              Python doesn’t allow compile-time metaprogramming for any reasonable definition of the word. Everything happens and is introspectable at runtime, which allows you to do similar things, but it’s not compile-time metaprogramming.

                                              One way to see this is that sys.argv is always available when executing Python code. (Python “compiles” byte code, but that’s an implementation detail unrelated to the semantics of the language.)

                                              On the other hand, Zig and RPython are staged. There is one stage that does not have access to argv (compile time), and another one that does (runtime).

                                              Related to the comment about RPython I linked here:



                                              1. 4

                                                Yours is a rather unconventional definition of complexity.

                                                1. 5

                                                  I am following the classic paper, “Out of the Tar Pit”, which in turn follows Brooks. In “Abstractive Power”, Shutt distinguishes complexity from expressiveness and abstractedness while relating all three.

                                                  We could always simply go back to computational complexity, but that doesn’t capture the usage in this thread. Edit for clarity: Computational complexity is a property of problems and algorithms, not a property of languages nor programming systems.

                                                  1. 3

                                                    Good faith question: I just skimmed the first ~10 pages of “Out of the Tar Pit” again, but was unable to find the definition that you allude to, which would exclude things like the comptime keyword from the meaning of “complexity”. Can you point me to it or otherwise clarify?

                                                    1. 4

                                                      Sure. I’m being explicit for posterity, but I’m not trying to be rude in my reply. First, the relevant parts of the paper; then, the relevance to comptime.

                                                      On p1, complexity is defined as the tendency of “large systems [to be] hard to understand”. Unpacking their em-dash and subjecting “large” to the heap paradox, we might imagine that complexity is the amount of information (bits) required to describe a system in full detail, with larger systems requiring more information. (I don’t really know what “understanding” is, so I’m not quite happy with “hard to understand” as a concrete definition.) Maybe we should call this “Brooks complexity”.

                                                      On p6, state is a cause of complexity. But comptime does not introduce state compared to an equivalent non-staged approach. On p8, control-flow is a cause of complexity. But comptime does not introduce new control-flow constructs. One could argue that comptime requires extra attention to order of evaluation, but again, an equivalent program would have the same order of evaluation at runtime.

                                                      On p10, “sheer code volume” is a cause of complexity, and on this point, I fully admit that I was in error; comptime is a long symbol, adding size to source code. In this particular sense, comptime adds Brooks complexity.

                                                      Finally, on a tangent to the top-level article, p12 explains that “power corrupts”:

                                                      [I]n the absence of language-enforced guarantees (…) mistakes (and abuses) will happen. This is the reason that garbage collection is good — the power of manual memory management is removed. … The bottom line is that the more powerful a language (i.e. the more that is possible within the language), the harder it is to understand systems constructed in it.

                                                      comptime and similar metaprogramming tools don’t make anything newly possible. It’s an annotation to the compiler to emit specialized code for the same computational result. As such, they arguably don’t add Brooks complexity. I think that this argument also works for inline, but not for @compileError.

                                          2. 18

                                            My understanding is that the lack of generics and comptime is exactly the differentiating factor here – the project aims at simplicity, and generics/compile time evaluations are enormous cost centers in terms of complexity.

                                            Yeah, I can see that. But under what conditions would I care how small, big, or icecream-covered the compiler is? Building/bootstrapping for a new platform is a one-time thing, but writing code in the language isn’t. I want the language to make it as easy as possible on me when I’m using it, and omitting features that were around since the 1990’s isn’t helping.

                                            1. 8

                                              Depends on your values! I personally see how, eg, generics entice users to write overly complicated code which I then have to deal with as a consumer of libraries. I am not sure that not having generics solves this problem, but I am fairly certain that the problem exists, and that some kind of solution would be helpful!

                                              1. 3

                                                In some situations, emitted code size matters a lot (and with generics, that can quickly grow out of hand without you realizing it).

                                                1. 13

                                                  In some situations

                                                  I see what you mean, but I think in those situations it’s not too hard to, you know, refrain from use generics. I see no reason to force all language users to not use that feature. Unless Hare is specifically aiming for that niche, which I don’t think it is.

                                                  1. 4

                                                    There are very few languages that let you switch between monomorphisation and dynamic dispatch as a compile-time flag, right? So if you have dependencies, you’ve already had the choice forced on you.

                                                    1. 6

                                                      If you don’t like how a library is implemented, then don’t use it.

                                                      1. 2

                                                        Ah, the illusion of choice.

                                              2. 10

                                                Where is the dividing line? What makes functions “not complex” but generics, which are literally functions evaluated at compile time, “complex”?

                                                1. 14

                                                  I don’t know where the line is, but I am pretty sure that this is past that :D


                                                  1. 17

                                                    Sure, that’s complicated. However:

                                                    1. that’s the inside of the inside of a library modeling a very complex domain. Complexity needs to live somewhere, and I am not convinced that complexity that is abstracted away and provides value is a bad thing, as much of the “let’s go back to simpler times” discourse seems to imply. I rather someone takes the time to solve something once, than me having to solve it every time, even if with simpler code.

                                                    2. Is this just complex, or is it actually doing more than the equivalent in other languages? Rust allows for expressing constraints that are not easily (or at all) expressable in other languages, and static types allow for expressing more constraints than dynamic types in general.

                                                    In sum, I’d reject a pull request with this type of code in an application, but don’t mind it at all in a library.

                                                    1. 4

                                                      that’s the inside of the inside of a library modeling a very complex domain. Complexity needs to live somewhere,

                                                      I find that’s rarely the case. It’s often possible to tweak the approach to a problem a little bit, in a way that allows you to simply omit huge swaths of complexity.

                                                      1. 3

                                                        Possible, yes. Often? Not convinced. Practical? I am willing to bet some money that no.

                                                        1. 7

                                                          I’ve done it repeatedly, as well as seeing others do it. Occasionally, though admittedly rarely, reducing the size of the codebase by an order of magnitude while increasing the number of features.

                                                          There’s a huge amount of code in most systems that’s dedicated to solving optional problems. Usually the unnecessary problems are imposed at the system design level, and changing the way the parts interface internally allows simple reuse of appropriate large-scale building blocks and subsystems, reduces the number of building blocks needed, and drops entire sections of translation and negotiation glue between layers.

                                                          Complexity rarely needs to be somewhere – and where it does need to be, it’s in often in the ad-hoc, problem-specific data structures that simplify the domain. A good data structure can act as a laplace transform for the entire problem space of a program, even if it takes a few thousand lines to implement. It lets you take the problem, transform it to a space where the problem is easy to solve, and put it back directly.

                                                    2. 7

                                                      You can write complex code in any language, with any language feature. The fact that someone has written complex code in Rust with its macros has no bearing on the feature itself.

                                                      1. 2

                                                        It’s the Rust culture that encourages things like this, not the fact that Rust has parametric polymorphism.

                                                        1. 14

                                                          I am not entirely convinced – to me, it seems there’s a high correlation between languages with parametric polymorphism and languages with culture for high-to-understand abstractions (Rust, C++, Scala, Haskell). Even in Java, parts that touch generics tend to require some mind-bending (producer extends consumer super).

                                                          I am curious how Go’s generic would turn out to be in practice!

                                                          1. 8

                                                            Obligatory reference for this: F# Designer Don Syme on the downsides of type-level programming

                                                            I don’t want F# to be the kind of language where the most empowered person in the discord chat is the category theorist.

                                                            It’s a good example of the culture and the language design being related.



                                                            which I linked here:

                                                  2. 3

                                                    In general, I feel like Hare just ends up being a Zig without comptime, or a Go without interfaces, generics, GC, or runtime. … I’d always use Zig or Rust instead of Hare or C.

                                                    What if you were on a platform unsupported by LLVM?

                                                    When I was trying out Plan 9, lack of LLVM support really hurt; a lot of good CLI tools these days are being written in Rust.

                                                    1. 15

                                                      Zig has rudimentary plan9 support, including a linker and native codegen (without LLVM). We’ll need more plan9 maintainers to step up if this is to become a robust target, but the groundwork has been laid.

                                                      Additionally, Zig has a C backend for those targets that only ship a proprietary C compiler fork and do not publish ISA details.

                                                      Finally, Zig has the ambitions to become the project that is forked and used as the proprietary compiler for esoteric systems. Although of course we would prefer for businesses to make their ISAs open source and publicly documented instead. Nevertheless, Zig’s MIT license does allow this use case.

                                                      1. 2

                                                        I’ll be damned! That’s super impressive. I’ll look into Zig some more next time I’m on Plan 9.

                                                      2. 5

                                                        I think that implies that your platform is essentially dead ( I would like to program my Amiga in Rust or Swift or Zig, too) or so off-mainstream (MVS comes to mind) that those tools wouldn’t serve any purpose anyway because they’re too alien).

                                                        1. 5

                                                          Amiga in Rust or Swift or Zig, too)

                                                          Good news: LLVM does support 68k, in part to many communities like the Amiga community. LLVM doesn’t like to include stuff unless there’s a sufficient maintainer base, so…

                                                          MVS comes to mind

                                                          Bad news: LLVM does support S/390. No idea if it’s just Linux or includes MVS.

                                                          1. 1

                                                            Good news: LLVM does support 68k Unfortunately, that doesn’t by itself mean that compilers (apart from clang) get ported, or that the platform gets added as part of a target triple. For instance, Plan 9 runs on platforms with LLVM support, yet isn’t supported by LLVM.

                                                            Bad news: LLVM does support S/390. I should have written VMS instead.

                                                            1. 1
                                                          2. 2

                                                            I won’t disagree with describing Plan 9 as off-mainstream ;) But I’d still like a console-based Signal client for that OS, and the best (only?) one I’ve found is written in Rust.

                                                      1. 23

                                                        Maintenance cost of bash scripts is lower

                                                        I would strongly disagree with this. Unless the script is very simple - only single commands, few variables - shell scripts can have many pitfalls that many people don’t realise until things go wrong. Pipelines and subshells and their side effects are not widely understood, for example.

                                                        c.f. ->

                                                        Don’t get me wrong, shell scripts do have their place, but for anything even vaguely complex they’re generally the wrong choice. Use the right tool for the job.

                                                        Oh and lastly:

                                                        Every machine has bash installed

                                                        As a BSD user, this is not true. :-)

                                                        1. 4

                                                          Unless the script is very simple

                                                          You can do a lot of useful stuff with simple bash scripts. I probably write one per week. Lets look at an example:


                                                          4 lines of code. Saves me a lot of hassle every time I need it.

                                                          Pipelines and subshells and their side effects are not widely understood

                                                          They can even be indeterministic:


                                                          1. 2

                                                            You can do a lot of useful stuff with simple bash scripts.

                                                            I’m not disputing that. I’ve been writing shell scripts for a very long time, and I use them where they’re appropriate. What I am disputing is your statement that the maintenance cost of shell scripts is lower compared to other languages. If you’re only ever writing simple scripts this is often true; but if you’re comparing shell to other languages there was probably a need to use the other language in the cases they were used.

                                                            They can even be indeterministic:

                                                            Indeed. You’re kind of making my argument for me. :-)

                                                            1. 5

                                                              With “maintenance cost” I do not mean the cost to change the functionality. I mean that from time to time you have to change your script because the language changes. I expect Bash to have less of these breaking changes than most other languages.

                                                              1. 3

                                                                from time to time you have to change your script because the language changes

                                                                What? I mean, there was python 2 to 3 and Ruby 1.8 to 1.9, but I don’t think breaking language changes are common?

                                                                1. 5

                                                                  Python in my opinion is an example for high maintenance cost. They sometimes do backwards-incompatible changes within a major version. For example Python 3.7 introduced “async” as a reserved keyword, breaking code that used “async” as a variable name.

                                                                  If you follow each version update of Python, you will probably recognize all breaking changes as deprecations in earlier versions. But if I would just write a script, leave it alone for 10 years and then try to run it with the latest Python version, I would not bet on it running without errors. Whereas for bash scripts I would assume they still work.

                                                                  But for bash scripts it totally depends on which programs you call.

                                                                  1. 1

                                                                    The needless breaking changes are my least favorite parts of Python and Node.

                                                            2. 2

                                                              You can put this directly in your .gitconfig. I have the following in mine:

                                                                # "commit fix <commit>" - Add index files to given commit without changing
                                                                # its message.  For more control, create a temp commit, then "rebase -i" and
                                                                # use fixup and edit (which lets you change commits msgs)
                                                                cfix =  "!f() { \
                                                                  local committofix=${1:-HEAD}; \
                                                                  git commit --fixup="$committofix"; \
                                                                  git rebase -i --autosquash "$committofix"^; \
                                                                }; f"
                                                            3. 0

                                                              Also NixOS.

                                                            1. 4

                                                              I don’t quite understand why people are interested in learning a language using spaced repetition and/or memorizing.

                                                              When you start working with a new language, you naturally adopt a spaced repetition-like approach to learn the constructs as you try to write code to accomplish tasks. Memorizing is never effective anyway, you won’t know where and how to use what you memorized.

                                                              I dunno maybe I’m the odd duck 🤷‍♂️

                                                              1. 9

                                                                I initially did this mainly for the less-used functions.

                                                                I used to pair-program alongside — one of the inital contributors to Rails — an absolute master of Ruby, and a really inspiring programmer to watch.

                                                                I was blown away by how he seemed to know the entire Ruby standard library, and have it all in his head, and at his fingertips. Knowing it was there to use, and already knowing how to use it, kept him in the flow and working so quickly and easily.

                                                                So that’s why I started doing flashcards for learning programming things. It’s worked out well.

                                                                Then when learning a new language I find it helps it get into my head quicker and deeper than if I were to just let it happen slowly and naturally. Similar reasons. Deliberately putting things into my memory that might not have worked their way there otherwise.

                                                                1. 2

                                                                  and have it all in his head, and at his fingertips

                                                                  This level of fluency really does make a big difference. It’s one of the reasons I think the modern trend of piling on tools is so damaging. With a large toolset, it is impossible to reach that kind of mastery with all your tools. And what you lose is usually not compensated by the “right tool for the job” benefit.

                                                                  1. 1

                                                                    Thank you Derek.

                                                                    Actually, I think I end up doing something similar but without the help of an app. I keep track of links for how to do various things. It’s much slower to work off of links of course. I could try using a dedicated app instead, both as a search engine and also spaced repetition learning

                                                                  2. 1

                                                                    I’ll admit that I’ve never used flash cards for computer code. As an educator, I have a slight bias against them. I’ve seen too many physics students use flash cards to memorize every problem from every assignment over the semester, only to bomb the final due to the questions not being exact copies of earlier problems.

                                                                    That said, I could also see two places where this would have helped me. I’m writing some Emacs scripts that talk to libpurple over DBus. I’ve been working on this off and on for a couple of months and doing quite a bit of debugging to figure out why the signals I get don’t match my expectations. Yesterday, learned that there has been a dbus-monitor command that would have told me everything I wanted to know. Had I done a set of flash cards of the dbus commands before setting out, this wouldn’t have been surprising information.

                                                                    Similarly, I’ve been maintaining two code paths to update the chat log, both on the signal of a message being sent and on a signal of a message being received. A journey into the documentation today found that there was a signal that triggered on both of these events that cut everything down to watching a single signal. A little time with flash cards of the available signals would have told me to look there before wasting time writing a bunch of duplicate code.

                                                                    1. 2

                                                                      Yesterday, learned that there has been a dbus-monitor command that would have told me everything I wanted to know. Had I done a set of flash cards of the dbus commands before setting out, this wouldn’t have been surprising information.

                                                                      Why not just read (or skim) the documentation?

                                                                      1. 1

                                                                        I did skim the documentation, but I quickly forgot 90% of it by the time I started writing. When I was in the middle of the code was when I needed this information and I didn’t remember it. It was on a second skim that I found the issue.

                                                                        That said, I’m not particularly advocating for flash cards. Just trying to provide for a possible example.

                                                                  1. 5

                                                                    For me, the C++ clutters an otherwise interesting problem. The basic algorithm is simple, but the edge cases (starting indexes / starting sum value, the literal array endpoints, keeping the lower dynamic endpoint <= upper one, no solution found) make it non-trivial to implement cleanly. This is my best effort in JS:

                                                                    function sum(target, arr) {
                                                                      let [a, b, cur] = [0, 0, arr[0]]
                                                                      while (cur != target  && b < arr.length) {
                                                                        if (cur < target || a == b) {
                                                                          b += 1
                                                                          cur += arr[b]
                                                                        else {
                                                                          cur -= arr[a]
                                                                          a += 1
                                                                      if (cur != target) return null
                                                                      return [a, b]

                                                                    Try it online!

                                                                    Another variation is to find the shortest such contiguous sub-sequence.

                                                                    1. 2

                                                                      Given an unsorted array (sequence) of non-negative integers, find a continuous sub-array (sub-sequence) which adds to a given number S. Implement the solution as a generic algorithm, but visualize the results with indices that are 1-based.

                                                                      Despite the fact that I don’t like those artificial interview questions, I was curious to see how my naîve solution would look like. I was using a sum variable inside the outer loop, which results in an O(N^2) algorithm: Go.

                                                                      This version isn’t very useful for the final solution that will be presented below, but allows us to calculate the complexity of this solution, O(N3), as we have 3 nested loops (with N being the number of elements in the sequence)

                                                                      Where is the third inner loop, is it the call to std::accumulate(?

                                                                      1. 2

                                                                        Where is the third inner loop, is it the call to std::accumulate(?

                                                                        I don’t understand singpolyma’s reply, so I might be missing something, but yes std::accumulate has linear time complexity.

                                                                        1. 1


                                                                          … I think they mean N3 also known as O(N)

                                                                          1. 1

                                                                            The loops are nested, so it’s not O(N). That said, I only see two nested loops, and the naive solution is to test every possible pair of endpoints, which is nC2, which is O(N^2).

                                                                            1. 1

                                                                              I wasn’t doing analysis, only commenting that O(N3) would be nonsense. If they mean O(N^3) that’s fine of course.

                                                                              1. 6

                                                                                You can tell it’s a copypaste error because nobody would write N3 to mean N×3, they’d write 3N.

                                                                                1. 2

                                                                                  That’s just how it looks in the cut and paste. In the article the 3 is super-scripted.

                                                                          1. 26

                                                                            I’ve used ruby for 10+ years, understand meta-programming very well, and think it is almost always a mistake. Indeed, every ruby feature that breaks locality of behavior is best avoided entirely. This includes mixins and inheritance.

                                                                            “magic” as a dismissive pejorative represents deep cultural wisdom. Please, keep using it.

                                                                            Could not disagree more with the take presented in the article.

                                                                            1. 4

                                                                              Honest question: If you’re preferred programming style avoids any of the kinds of code you can only write using a dynamically typed language… why not use a statically typed language? It seems like you’re already writing code that would be amenable to static types and doing that gives you all of the nice stuff provided by static analysis: refactoring, code navigation, better error checking, etc.

                                                                              1. 2

                                                                                the kinds of code you can only write using a dynamically typed language

                                                                                I’m not aware of any such code.

                                                                                1. 2

                                                                                  To be clear, I didn’t say ruby was my favorite language (though I like it fine)… just that I’ve used it a long time professionally. I think starting on a new project, all else being equal, I would usually choose a static language for the reasons you mentioned.

                                                                                  With that said, the equation is not exactly that simple in my opinion. This is something I’d need more time to articulate fully, but subjectively, I do think there is a different experience to coding in ruby/js/etc vs Go, or even something concise but statically typed like Haskell. I’m not sure what the value of that thing is, and it probably doesn’t make up for what you lose, but there is something there.

                                                                                  In ruby specifically, the built in Enumerable methods are nice, and the syntax is pretty. I don’t consider these huge factors, but those would be the answers to your question.

                                                                                  EDIT: Maybe one example of what that “thing” is, from J/APL, where the integers 0 and 1 are your boolean types, and this allows you to write stuff like +/ 1&p: to count primes. So there can be advantages to “loose types” as well as footguns, with none of this mitigating my complaints about features that break locality of behavior.

                                                                                  1. 2

                                                                                    Rich Hickey and Clojure has come up with Clojure.spec as a half way house….

                                                                                    You can have “plain old data” and a mechanism to specify the shape of it and validate and generate plain old data.

                                                                                    I’ll admit to being conflicted.

                                                                                    I remember building the most amazing arrays of hashes of hashes of arrays… in perl…. and never being able to maintain what I wrote.

                                                                                    I think he has an interesting idea, but I’m not convinced.

                                                                                  2. 2

                                                                                    This is a good question, but I think the question of “can you reliably determine which function is being called here without running the code” is more or less orthogonal to being able to statically determine the type of a given value.

                                                                                    Most mainstream dynamic languages are bad at both, but languages like Erlang, Racket, Clojure, and Lua handle the first one fairly reliably (the main exception being methods, which are often not used very widely) without making any attempt to solve the latter.

                                                                                  3. 1

                                                                                    You sound like go programmer that is programming go in ruby. ;-)

                                                                                    I’d rephrase your…

                                                                                    Locality of Behavior is the principle that: The behaviour of a unit of code should be as obvious as possible by looking only at that unit of code


                                                                                    Locality of Behavior is the principle that:

                                                                                    The behaviour of a unit of code should be as obvious as possible by looking only at that unit of code, assuming the dependencies that this unit of code invokes are well named, and their effect on this code is well understood.

                                                                                    ie. You don’t have understand how the dependencies do it, just what they do, and in particular, what the effect on this portion of the code is.

                                                                                    1. 1

                                                                                      You sound like go programmer that is programming go in ruby. ;-)

                                                                                      I learned Go years after learning ruby, and hated it at first but learned to appreciate it.

                                                                                      The behaviour of a unit of code should be as obvious as possible by looking only at that unit of code, assuming the dependencies that this unit of code invokes are well named, and their effect on this code is well understood. ie. You don’t have understand how the dependencies do it, just what they do, and in particular, what the effect on this portion of the code is.

                                                                                      I used to think this way too. No more. The dependencies have to all be explicit as well. That is, myDep.doSomething where myDep is explicitly passed to your code is fine. But doSomething where doSomething exists because of a mixin or from your parent class or because of method_missing or some other magic is no good.

                                                                                      1. 0

                                                                                        The point about “functionality inherited from parent” isn’t to provide magic to the child.

                                                                                        The whole point is Liskov’s Substitution Principle. The child IS an instance of the parent class and can be used wherever the parent can be used.

                                                                                        People keep stepping away from the L in SOLID and then wondering why it all hurts.

                                                                                        LSP isn’t a guideline. It’s a mathematical rule. Violate LSP you have a bug, it’s that simple.

                                                                                        If you think in terms of Design by Contract all the time, LSP (and complying with it) becomes intuitive.

                                                                                        1. 1

                                                                                          Yes, yes, I know all the arguments, I’ve read all the books. I know all about SOLID and can write that kind of code just fine. I assure this is not a problem with me being uninformed, or having too little experience for the ideas to become “intuitive.”

                                                                                          It is quite the reverse, and my recommendation is never to use inheritance (EDIT: to clarify, I am talking about implementation inheritance. Programming to an interface is just fine). If this strikes you as a novel stance, just google “inheritance is evil”. If you want specific recommendations for decent articles, let me know and I can dig through my bookmarks.

                                                                                          1. 0

                                                                                            Eh, seen a bunch of that sort of articles go by.

                                                                                            I look at the examples they give and start screaming at the screen… yup, if you do that, of course it will hurt.

                                                                                            So Don’t Do That.

                                                                                            Not convinced.

                                                                                            I guess we’ll have agree to disagree… unless I have to maintain your code! :-)

                                                                                  1. 10

                                                                                    I may just be missing the point because I’m cranky (I should probably drink some water…) but, in light of the stated aim of giving practitioners a beneficial definition, this definition-narrowing project feels a bit quixotic?

                                                                                    I guess the reason is a little ironic.

                                                                                    I agree that we don’t communicate clearly about abstraction, and that a lot of the lack of clarity is a byproduct of how many ~distinct practices we are sweeping under this particular rug. But the project doesn’t seem to stop and ask if, despite the lack of clarity, the term is closing over this many things for ~good reasons (in a way that is consistent with the word’s etymology).

                                                                                    IANAPLT, but abstracting is, much like theorizing, an exercise in making sense of things–and forging (neologizing) the grammar necessary to do so.

                                                                                    1. 7

                                                                                      One of the early academic culture shocks I had back when, as a young engineering student, I tried to seriously dabble in humanities and enroll in philosophy or history classes, was that almost every serious book I picked up started with an entire section consisting of an inventory of how every major author defined the thing that book was about. Every course started with a very serious discussion about said definitions, too, and it was pretty challenging for me. Smartass (student, yes?) me thought this was ridiculous: I thought, you know, if no one has been able to even come up with a satisfactory definition of what historiography is, for example, maybe it shouldn’t be a course?

                                                                                      It took me a while to realize that there’s a difference between advancing a particular classification, which has real investigative value, and simply quibbling on classification and definition, which is, well, just quibbling. The former, I think, has exactly the property that you mention, or perhaps more aptly put, its “mathematical inverse”:

                                                                                      But the project doesn’t seem to stop and ask if, despite the lack of clarity, the term is closing over this many things for ~good reasons (in a way that is consistent with the word’s etymology).

                                                                                      There are about as many definitions and conventions about the Middle Ages, for example, as there are authors. But choosing one over the other does not in any way impede the study of medieval culture. In fact, no matter which one you’re partial to, considering other definitions may help shed light over a particular problem or phenomenon. All major works on the topic disagree about its boundaries somewhat, but everyone understands how it works and they’re universally useful.

                                                                                      Then there are of course the various book sections, papers, articles and essays which spend pages trying to answer questions like “should X be classified as a philosopher of late antiquity, or early middle ages?”. Unsurprisingly, no one finds those too important.

                                                                                      I think there’s value to this approach in technical debates, too. It’s not a problem that only “young” branches of engineering have. Software engineering isn’t the only one that’s struggling with definitions and it’s not because it’s young – there are plenty of electrical engineers (myself included) who are really antsy about transformers being classified as electrical machines, since nothing is spinning inside a transformer, and both are more like 150-200 years old at this point. But it is a tight margin to walk on. The border between good-natured investigation and quibbling is pretty fuzzy.

                                                                                      1. 2

                                                                                        Historiography is pretty much universally agreed to be the history of how histories have been told over historical time. But that just pushes the problem back one square because then you have to define history, and I don’t think there’s any consensus there. Everyone agrees it has to do with the past, but it’s pretty slippery beyond that.

                                                                                        My take is history is the way that the past continues to be important today. Something is important if it makes a causal difference (it doesn’t just wash out as chaotic noise). Causal influence can broken down into physical (the Roman aqueducts are still there, eg) or ideological (the idea of a Roman empire influences a lot of European behavior) influence. And so on, each of those concepts can be broken down and clarified basically forever. :-)

                                                                                        1. 2

                                                                                          Historiography is pretty much universally agreed to be the history of how histories have been told over historical time.

                                                                                          Well, yes, on a high level, but the fact that this definition is a little recursive is exactly why it’s usually not formulated quite that way, and it’s also not the only definition that’s presented in introductory courses, or at least not in all introductory courses, I guess?

                                                                                          The one I was given went something along the lines of the study of the methods that historians used to investigate, present, and interpret history over time. This isn’t a trivial distinction, because the methods and even the object of historical investigation have evolved over time, even in recent times. For example, for the longest time, historians were concerned with understanding the context in which a historical work was written primarily in order to assess its veridicity.

                                                                                          E.g. European historians in antiquity and the middle ages sometimes went out of their way to paint a favourable image of their protector, or an unfavourable image of an otherwise apt ruler who, unfortunately, happened not to be Christian. Understanding when they were presenting fact and when they were talking out of their ass is obviously important, and sometimes you have to resort to indirect evidence to figure that out, because there are no other sources to verify a claim from. But in time, people have tried to make this exercise even more productive, and use it to understand how social values have changed, how the role that Christian doctrine (itself in perpetual evolution) has played in the interpretation of history, or even how various ethical or ecclesiastical views have spread over time.

                                                                                          These things – and this is the second reason why introductory lectures are a little more complicated than “this is what it is, now let’s get to it” – haven’t always been a part of historiography, or they haven’t played the same role they play today. You have to explain how this has changed over time, and how historiographers themselves have viewed their field, even views which are now completely obsolete, otherwise someone will read a book from the 1910s and get entirely the wrong message.

                                                                                          The term itself is polysemantic – it’s commonly used to refer not only to the discipline itself, but also to what used to be its primary object of study, i.e. written sources of history and especially secondary sources. It seems obvious to you and me that, when someone says “the siege and fall of Constantinople are prominently featured in Western historiography” means that there are lots of history books about it, not that historiographers (in the modern sense, which is nothing like what historiographer meant back in the 17th century meant) paid a lot of attention to it. But it’s really something that has to be stated, even if just for clarity.

                                                                                      2. 4

                                                                                        I found it deeply insightful.

                                                                                        Specifically, as a formal framework for giving precision to intuition I’ve built up over the years. To use the article’s framework, I found an abstraction mapping my personal heuristics and models about abstraction to this simpler, unifying idea.

                                                                                        1. 2

                                                                                          I wouldn’t argue that it’s uninsightful.

                                                                                          I think it’s a kind of missing-the-forest-for-the-trees. It focuses on things we might touch when abstracting and not on what we accomplish by abstracting. It proposes a narrower definition of the term in the more literal context, and doesn’t acknowledge–let alone fill–the hole it’s leaving in the more motivational one.

                                                                                          1. 3

                                                                                            I think you might be misreading the intent. The author’s primary interest is applying “academic” insights like these back in the real world. The motivation here is to improve your skills as a code designer. Granted, it would have been helpful to see a problem worked though applying these insights, but I took this as a high-level piece, where those kinds of lessons were left as exercises to the reader.

                                                                                            As I read it, the “narrower definition” you’re referring to is meant as framework for evaluating real-world abstractions, and perhaps for finding them too. So if something “feels funny” about a design, you can analyze it through this lens, finding the two domains, evaluating precisely how the abstract domain helps you – which operations track between the concrete domain and abstract one and does that simplify your problem? It might help pinpoint where an abstraction is “leaky” and how to fix that.

                                                                                            1. 1

                                                                                              Not sure there is daylight between this statement of motive and the one in my first post (but of course I still agree that I may be misunderstanding).

                                                                                              In any case, names are abstractions. Part of naming things is deciding where they start and end. In part by association and in part because they can be described with slightly less woo, the tools and techniques we use to separate, extract, group, collect–and to name the things separated, extracted, grouped, and collected–become a synechdoche for what we’re here to do.

                                                                                              I suppose the traditional way to think of this is carving nature at its joints, but I would resist framing it that way. There’s a risk of missing the point and reifying joints just because they appear to be natural. Sometimes this is right, and sometimes not. Our perception lies, of course, and two creatures that appear related are in fact far from it.

                                                                                              The article focuses on a narrow thing:

                                                                                              abstractions are mappings between a complex concrete world and a simple idealized one. For concrete data types, an abstraction maps a complicated data structure to the basic data it represents. For systems, an abstraction relates the tiny state changes from every line implementing a TCP stack with the information found in the actual protocol diagram. These abstractions become useful when one can define interesting operations purely on the abstract form

                                                                                              It tries to remove from our quiver (and keep for itself) a term we use to acknowledge that namemaking–including the guesses we make about where the seams lurk–is how we weave a grammar that helps humans (continually) make sense of the program/problem domains.

                                                                                              The ~world of this grammar need not be simple or idealized. It may often be–but is at its most fraught when it tries desperately to avoid or simplify intrinsic complexity that the humans must inevitably come to terms with. We can likewise weave simplified/idealized grammars that do not make much sense to many humans.

                                                                                      1. 2

                                                                                        Is it considered bad form to make each of those methods return an object and define the methods on each of those objects? If you set up like so:

                                                                                        class Message
                                                                                          attr_accessor :message_string
                                                                                          # returns a DecodedMessage object
                                                                                          def decode

                                                                                        …and so on, you’d be able to end up with message.decode.build_event.process. You have to create explicit objects for each of those intermediate steps so those methods have somewhere to live, so it’s more verbose and indirect overall, but you get some very readable code.

                                                                                        1. 2

                                                                                          A better way to do this generically is with dry-monads and the Result type. If each function returns a Result, you can easily compose them together:

                                                                                          1. 2

                                                                                            You might think so, because it embraces ruby’s OO spirit, but in practice this solution would be far worse: the overhead of three abstractions, the bulkiness of having to create 3 classes, the way it breaks locality of behavior – the cure would be far worse than the disease.

                                                                                            1. 1

                                                                                              My only concern would be whether those objects are good abstractions.

                                                                                            1. 121

                                                                                              I used to give the same advice, but I completely changed my opinion over the past 10 years or so. I eventually put in the time and learned shell scripting. These days my recommendation is:

                                                                                              1. Learn to use the shell. It’s a capable language that can take you very far.
                                                                                              2. Use ShellCheck to automatically take care of most of the issues outlined in the article.

                                                                                              I really don’t want to figure out every project’s nodejs/python/ruby/make/procfile abomination of a runner script anymore. Just like wielding regular expressions, knowing shell scripting is a fundamental skill that keeps paying dividends over my entire career.

                                                                                              1. 60


                                                                                                My advice is:

                                                                                                • Always use #!/usr/bin/env bash at the beginning of your scripts (change if you need something else, don’t rely on a particular path to bash though).
                                                                                                • Always add set -eou pipefail after that.
                                                                                                • Always run shellcheck.
                                                                                                • Always run shfmt.
                                                                                                • Always pay attention to what version of bash you need to support, and don’t go crazy with “new” features unless you can get teammates to upgrade (this is particularly annoying because Apple ships an older version of bash without things like associative arrays).
                                                                                                • Always use the local storage qualifier when declaring variables in a function.
                                                                                                • As much as possible, declare things in functions and then at the end of your script kick them all off.
                                                                                                • Don’t use bash for heavy-duty hierarchical data munging…at that point consider switching languages.
                                                                                                • Don’t assume that a bashism is more-broadly acceptable. If you need to support vanilla sh, then do the work.

                                                                                                While some people like the author will cry and piss and moan about how hard bash is to write, it’s really not that bad if you take those steps (which to be fair I wish were more common knowledge).

                                                                                                To the point some folks here have already raised, I’d be okay giving up shell scripting. Unfortunately, in order to do so, a replacement would:

                                                                                                • Have to have relatively reasonable syntax
                                                                                                • Be easily available across all nix-likes
                                                                                                • Be guaranteed to run without additional bullshit (installing deps, configuring stuff, phoning home)
                                                                                                • Be usable with only a single file
                                                                                                • Be optimized for the use case of bodging together other programs and system commands with conditional logic and first-class support for command-line arguments, file descriptors, signals, exit codes, and other nixisms.
                                                                                                • Be free
                                                                                                • Don’t have long compile times

                                                                                                There are basically no programming languages that meet those criteria other than the existing shell languages.

                                                                                                Shell scripting is not the best tool for any given job, but across every job it’ll let you make progress.

                                                                                                (Also, it’s kinda rich having a Python developer tell us to abandon usage of a tool that has been steadily providing the same, albeit imperfect, level of service for decades. The 2 to 3 switch is still a garbage fire in some places, and Python is probably the best single justification for docker that exists.)

                                                                                                1. 26

                                                                                                  While some people like the author will cry and piss and moan about how hard bash is to write, it’s really not that bad if you take those steps (which to be fair I wish were more common knowledge).

                                                                                                  I think “nine steps” including “always use two third-party tools” and “don’t use any QoL features like associative arrays” does, in fact, make bash hard to write. Maybe Itamar isn’t just “cry and piss and moan”, but actually has experience with bash and still think it has problems?

                                                                                                  1. 2

                                                                                                    To use any language effectively there are some bits of tribal knowledge…babel/jest/webpack in JS, tokio or whatever in Rust, black and virtualenv in Python, credo and dialyzer in Elixir, and so on and so forth.

                                                                                                    Bash has many well-known issues, but maybe clickbait articles by prolific self-pronoters hat don’t offer a path forward also have problems?

                                                                                                    1. 15

                                                                                                      If your problem with the article is that it’s clickbait by a self-promoter, say that in your post. Don’t use it as a “gotcha!” to me.

                                                                                                      1. 2

                                                                                                        I think there’s merit here in exploring the criticism, though room for tone softening. Every language has some form of “required” tooling that’s communicated through community consensus. What makes Bash worse than other languages that also require lots of tools?

                                                                                                        There’s a number of factors that are at play here and I can see where @friendlysock’s frustration comes from. Languages exist on a spectrum between lots of tooling and little tooling. I think something like SML is on the “little tooling” where just compilation is enough to add high assurance to the codebase. Languages like C are on the low assurance part of this spectrum, where copious use of noisy compiler warnings, analyzers, and sanitizers are used to guide development. Most languages live somewhere on this spectrum. What makes Bash’s particular compromises deleterious or not deleterious?

                                                                                                        Something to keep in mind is that (in my experience) the Lobsters userbase seems to strongly prefer low-tooling languages like Rust over high-tooling languages like Go, so that may be biasing the discussion and reactions thereof. I think it’s a good path to explore though because I suspect that enumerating the tradeoffs of high-tooling or low-tooling approaches can illuminate problem domains where one fits better than the other.

                                                                                                        1. 2

                                                                                                          I felt that I sufficiently commented about the article’s thesis on its own merits, and that bringing up the author’s posting history was inside baseball not terribly relevant. When you brought up motive, it became relevant. Happy to continue in DMs if you want.

                                                                                                        2. 6

                                                                                                          You’re really quite hostile. This is all over scripting languages? Or are you passive aggressively bringing up old beef?

                                                                                                      2. 9

                                                                                                        Integrating shellcheck and shfmt to my dev process enabled my shell programs to grow probably larger than they should be. One codebase, in particular, is nearing probably like 3,000 SLOC of Bash 5 and I’m only now thinking about how v2.0 should probably be written in something more testable and reuse some existing libraries instead of reimplementing things myself (e.g., this basically has a half-complete shell+curl implementation of the Apache Knox API). The chief maintenance problem is that so few people know shell well so when I write “good” shell like I’ve learned over the years (and shellcheck --enable=all has taught me A TON), I’m actively finding trouble finding coworkers to help out or to take it over. The rewrite will have to happen before I leave, whenever that may be.

                                                                                                        1. 11

                                                                                                          I’d be interested in what happens when you run your 3000 lines of Bash 5 under . Oil is the most bash compatible shell – by a mile – and has run thousands of lines of unmodified shell scripts for over 4 years now (e.g.

                                                                                                          I’ve also made tons of changes in response to use cases just like yours, e.g.

                                                                                                          Right now your use case is the most compelling one for Oil, although there will be wider appeal in the future. The big caveat now is that it needs to be faster, so I’m actively working on the C++ translation (oil-native passed 156 new tests yesterday).

                                                                                                          I would imagine your 3000 lines of bash would be at least 10K lines of Python, and take 6-18 months to rewrite, depending on how much fidelity you need.

                                                                                                          (FWIW I actually wrote 10K-15K lines of shell as 30K-40K lines of Python early in my career – it took nearly 3 years LOL.)

                                                                                                          So if you don’t have 1 year to burn on a rewrite, Oil should be a compelling option. It’s designed as a “gradual upgrade” from bash. Just running osh will work, or you can change the shebang line, run tests if you have them, etc.

                                                                                                          There is an #oil-help channel on Zulip, liked from the home page

                                                                                                          1. 2

                                                                                                            Thanks for this nudge. I’ve been following the development of Oil for years but never really had a strong push to try it out. I’ll give it a shot. I’m happy to see that there are oil packages in Alpine testing: we’re deploying the app inside Alpine containers.

                                                                                                            Turns out that I was very wrong about the size of the app. It’s only about 600 SLOC of shell :-/ feels a lot larger when you’re working on it!

                                                                                                            One thing in my initial quick pass: we’re reliant on bats for testing. bats seemingly only uses bash. Have you found a way to make bats use Oil instead?

                                                                                                            1. 1

                                                                                                              OK great looks like Alpine does have the latest version:

                                                                                                              I wouldn’t expect this to be a pain-free experience, however I would say should definitely be less effort than rewriting your whole program in another language!

                                                                                                              I have known about bats for a long time, and I think I ran into an obstacle but don’t remember what it was. It’s possible that the obstacle has been removed (e.g. maybe it was extended globs, which we now support)


                                                                                                              In any case, if you have time, I would appreciate running your test suite with OSH and letting me know what happens (on Github or Zulip).

                                                                                                              One tricky issue is that shebang lines are often #!/bin/bash, which you can change to be #!/usr/bin/env osh. However one shortcut I added was OSH_HIJACK_SHEBANG=osh


                                                                                                            2. 1

                                                                                                              Moving away from Python? Now it has my interest… in the past I skipped past know it’d probably take perf hits and have some complicaged setup that isn’t a static binary.

                                                                                                              1. 2

                                                                                                                Yes that has always been the plan, mentioned in the very first post on the blog. But it took awhile to figure out the best approach, and that approach still takes time.

                                                                                                                Some FAQs on the status here:

                                                                                                                Python is an issue for speed, but it’s not an issue for setup.

                                                                                                                You can just run ./configure && make && make install and it will work without Python.

                                                                                                                Oil does NOT depend on Python; it just reuses some of its code. That has been true for nearly 5 years now – actually since the very first Oil 0.0.0. release. Somehow people still have this idea it’s going to be hard to install, when that’s never been the case. It’s also available on several distros like Nix.

                                                                                                                1. 1

                                                                                                                  What is the status of Oil on Windows (apologies if it’s in the docs somewhere, couldn’t find any mentioning of this). A shell that’s written in pure C++ and has Windows as a first class citizen could be appealing (e.g. for cross-platform build recipes).

                                                                                                                  1. 1

                                                                                                                    It only works on WSL at the moment … I hope it will be like bash, and somebody will contribute the native Windows port :-) The code is much more modular than bash and all the Unix syscalls are confined to a file or two.

                                                                                                                    I don’t even know how to use the Windows sycalls – they are quite different than Unix! I’m not sure how you even do fork() on Windows. (I think Cygwin has emulation but there is way to do it without Cygwin)


                                                                                                          2. 4

                                                                                                            To the point some folks here have already raised, I’d be okay giving up shell scripting. Unfortunately, in order to do so, a replacement would: […] There are basically no programming languages that meet those criteria other than the existing shell languages.

                                                                                                            I believe Tcl fits those requirements. It’s what I usually use for medium-sized scripts. Being based on text, it interfaces well with system commands, but does not have most of bash quirks (argument expansion is a big one), and can handle structured data with ease.

                                                                                                            1. 4

                                                                                                              Always use #!/usr/bin/env bash at the beginning of your scripts (change if you need something else, don’t rely on a particular path to bash though).

                                                                                                              I don’t do this. Because all my scripts are POSIX shell (or at least as POSIX complaint as I can make them). My shebang is always #!/bin/sh - is it reasonable to assume this path?

                                                                                                              1. 4

                                                                                                                you will miss out on very useful things like set -o pipefail, and in general you can suffer from plenty of subtle differences between shells and shell versions. sticking to bash is also my preference for this reason.

                                                                                                                note that the /usr/bin/env is important to run bash from wherever it is installed, e.g. the homebrew version on osx instead of the ancient one in /bin (which doesn’t support arrays iirc and acts weirdly when it comes across shell scripts using them)

                                                                                                                1. 4

                                                                                                                  My shebang is always #!/bin/sh - is it reasonable to assume this path?

                                                                                                                  Reasonable is very arbitrary at this point. That path is explicitly not mandated by POSIX, so if you want to be portable to any POSIX-compliant system you can’t just assume that it will exist. Instead POSIX says that you can’t rely on any path, and that scripts should instead be modified according to the system standard paths at installation time.

                                                                                                                  I’d argue that these days POSIX sh isn’t any more portable than bash in any statistically significant sense though.

                                                                                                                  1. 2

                                                                                                                    Alpine doesn’t have Bash, just a busybox shell. The annoying thing is if the shebang line fails because there is no bash, the error message is terribly inscrutable. I wasted too much time on it.

                                                                                                                    1. 2

                                                                                                                      nixos has /bin/sh and /usr/bin/env, but not /usr/bin/bash. In fact, those are the only two files in those folders.

                                                                                                                    2. 3

                                                                                                             hardcodes #!/bin/sh. POSIX definitely doesn’t say anything about shs location but I really doubt you won’t find a sh at /bin/sh on any UNIX system. Can anybody name one?

                                                                                                                    3. 2

                                                                                                                      I would add, prefer POSIX over bash.

                                                                                                                    4. 18

                                                                                                                      I checked, and shellcheck (at least the version on my computer) only catches issue #5 of the 5 I list.

                                                                                                                      1. 14

                                                                                                                        That’s because the other ones are options and not errors. Yes, typically they are good hygiene but set -e, for example, is not an unalloyed good, and at least some experts argue against using it.

                                                                                                                        1. 3

                                                                                                                          Not for lack of trying:

                                                                                                                          There are tons of pedants holding us back IMO. Yes, “set -e” and other options aren’t perfect, but if you even know what those situations are, you aren’t the target audience of the default settings.

                                                                                                                        2. 17

                                                                                                                          I eventually put in the time

                                                                                                                          Yup, that’s how you do it, It’s a good idea to put in the the time to understand shell scripting. Most of the common misconceptions come out of misunderstanding. The shell is neither fragile (it’s been in use for decades, so it’s very stable) nor ugly (I came from JavaScript to learning shell script, and it seemed ugly indeed at first, now I find it very elegant). Keeping things small and simple is the way to do it. When things get complex, create another script, that’s the UNIX way.

                                                                                                                          It’s the best tool for automating OS tasks. That’s what it was made for.

                                                                                                                          +1 to using ShellCheck, I usually run it locally as

                                                                                                                          shellcheck -s sh

                                                                                                                          for POSIX compliance.

                                                                                                                          I even went as far as generating my static sites with it You’re using the shell daily for displaying data in the terminal, it’s a great tool for that, why not use the same tool for displaying data publicly.

                                                                                                                          1. 6

                                                                                                                            No, it really is ugly. But I’m not sure why that matters

                                                                                                                            1. 13

                                                                                                                              I believe arguing if beauty is subjective or not is off topic. 😛

                                                                                                                          2. 16

                                                                                                                            I went the opposite direction - I was a shell evangelist during the time that I was learning it, but once I started pushing its limits (e.g. CSV parsing), and seeing how easy it was for other members of my team to write bugs, we immediately switched to Python for writing dev tooling.

                                                                                                                            There was a small learning curve at first, in terms of teaching idiomatic Python to the rest of the team, but after that we had much fewer bugs (of the type mentioned in the article), much more informative failures, and much more confidence that the scripts were doing things correctly.

                                                                                                                            I didn’t want to have to deal with package management, so we had a policy of only using the Python stdlib. The only place that caused us minor pain was when we had to interact with AWS services, and the solution we ended up using was just to execute the aws CLI as a subprocess and ask for JSON output. Fine!

                                                                                                                            1. 15

                                                                                                                              I tend to take what is, perhaps, a middle road. I write Python or Go for anything that needs to do “real” work, e.g. process data in some well-known format. But then I tie things together with shell scripts. So, for example, if I need to run a program, run another program and collect, and then combine the outputs of the two programs somehow, there’s a Python script that does the combining, and a shell script that runs the three other programs and feeds them their inputs.

                                                                                                                              I also use shell scripts to automate common dev tasks, but most of these are literally one-ish line, so I don’t think that counts.

                                                                                                                              1. 2

                                                                                                                                This makes sense to me

                                                                                                                              2. 8

                                                                                                                                we immediately switched to Python for writing dev tooling.

                                                                                                                                FWIW when shell runs out of steam for me, I call Python scripts from shell. I would say MOST of my shell scripts call a Python script I wrote.

                                                                                                                                I don’t understand the “switching” mentality – Shell is designed to be extended with other languages. “Unix philosophy” and all that.

                                                                                                                                I guess I need to do a blog post about this ? (Ah I remember I have a draft and came up with a title – The Worst Amounts of Shell Are 0% or 100% (requires login)

                                                                                                                                (Although I will agree that it’s annoying that shell has impoverished flag parsing … So I actually write all the flag parsers in Python, and use the “task file” pattern in shell.)

                                                                                                                                1. 2

                                                                                                                                  What is the “task file” pattern?

                                                                                                                                  1. 5

                                                                                                                                    It’s basically a shell script (or set of scripts) you put in your repo to automate common things like building, testing, deployment, metrics, etc.

                                                                                                                                    Each shell function corresponds to a task..

                                                                                                                                    I sketched it in this post, calling it “semi-automation”:


                                                                                                                                    and just added a link to:


                                                                                                                                    (many code examples from others in that post, also almost every shell script in is essentially that pattern)

                                                                                                                                    There are a lot of names for it, but many people seem to have converged on the same idea.

                                                                                                                                    I don’t have a link handy not but Github had a standard like this in the early days. All their repos would have a uniform shell interface so that you could get started hacking on it quickly.

                                                                                                                              3. 5

                                                                                                                                You should investigate just for task running. It’s simple like make but none of the pitfalls of it for task running.

                                                                                                                              1. 9

                                                                                                                                Waiting for the person who really really likes the cgi module and the optparse to comment on this.

                                                                                                                                1. 5

                                                                                                                                  Hi… as the maintainer for a project that uses the cgi module, I am frustrated that I am now going to have to vendor FieldStorage vs just being able to use it from the standard library.

                                                                                                                                  I even posted as such when this was proposed in 2019, but alas it did not make the cut. Shame really, as now those things are going to exist in countless different projects.

                                                                                                                                  Also, the replacement (multipart) is still marked as “beta” on PyPi, and adds yet another dependency that I as a maintainer need to watch for/understand their upgrade cycles, make sure that their API doesn’t have breaking changes.

                                                                                                                                  Anyway, disappointed that they are removing something that is still very useful. Surprisingly (or maybe not) the cgi module is still fairly heavily used in embedded systems/devices that want to run a quick Python script or two through a httpd of some sort without needing to run it full time or using a Python web server.

                                                                                                                                  1. 4

                                                                                                                                    now those things are going to exist in countless different projects.

                                                                                                                                    Could they not go in a pip library?

                                                                                                                                    1. 3

                                                                                                                                      The problem with “pull in random pip library” is one day the maintainer disappears.

                                                                                                                                      That in itself is understandable. But the time between that and people realizing, and the time between that and actually getting to “OK let’s fork it I suppose”, and to actually setting up a new maintainer can be years. Again, the maintainer can do what they want, but the knock-on effects exist, and it’s not for lack of people wanting to manage releases!

                                                                                                                                      This is a common problem with Django-related projects. Django doesn’t break backwards compat super often, and loads of people stick on LTS things, but you end up with packages that are just stuck without fixes of the “rename this import” variety.

                                                                                                                                      This is part of why I do work with Jazzband (to just fix stuff), but I generally believe that our future will be much nicer if packages with any sort of real usage end up in some system that guarantees that we can find new maintainers if needed.

                                                                                                                                      Maybe we need python-contrib that just includes the things “most people use” (requests/attrs etc).

                                                                                                                                      1. 3

                                                                                                                                        The problem with “pull in random pip library” is one day the maintainer disappears.

                                                                                                                                        This problem doesn’t go away from having it in the standard library. The maintainer goes away but the standard library as a whole still has maintainers. Bugs in that bit don’t get fixed because no one left remembers how it works. Eventually there’s a critical security vulnerability and the rest of the maintainers realise that the last person who knows how to fix it left the project a decade ago.

                                                                                                                                        If anything, it’s more obvious in a separate project if there are still active maintainers.

                                                                                                                                        1. 3

                                                                                                                                          I’m not familiar with the maintenance of python’s stdlib in particular, but in the general case I’ve noticed a stark difference. For example, in ruby or Go or JS, you’ll be far more likely to run into bugs or other problems with 3rd party modules. Corporate backing, the sheer number of people relying on it, the official identity of the language being tied to it…. these cultural forces seem to make a difference.

                                                                                                                                          1. 1

                                                                                                                                            C and C++ have much smaller standard libraries but in both there are parts of common implementations of the standard library that very few people understand and that haven’t been touched for over 10 years. Mostly they work and so no one needs to touch the code but if there ever is a bug then good luck

                                                                                                                                          2. 1

                                                                                                                                            Actually there is a core difference, in which the standard library has its own maintainers so if you personally want to start maintaining a thing that is not getting love, there are contact points that are, in theory, available and have the keys to the kingdom.

                                                                                                                                            This is colored by my personal experience, but usually the lack of maintenance becomes a problem not from security issues, but from slow API changes in related libraries. Most people who have a habit of jumping into unknown code can look at a thing and try to untangle the mess in a implementation-compatible way. But if they don’t have release rights that work doesn’t go anywhere.

                                                                                                                                            EDIT: There are exceptional cases to the contrary, but what I see a lot of is not projects without willing maintainers, but projects in limbo because passing the torch is difficult without a whole flow for determining that.

                                                                                                                                          3. 2

                                                                                                                                            If you vendor it you start with this problem, and worse because someone may actually be maintaining their vendorerd copy and you won’t even know! Better to maintain a library people can use than everyone maintain their own copy. That’s why we invented these systems :)

                                                                                                                                      2. 5

                                                                                                                                        PEP 594 does not plan to remove optparse:

                                                                                                                                        Although it has been deprecated for many years, it’s still too widely used to remove it.

                                                                                                                                        1. 4

                                                                                                                                          Click is a good example of why it’s hard to remove: it’s a popular argument parsing library that was purposely built on top of optparse.

                                                                                                                                          1. 6

                                                                                                                                            Click does not use optparse. It vendors a fork.

                                                                                                                                            1. 1

                                                                                                                                              True in part (it doesn’t use the standard library version of optparse, but it still uses a copy of it, specifically because optparse was already deprecated when Click was created), but the point still stands: there are projects out there that will need to do something similar, or they’ll break.

                                                                                                                                      1. 3

                                                                                                                                        Fun problem. Would you mind if I posted it on with an attribution to you?

                                                                                                                                        My solution in J, using a function that takes the wordlist as the left argument and the counts as the right argument: {~[:(I.?@{:)+/\

                                                                                                                                        Try it online!

                                                                                                                                        1. 8

                                                                                                                                          My professional life has been so much better since I started using primarily acceptance tests. Yes they take a while to run but they test the actual user journey. Unit tests with their numerous mocks are like pouring concrete on code and make even simple refactors arduous.

                                                                                                                                          I don’t even really know where an integration test fits in, like an acceptance test with a mock database?

                                                                                                                                          From my position at the moment I don’t see why people do it to themselves, acceptance tests FTW.

                                                                                                                                          1. 11

                                                                                                                                            I’ve had the reverse experience.

                                                                                                                                            A code base with mostly acceptance tests, that took an hour to run in the CI. Absolute velocity killer. Then some of the tests start flaking, forcing multiple CI runs, and merging basic changes takes half a day.

                                                                                                                                            Unit tests with their numerous mocks are like pouring concrete on code and make even simple refactors arduous.

                                                                                                                                            If most of your unit tests are like this, it’s usually a signal that the code needs to be designed differently. Unit tests shine for testing pure functions. Yes, for testing high-level “service coordination” code that has non-trivial business rules, the kind of tests with mocks for each service make sense. But even there, if the code is designed well, the tests themselves can be simple.

                                                                                                                                            Code design is the elephant in the room whenever testing is discussed. It’s impossible to talk about testing strategies without talking about code design, though.

                                                                                                                                            Michael Feathers - the deep synergy between testability and good design

                                                                                                                                            Finally, there is the connection between testing and the minute-by-minute experience of building software, which an acceptance-only approach performs poorly on:

                                                                                                                                            Gary Bernhardt’s Fast Test, Slow Test

                                                                                                                                            1. 2

                                                                                                                                              What type of code are you writing? I ask, because I often find talks about testing are very hard to apply to the work I do. I work on a legacy C base, using a proprietary library/device driver for about half of it [0]. My team might get four deployments per year, and our customer (the Oligarchic Cell Phone Company) has sprints that last years [1]. So an hour to run tests isn’t that bad [2].

                                                                                                                                              Another question I keep asking, but never get a satisfactory answer to, is, what is a unit? Our “program” actually consists of several programs—two accepting network requests [3], which then forward it to our business logic unit, which in turn makes two concurrent requests to outside databases [4] [5] which are then sliced, diced and merged into a reply. And because of the way the code is written (C code), in my opinion, the “units” are the individual programs. Could the “business logic” be a pure function? Maybe? At this point, it would require a major restructuring of the code and given current management, I don’t see that happening at all.

                                                                                                                                              I started listening to the Michael Feathers talk you linked to, but the assumption he makes is that you are writing in an object oriented programming language, so a lot of his advice doesn’t apply, or is hard to apply. Not all of us are using OOP.

                                                                                                                                              [0] A proprietary network stack, which we are only licensed to run on Solaris 5.10/5.11 on SPARC. The less said about The Protocol Stack From Hell, the better (the thing falls over if you breath on it hard, and we’re not paid to debug third party code, which itself is in legacy mode …)

                                                                                                                                              [1] For instance, it took five years for our customer to add one (1) piece of information to one of the SIP headers we receive. Five years. I don’t think they practice continuous integration over there.

                                                                                                                                              [2] Due to an overly-specific test plan, our first test platform (which wasn’t entirely automatic) took over five hours to run 120 tests. If only we didn’t specify the syslog output …

                                                                                                                                              [3] Via SS7 and SIP

                                                                                                                                              [4] Via DNS. We make NAPTR requests.

                                                                                                                                              [5] We are also under a real time deadline, with only two seconds to return a result, thus why we make the requests concurrently.

                                                                                                                                            2. 3

                                                                                                                                              Acceptance tests are important because they exercise the project from the perspective of the user, which is, ultimately, the thing that matters. But they model the project as a black box, a single unit whole thing, which means they’re totally agnostic to the internals. For nontrivial projects, this is a problem. Tests assert correctness and prevent regressions and all that good stuff, but they’re also the best tool we have to motivate and enforce coherence, encapsulation, and all the other virtues of software engineering. Each meaningful layer of abstraction in your project needs tests, if it’s gonna survive the test of time.

                                                                                                                                              1. 5

                                                                                                                                                Unit tests also from the perspective of a user, where the user is another dev or team in the project or company that will need to use or rely on this API.

                                                                                                                                            1. 45

                                                                                                                                              I learned AWS on the job.

                                                                                                                                              1. 8

                                                                                                                                                I am learning on the job at the moment, and this youtube channel is the best resource I’ve found.

                                                                                                                                                He’s got a nice style and a knack for cutting through irrelevant details to expose the essence of the services. You can try this excellent:

                                                                                                                                                18 minute overview of the core AWS services

                                                                                                                                                to see if you like it.

                                                                                                                                                I’ve also done (or partially done) a few on LinkedIn learning and somewhere else I forgot, and they weren’t as good as this free resource.

                                                                                                                                                1. 1

                                                                                                                                                  very strange list of “most important” IMO. But what is most important to one will not be most important to another.

                                                                                                                                                2. 1

                                                                                                                                                  +1. I did a 3 day course offered by AWS. It was helpful to learn the foundational blocks like access management, ARNs and course taught us how to build a web service.

                                                                                                                                                  For any new services, I’ve found their docs helpful and there are usually tutorial style documents that teach you how to do a specific thing with the service.

                                                                                                                                                  1. 1

                                                                                                                                                    I did this as well but I recommend not having to learn how to deploy, configure, and maintain Hadoop clusters at the same time as in my case.

                                                                                                                                                  1. 31

                                                                                                                                                    Serve static html from the backend, and forego dynamic behavior on the front end. You’ll spend a lot less time doing things unrelated to your goal.

                                                                                                                                                    1. 11

                                                                                                                                                      This is the answer. Don’t mess around with SPAs, they will make your life much harder. Just write a traditional web app. Use express.js if you know JS, or sinatra if you know ruby, etc. These are tools you can actually start using in an hour or less, that can serve routes in a few lines of code, and whose features you can learn incrementally as needed. If you want to add a little slickness to your UI (transitions, fetches without page reloads, etc), while staying in the classic html/css paradigm you already know, have a look at

                                                                                                                                                      1. 2

                                                                                                                                                        Use express.js if you know JS, or sinatra if you know ruby, etc

                                                                                                                                                        It’s C++. There’s a lot of it already; the UI is just a layer on top. As I said I used a couple of libraries to give me a rudimentary routing system (based on path regexes) and Jinja-like templates.

                                                                                                                                                        1. 2

                                                                                                                                                          I have a lot of webdev experience with various stacks, but not C++. From my (limited) general experience with C++, I would imagine it would be a cumbersome web development experience, but if it’s working for you, go for it. I had assumed you were asking for something else. Either way, I think my original advice remains the same. Backend language you are familiar with, minimalist framework you can learn very quickly (or none at all, eg, in golang the std lib is enough), and htmx for UI effects/fanciness if needed.


                                                                                                                                                          From your other reply:

                                                                                                                                                          And I wouldn’t be considering HTML for the UI at all if it couldn’t do dynamic behavior; that’s table stakes. It’s not 1995 anymore and my goal isn’t to rebuild Geocities :)

                                                                                                                                                          It really depends how complex the dynamic parts need to be. At some point, if the frontend logic gets out of control, something like React may be warranted. But ime most sites that use React don’t need it. For a “social-networking”-esque site, I could see this going either way. But my instinct is still to start with a traditional architecture augmented by htmx until you find yourself needing more.

                                                                                                                                                      2. 7

                                                                                                                                                        The HTML has to be dynamically generated based on the (C++) model layer. This isn’t a static website. But I’m fairly satisfied with the C++ template engine I’m using.

                                                                                                                                                        And I wouldn’t be considering HTML for the UI at all if it couldn’t do dynamic behavior; that’s table stakes. It’s not 1995 anymore and my goal isn’t to rebuild Geocities :)

                                                                                                                                                        1. 6

                                                                                                                                                          I have written… a significant amount of javascript in the past ~15 years.

                                                                                                                                                          I would recommend you start by thinking carefully about where you want the state to live. Having some state in the javascript, some in the URL, some in HTTP cookies and some in the server (‘stateful connections’) is a recipe for trouble. IME, storing auth state in cookies and everything else in the URL has consistently been the most compatible with how browsers want pages to behave, at the cost of restricting how much state you can store (to a few kb).

                                                                                                                                                          For most content, it’s easier to get a better result if you use <a href links rather than javascript (generating different HTML in the server process based on the URL).

                                                                                                                                                          For parts which constantly receive new input from the server (eg a chat window), you can render an iframe to a URL which holds the connection open (using chunked encoding) and sends new content as it arrives. This requires very little code, is fast, and retains state correctly across page refreshes with no action on your part.

                                                                                                                                                          Beyond that, a very small amount of javascript to toggle classes is typically enough.

                                                                                                                                                          1. 1

                                                                                                                                                            Thanks. Fortunately there is pretty much no UI-specific state I need to keep. It’s all running on a client device so there’s by definition only a single user.

                                                                                                                                                          2. 5

                                                                                                                                                            Here’s a few fairly powerful vanilla techniques:

                                                                                                                                                            • Replacement for onclick: document.querySelectorAll(...).addEventListener('click', function(event) { const element = this; ... }
                                                                                                                                                            • Dynamically fetch HTML rendered by the server: fetch("/url...", { headers: { 'Content-Type': 'text/html' }, ... }).then(resp => resp.text()).then(htmlText => { element.innerHTML = htmlText; })
                                                                                                                                                        1. 8

                                                                                                                                                          The follow-up article, How to make MPAs that are as fast as SPAs, is interesting too.

                                                                                                                                                          1. 10

                                                                                                                                                            Inline everything, including CSS and JavaScript

                                                                                                                                                            This will run counter to removing unsafe-inline for scripts in styles for CSP. This also will hurt caching, especially when switching pages. I’d gladly trade a teeny bit of time for security and caching. You could add nonces for the inline stuff, but you are also trading adding complexity to your system for what I view as little gain.

                                                                                                                                                            Minify and gzip all the things

                                                                                                                                                            2022 and not mentioning Brotli despite wide support for several years?

                                                                                                                                                            1. 4

                                                                                                                                                              I use ES modules to create tiny bundles of JS that I can load only on the pages that need them. Modules let me reuse shared snippets across pages without having to copy/paste/repeat, which would be an unmaintainable nightmare.

                                                                                                                                                              IIRC ES modules were not around at jQuery/knockout time. Yes, you could make a tiny “calendar.js” and “header.js” etc, but then had to bite into a lot of other complexity, like onDocument load and other fun things. And imports were not inlined. I might be mistaken, but js imports are a relatively new thing (and still not supported by IE) . Also Hugo and static site generation wasn’t around back then. Same for service workers (also not fully supported by IE). Remember when you had to polyfill to use ?

                                                                                                                                                              I agree with OP, SPA’s are often overkill, but it seems too easy to dismiss that we got where we are today traveling trought a history of successes and mistakes. SPA’s would not have been a thing without the horrors of PHP or JSP (Java Server Pages). Little to no one would have used jquery if the browser apis had decent dom manipulation and sane apis for ajax requests. Etc etc etc.

                                                                                                                                                              This to say, the best and the ugliest part of web developement is the crazy pace of change. We are pretty much living an IRL genetic algorithm.

                                                                                                                                                              1. 7

                                                                                                                                                                Also Hugo and static site generation wasn’t around back then.

                                                                                                                                                                I agree with you, but static site generation as a practice it probably at least as old as dynamic site generation. Of the more modern tooling, Jekyll is just 2 years younger than jQuery itself and two years older than Knockout.js (2008 vs 2006 vs 2010)

                                                                                                                                                                1. 13

                                                                                                                                                                  There was a blog tool called Movable Type which generated static pages dating back to 2001. :)

                                                                                                                                                                  I strongly expect that people writing Perl scripts to output files into a “www/” directory predates cgi-bin. :)

                                                                                                                                                                  1. 5

                                                                                                                                                                    And the original incarnation of Blogger (1999?) did not host your blog, rather it FTPd the generated static files to your configured web host. (BlogSpot came a bit later, as a default host for people who didn’t have their own.)

                                                                                                                                                                    1. 2

                                                                                                                                                                      Ooh I didn’t know that.

                                                                                                                                                                      My recollection is that MT had a big performance advantage at the time over a lot of competing blogging software for a while because of the static generation approach.

                                                                                                                                                                    2. 2

                                                                                                                                                                      I truly believe that the static nature of MT was a disadvantage at the time, and purely DB-backed blog platforms/CMS’s like Wordpress cleaned their clock because of it.

                                                                                                                                                                      Now everyone who is cool sings the praises of static rendering.

                                                                                                                                                                      1. 3

                                                                                                                                                                        Well… it was really noticeable at the time I think that the blogs which were built on MT generally didn’t fall over when they got slashdotted. The ones which did things like making DB queries and calling sprintf() a lot in response to each HTTP request, did. :)

                                                                                                                                                                    3. 6

                                                                                                                                                                      My own static blog’s dependencies are Make (from 1976), m4 (from 1977), and rsync (a relative youngster from 1996).

                                                                                                                                                                      1. 2

                                                                                                                                                                        I would like to see this setup, in particular how you use make, if you’re willing to share.

                                                                                                                                                                        1. 3

                                                                                                                                                                          I am willing to share, but I’m not particularly willing to endorse or recommend my approach!

                                                                                                                                                                          In particular the listing stuff gets very ugly because m4 does not have any iteration capabilities; I have to define it myself using a recursive macro.

                                                                                                                                                                          1. 3

                                                                                                                                                                            Ha! Thanks. I was just curious how it worked. I always love to simple lightweight solutions using untrendy tools.

                                                                                                                                                                            1. 1

                                                                                                                                                                              Writing it was educational; now I know what not to do in the future! But in a situation where you don’t need any iteration, well then it’s great!

                                                                                                                                                                      2. 2

                                                                                                                                                                        I remember using PyBlosxom, which could do either static or dynamic generation around 2005? And it was a port of Blosxom, which was in Perl, and must have been a bit older.

                                                                                                                                                                        (Edit: Blosxom appears to have been started in 2002.)

                                                                                                                                                                        1. 1

                                                                                                                                                                          I still use Blosxom (OG Perl version) in static mode.

                                                                                                                                                                          It was dead easy to implement a Gemini site with it.

                                                                                                                                                                          => gemini://

                                                                                                                                                                          1. 2

                                                                                                                                                                            That’s awesome!

                                                                                                                                                                      3. 3

                                                                                                                                                                        and still not supported by IE

                                                                                                                                                                        Since IE has not been developed for years and never will be again, it will never support anything new because no new releases are coming.

                                                                                                                                                                        1. 1

                                                                                                                                                                          We are pretty much living an IRL genetic algorithm.

                                                                                                                                                                          OMG you’re right! This makes me feel simultaneously better and worse about the state of web development…

                                                                                                                                                                          1. 3

                                                                                                                                                                            “A world run by a bored god with one finger on the fast-forward button”, to (mis?)quote William Gibson.