1.  

    As a genuine question from someone who hasn’t used procedural programming productively before, what would be the benefits of a procedural language to justify its choice?

    1.  

      I would say less conceptual/cognitive overhead, but I don’t know if that’s something that can be said of this language as a whole, as I have no experience with it.

      By that I mean something like: I have a rough idea of what code I want from the compiler, how much mental gymnastics is required to arrive at the source-level code that I need to write?

      I would imagine that’s an important consideration in a language designed for game development.

      1.  

        Yeah, it makes perfect sense.

        To dumb down Kit’s value prop, it’s a “Better C, for people who need C (characteristics)”.

      2.  

        On top of alva’s comment, they compile fast and are easy to optimize, too.

        1.  

          I looked this up for some other article on lobste.rs. I found wikipedia to have a nice summary

          https://en.wikipedia.org/wiki/Procedural_programming

          Imperative programming

          Procedural programming languages are also imperative languages, because they make explicit references to the state of the execution environment. This could be anything from variables (which may correspond to processor registers) to something like the position of the “turtle” in the Logo programming language.

          Often, the terms “procedural programming” and “imperative programming” are used synonymously. However, procedural programming relies heavily on blocks and scope, whereas imperative programming as a whole may or may not have such features. As such, procedural languages generally use reserved words that act on blocks, such as if, while, and for, to implement control flow, whereas non-structured imperative languages use goto statements and branch tables for the same purpose.

          My understanding is that if you use say C you are basically using procedural language paradigms.

          1.  

            Interesting. So basically what was registering in my mind as imperative programming is actually procedural.

            Good to know. Thanks for looking it up!

            1.  

              I take “imperative” to mean based on instructions/statements, e.g. “do this, then do that, …”. An “instruction” is something which changes the state of the world, i.e. there is a concept of “before” and “after”. Lots of paradigms can sit under this umbrella, e.g. machine code (which are lists of machine instructions), procedural programming like C (where a “procedure”/subroutine is a high-level instruction, made from other instructions), OOP (where method calls/message sends are the instructions).

              Examples of non-imperative languages include functional programming (where programs consist of definitions, which (unlike assignments) don’t impose a notion of “before” and “after”) and logic programming (similar to functional programming, but definitions are more flexible and can rely on non-deterministic search to satisfy, rather than explicit substitution)

              1.  

                If functional programs don’t have a noton of before and after, how do you code an algorithm? Explain newton’s method as a definition.

                  1.  

                    both recursion and iteration say “do this, then do that, then do … “. And “let” appears to be assignment or naming so that AFTER the let operation a symbol has a meaning it did not have before.

                    open some namespaces
                    open System
                    open Drawing    
                    open Windows.Forms
                    open Math
                    open FlyingFrog
                    

                    changes program state so that certain operations become visible AFTER those lines are executed, etc.

                    1.  

                      It is common for computation to not actually take place until the result is immediately needed. Your code may describe a complicated series of maps and filters and manipulations and only ever execute enough to get one result. Your code looks like it describes a strict order the code executes in, but the execution of it may take a drastically different path.

                      A pure functional programming language wouldn’t be changing program state, but passing new state along probably recursively.

                      1.  

                        but you don’t really have a contrast with “imperative” languages - you still specify an algorithm. In fact, algorithms are all over traditional pure mathematics too. Generally the “state” being changed is on a piece of paper or in the head of the reader, but …

        1.  

          The only thing which stops me from diving into Emacs is a reliable terminal emulator. I know ansi-term, but I also know its bugs, like completely ignoring window size change and SIGWINCH forwarding.

          1. 5

            I went “full Emacs” some months ago, when I switched to Eshell as my main shell… I recommend giving it a try! I’ve not opened a regular terminal since, I think.

            Edit: the above was possible because over time Emacs has supplanted my other usages of a terminal:

            • Magit
            • mu & mu4e for email
            • ag-mode as grep replacement
            • dired for file/directory exploration
            • TRAMP for accessing remote files (as we’ve migrated to ephemeral machines without ssh access I don’t really use this any more)
            1.  

              But you’re probably not working with numerous remote machines, mainly baremetal, with no ability to safely export configuration files / profile to the destination user. I know TRAMP, but it’s not about remote file access, but remote execution/support.

              Also, even if I want to go “full Emacs”, there are still some advanced n/curses application which I don’t or can’t leave at all, often on these “remote machines”.

              Let’s say I just want a proper terminal emulation in Emacs – ansi-term is quite close to that. but it fails in full-screen sessions (basically misinterpreting/dropping ECMA “escape” codes), as well as missing SIGWINCH forwarding I mentioned. Of course I don’t force anyone to write it exclusively for me, but I’m kinda shocked there’s still no proper support for complete VT100/VT5xx capabilities in GNU Emacs, as it’s about 30+ years old.

              1.  

                It’s important to note that Eshell was not meant as a shell replacement by its authors, but as a way to easily get output from commands to a buffer, so it is very limited in features.

                That said, if it works for you, you do you!

              2. 5

                That’s a rather unconventional complaint about Emacs; normally it’s the other way around (“why would I want a terminal in my text editor?”) ;)

                You mentioned ansi-term, and others have mentioned Eshell, so I thought I’d mention that I use shell-mode for most commandline stuff. It’s a “dumb” terminal and hence can’t handle things like curses, but that lets it act more like a normal Emacs buffer (e.g. I can navigate and edit the content, without the terminal or running command intercepting my keypresses). It seems to handle window size changes and SIGWINCH perfectly well (i.e. ROWS and COLUMNS get updated).

                You’re right that Emacs is pretty bad at non-dumb terminals, like with ansi-term. I don’t bother with those modes at all and use a separate terminal (st + dtach + dvtm) for things which need that, like curses applications (that’s also much faster!). IMHO the value of Emacs is to have a standard way to navigate/search/edit buffers of text (plus easy scripting). TUIs like curses are more like character-based graphics than actual text, they’re not meant to be user navigable or editable (it will mess us the display and changes might get overwritten). Likewise applications which control user interaction (e.g. capturing key presses) don’t work well with the unified way of doing things (keybindings, etc.) that Emacs imposes.

                Note that there are many ways to do things with Emacs, so if terminals are your main pain point you could try (if you haven’t already) different mixes of components. For example, you could try running Emacs inside a terminal, or inside screen/tmux/dvtm so your terminals are alongside Emacs rather than inside it. You could use emacsclient/emacs –daemon to have multiple instances connected to a single session. You mention TRAMP in another comment; note that it can handle remote shells as well as just files. This is most obvious in eshell, where we can do cd /ssh:user@remote:/home/user and carry on as normal, with TRAMP sending our commands over to the remote machine. If we’re connected to a remote machine with TRAMP (e.g. in eshell, or dired) when we run M-x shell the resulting shell will be running on that machine, and things like tab-completion for filenames will be sent over TRAMP.

                1.  

                  This is probably the most comprehensive answer I’ll find in the web to this date, thank you :)

                  But, in shell-mode (M-x shell) $COLUMNS and $LINES aren’t updated at all. However, they do in M-x eshell. Pretty weird considering that first one knows a bit about ANSI/ECMA, while eshell does not.

                  TRAMP is a very nice tool even if I didn’t knew about remote execution, but now seems to be even better, thanks again :)

                  But still, sometimes I have the need to spawn these curses applications so I’m constantly looking for the alternative options - running external terminal emulator seems to be trade-off, but it’s still “out of the block” in Emacs world. And, when diving into Emacs, I took the “use Emacs for (almost) everything” approach as I think it should be taken this way.

                  There’s a thing about XEmbed, but I can’t find any reliable docummentation, seems to be experimental feature yet.

                  That EXWM might be great with xterm being fed by ELisp-structured data parsed into X resources, but it’ll require some additional code :)

                2. 5

                  One option is to use Emacs as a window manager via EXWM—then you can have an xterm as a buffer!

                  1.  

                    I could web-search for it, but I’d rather ask a person: is that one based on xwidget?

                    1.  

                      It’s actually just a window manager written in Emacs Lisp using a pure Emacs Lisp implementation of the X11 protocol!

                      1.  

                        Ah, like StumpWM, except elisp instead of common lisp.

                      2.  

                        No, it is a completely unrelated project

                    2.  

                      What specifically do you need terminal emulation for? In my experiance curses applications (like WICD) worked well enough, maybe being slightly slugish.

                    1. 3

                      I can’t remember who posted the interesting type safety idea of making sure that inches and meters were never combined.

                      1. 3

                        You can get this in haskell, along with a huge variety of other units, and conversions

                        http://hackage.haskell.org/package/dimensional

                        1. 1

                          I’m not sure if that one was me. I did push for it for miles and kilometers given we already lost a spacecraft to a conversion error. The technique is types that make things unique by default. Type Miles as Int and Kilometers as Int are treated as two, different types that have to be casted to ensure developer knows something could happen. Ada has this.

                          There’s also the possibility that they say distance in two modules from two teams with one being miles and one kilometers. That could’ve been how that spacecraft error played out. The types wouldn’t help. So, I further wanted a collection of various units such as miles or kilometers in the standard library where developers explicitly used them to catch more of these errors. That led me to Frink. I bookmarked it in case anyone wanted to work on something like this. It would probably be a nice, head start. Most of it is just formulas but some stuff like currencies are dynamic. Not sure best route for handling that.

                          1. 2

                            smart https://en.wikibooks.org/wiki/Ada_Programming/Type_System

                            I think Tedu may have described how to get something like that in C with structs

                            1. 1

                              I saw the same idea for handling user input. Functions that take user input would be typed as DirtyString or something to that effect and would need to be sanitized before being used elsewhere. For me, it was an enlightening use of types.

                              1. 2

                                I actually did that! It was like having safe vs unsafe in modules. Good for highlighting problems.

                                The other thing I used it for was Red/Black separation to stop leaks. You make sure stuff with secrets is Red, stuff that’s potentially-hostile is Black, watch CPU/cache/memory interactions, and type system helps with that as they get mixed at various points. I literally added it to variable names, though, which let me do it manually instead of writing a type checker.

                            2. 1

                              F# has that https://fsharpforfunandprofit.com/posts/units-of-measure/

                              I’ve never used F# though

                            1. 4

                              The base assumption is that libraries are installed in a shared space. Thats the root problem; is that we are in a mindset that HD space is rare and at a premium. That, in most problem domains, is false.

                              The better solution is for a program to query ldconfig for a version list of libraries for package X, and let the program choose the requested lib version. Yes, that does bloat the system for som definition of “bloat”. But as programs update, the system install program (dpkg/rpm) can make decisions to purge unneeded libs.

                              The big problem here is, that it does move smarts into ldconfig. Generally, the stupider the programs, the better.

                              1. 2

                                A similar problem (but with smaller ‘N’ I guess) occurs in linking packages into a binary/modules into a script. Unless the language supports having two versions of a lib linked in the same binary at the same time, I think you still have the same problem.

                                1. 1

                                  Indeed. We’re dealing with this in my company too. Devs keep wanting to push and use the newest powershell, where some of our clients don’t play the MS upgrade treadmill regularly.

                                  I’ve asked them to incorporate a version check at the start of the script, to at least bail if script_version is greater than installed. Ideally, they shouldn’t be pushing the newest and shiniest thing unless there’s a direct imperative to do so. But alas..

                                2. 1

                                  This is pretty much how Nix works. It does take up more disk space, but that’s usually not a problem (my laptop was running low on space, so I’m currently running nix-store --optimise to hardlink duplicate files; freed up 50GB so far).

                                  Another important difference between Nix and dpkg/rpm/etc. is that Nix doesn’t do dependency solving. However, it is possible to have Nix run shell scripts as part of its evaluation phase (where it’s calculating what package to use); this is known as “import from derivation” and is generally frowned upon, but I’ve written Nix functions which will e.g. call out to cabal to solve the dependency versions of a Haskell package :)

                                1. 1

                                  My first thought was “how did you start your commit messages with #, when that’s the comment character; is there some weird way to escape it, and why bother?”. Then I remembered that git can be used without magit :P

                                  I didn’t know about the default comment character choice, but stuff like that smells really off to me. I get why it’s there, but I wouldn’t like to be pushing things that far from ‘single responsibility’. I also think enabling “auto” features by default would break the principle of least surprise; maybe better to have it look through the commit history.

                                  1. 8

                                    Always a joy to read Conor’s writing :)

                                    Note that Epigram has been dead for a while, Idris is its spiritual successor (I believe it actually evolved from an attempt to build an Epigram compiler). Idris is explicitly aiming to be a “real” programming language; Agda is very similar, but is more often used from a mathematical/logical side of Curry-Howard, rather than the programming side.

                                    Neither Idris or Agda have the 2D syntax of Epigram, but they both have powerful Emacs modes which can fill in pieces of code (Haskell’s “typed holes” is the same idea, but (as this paper demonstrates) Haskell’s types are less informative).

                                    1. 10

                                      Indeed, I suppose it’s that Idris evolved from what was intended to be the back end of Epigram. It certainly owes a lot to Conor McBride and James McKinna’s work on Epigram. I don’t know if “real programming language” is exactly the right way to put it, though, so much as being the language I wanted to have to explore the software development potential of dependent types. Maybe “real” will come one day :).

                                      1. 1

                                        Will we see a longer form post/paper or something that isn’y merely Twitter teasers about Blodwen anytime soon? :)

                                        1. 6

                                          Hopefully! I know I need to get on with writing things so that more people can join in properly. I have plenty of time to work on it for a change in the next few months, so that’s nice…

                                          1. 1

                                            Do you have a writeup about it? I’m wondering why you’re replacing Idris which is somewhat established already, I mean that probably is the reason you’re replacing it, but still I wonder what concretely necessitated a whole new language instead of a 2.0

                                            1. 5

                                              It isn’t a whole new language, it’s a reimplementation in Idris with some changes that experience suggests will be a good idea. So it’s an evolution of Idris 1. I’ll call it Idris 2 at some point, if it’s successful. It’s promising so far - code type checks significantly faster than in Idris 1, and compiled code runs a bit faster too.

                                              Also, I’ve tried to keep the core language (which is internally called ‘TTImp’ for ‘type theory with implicits’) and the surface language cleanly separated. This is because I occasionally have ideas for alternative surface languages (e.g. taking effects seriously, or typestate, or maybe even an imperative language using linear types internally) and it’ll be much easier to try this if I don’t have to reimplement a dependent type checker every time. I don’t know if I’ll ever get around to trying this sort of thing, but maybe someone else will…

                                              I started this because the Idris implementation has a number of annoying problems (I’ll go into this some other time…) that can only be fixed with some pretty serious reengineering of the core. So I thought, rather than reengineer the core, it would be more fun to see (a) if it was good enough to implement itself, and (b) if dependent types would help in any way.

                                              The answer to (a) turned out to be “not really, but at least we can make it good enough” and to (b) very much so, especially when it comes to name manipulation in the core language, which is tricky to get right but much much easier if you have a type system telling you what to do.

                                              I don’t have any writeup on any of this yet. It’ll happen eventually. (It has to, really - firstly because nobody ever made anything worthwhile on their own so a writeup is important for getting people involved, and secondly because it’s kind of what my job is :))

                                          2. 2

                                            I just follow along with the commits. Edwinb is usually pretty good with his commit messages, so you can kind of get a story of the development from that! :)

                                            1. 1

                                              I’ve got to admit it’s very weird reading a reply by someone with your identical name/spelling, thanks!

                                            2. 1

                                              What’s Blodwen?

                                              1. 2

                                                An implementation of Idris in Idris: https://github.com/edwinb/Blodwen/

                                                Has a neat implementation of Quantitative Type Theory that I’m hoping to integrate in my own programming language!

                                                1. 1

                                                  Nice! What’s your language? Btw your second link is broken

                                                  1. 3

                                                    Fixed! This is mine: https://github.com/pikelet-lang/pikelet - scratching my itch of Rust not being enough like Idris, and Idris being not designed with low level systems programming in mind. Probably won’t amount to much (it’s rather ambitious), but it’s been fun playing around, learning how dependent type checkers work! I still need to learn more about what Epigram and Idris do, but it takes passes of deepening to really get a handle on all the stuff they learned. I’m probably making a bunch of mistakes that I don’t know about yet!

                                                    1. 1

                                                      Nice logo. Looks harmless and fun.

                                                    2. 2
                                            3. 1

                                              Thanks for putting this in context, that’s really useful.

                                              Also: sounds like I’m missing a (200x) in the title, if you know the correct year.

                                            1. 8

                                              The browser vendors weren’t willing to change their products be compliant with the RFC, because websites already rely on the non-compliant behavior. They wrote their own spec that matches the behavior that the web relies on. That behavior apparently cannot be modeled using EBNF.

                                              If you can write a grammar that matches the behavior of Internet Explorer, Chrome, and Firefox well-enough to not break web pages that currently work with all of them, I’m sure they’d love to see it.

                                              1. 2

                                                Yea, I find the sentiment being expressed here kind of silly. “Browser spec authors should give up on actually caring about writing specs that (a) actually standardize behavior, as opposed to describing some ideal that no one implements, and (b) maintain backwards compatibility with the actual web because it makes it hard to implement URI parsing for my hobby project!”

                                                The implication is also that the spec writers are either dumb or purposely writing bad specs.

                                                Maybe your hobby project doesn’t need to work with real world data, but if a spec writer decides things should just stop working, no browser will implement the spec. And if a browser decides the same, people will just stop using that browser.

                                                It’s unfortunate that the state of things is like this. But people still write browser engines and spec compliant URI parsers from scratch in spite of this, because their goals are aligned with that of the spec—to work with real world data.

                                                If you don’t care about parsing all the URIs browsers need to parse, there’s absolutely no shame in only parsing a restricted syntax. In fact, depending on your problem domain, this might be better engineering.

                                                1. 2

                                                  I would say that this is a good use-case for warnings. Browsers (especially Chrome) have been discouraging poor practices for a while now. For example, HTTP pages don’t get the “padlock” that a significant proportion of users have been trained to look for; then HTTP resources on HTTPS pages were targetted; and now HTTP itself gets flagged as “not secure”.

                                                  If it makes sense to flag the entire HTTP Web, then I’d find it perfectly acceptable to (gradually) roll this out for malformed data too; e.g. if a page’s URL doesn’t follow a formal spec, if its markup or CSS is malformed, etc. This is justifiable in a few ways:

                                                  • As the article’s langsec argument states: if data isn’t validated against a (decidable) formal language before getting processed, there is potential for exploits.
                                                  • In the case of backwards compatibility, we can consider “legacy” content (created before the relevant specs) as if it complies with some hypothetical previous spec, like “Works for me in Netscape Navigator”. In which case that hypothetical spec is out of date, in the same way that e.g. Windows98 is out of date: it’s no longer maintained, and won’t receive any security patches. Hence it’s legitimate to consider such legacy formats as not secure.
                                                  • In the case of forwards compatibility, it can still be considered as not secure, since we’re skipping parts of the data we don’t understand, which were presumably added for a reason and may be needed for security.
                                                  1. 2

                                                    I agree with you in principle, but I don’t think this is a good idea in practice.

                                                    As the article’s langsec argument states: if data isn’t validated against a (decidable) formal language before getting processed, there is potential for exploits.

                                                    Using yacc, say, instead of writing your own recursive descent parser, doesn’t magically make your parser safe—in fact, far from it. I don’t buy the safety argument at all, having had the joy of encountering bizarre parser generator bugs.

                                                    Moreover the point is that the URI grammar here can’t be described as a CFG. So forget about even using a nice standard LR parser generator that outputs to a nice automaton, so the idea of having security by relying on parser generators used by lots of other people is moot.

                                                    In the case of backwards compatibility, we can consider “legacy” content (created before the relevant specs) as if it complies with some hypothetical previous spec, like “Works for me in Netscape Navigator”. In which case that hypothetical spec is out of date, in the same way that e.g. Windows98 is out of date: it’s no longer maintained, and won’t receive any security patches. Hence it’s legitimate to consider such legacy formats as not secure.

                                                    And now you have two parsers, so not only are any potential security exploits are still there, but it is now even more likely that you’ll forget about some edge case.

                                                    1. 1

                                                      I agree, but note that I’m not arguing that something or other is secure. Rather, I’m saying that adding a “Not Secure” warning might be justifiable in some cases, and this lets us distinguish between the extreme positions of “reject anything that disagrees with the formal spec” and “never remove any legacy edge-cases”.

                                                      The usual response to warnings is that everybody ignores them, but things like “Not Secure” seem to be taken seriously by Web devs. Google in particular has used that to great effect by incentivising HTTPS adoption, disincentivising HTTP resources on HTTPS pages, etc.

                                                      And now you have two parsers, so not only are any potential security exploits are still there, but it is now even more likely that you’ll forget about some edge case.

                                                      That is indeed a problem. Splitting parsing into a “strict parser” and “lax parser” can obviously never remove exploits, since we’re still accepting the old inputs. It can be useful as an upgrade path though, to “ratchet up” security. It gives us the option to disable lax parsing if desired (e.g. a ‘hardened mode’), it allows future specs to tighten up their requirements (e.g. in HTTP 3, or whatever), it allows user agents which don’t care about legacy inputs to avoid these problems entirely, etc.

                                                      I definitely agree that having backwards compatible users agents using multiple parsers is bad news for security in general, since as you say it increases attack surface and maintenance issues. That burden could be reduced with things like parser generators (not necessarily yacc; if it’s sufficiently self-contained, it could even be formally verified, like the “fiat cryptography” code that’s already in Chrome), but it’s certainly real and is probably the biggest argument against doing this in an existing general-purpose user agent

                                                      1. 2

                                                        I see. Yea, it seems like some way to deprecate weird URLs would be best. Just not sure how to do it. “Not Secure” seems like a reasonable idea, but I wonder if it’d become overloaded.

                                              1. 1

                                                I use bash on NixOS with whatever completion it provides by default (program names in PATH and filenames in current directory). I also use most shells from within Emacs shell-mode, which I think does its own completion.

                                                I get incredibly frustrated by certain completion systems, I think Debian’s default, which will sometimes refuse to tab-complete filenames unless they contain certain patterns. For example mplayer or vlc commands will only tab-complete filenames if they contain strings like .mp4 or .avi. I find such behaviour surprising and obnoxious, so would highly recommend not doing that in other tab-completion systems :)

                                                1. 1

                                                  Yeah I get frustrated with them too. It’s not so much that they only complete .mp4 only by default, but if you want to change that, you are confronted with a nightmare of inscrutable and untestable code !!!

                                                1. 4

                                                  Just once, I would like to see one of these TDD proponents tackle an actual hard problem, such as writing a device driver (which includes an interrupt handler). I’d love to see how TDD would work in such a situation.

                                                  1. 1

                                                    I’m not a TDD proponent, and I’ve never written device drivers, but I tend to test the impact of asynchronous external events (e.g. interrupts) using generative testing; e.g. using QuickCheck to generate a sequence of plausible or pathological actions as part of the test’s inputs, and firing them during the test’s execution of the code. This can use dummy or mock I/O actions, since we’re mostly testing the logic; or we can run in a standalone simulator/interpreter/VM.

                                                    I believe that this is quite common for testing networked systems (where the “actions” are incoming packets or requests, rather than interrupts).

                                                    1. 1

                                                      Not really about device drivers, but Mike Bland wrote a series on how to test the openssl code that was involved in heartbleed: https://martinfowler.com/articles/testing-culture.html I guess this subject matter could be considered harder than bog-standard business CRUD.

                                                    1. 12

                                                      On one hand, I’m sympathetic to the idea of bringing systems programming more in line with “systems thinking” and “systems theory” in other fields. On the other, I think under that definition of “systems programming” we don’t have any truly systems-oriented languages yet.

                                                      1. 7

                                                        That’s actually what Oil is supposed to be! (eventually) Maybe this connection isn’t obvious, but one way to think of it is:

                                                        • A Unix shell is a language for describing processes on a single machine.
                                                        • A distributed system is a set of Unix processes spread across multiple machines. [1]
                                                        • So you can imagine extending shell to talk about processes on multiple machines. There are existing languages that do this, although sometimes they are only “config files”. I believe you need the expressiveness of a full language, and an extension of the shell is the natural candidate for that language.

                                                        I briefly mentioned this 18 months ago, in the parts about Borg/Kubernetes: Project Goals and Related Projects

                                                        But I haven’t talked about it that much, because I want to keep things concrete, and none of this exists.

                                                        I also mentioned Kubernetes in this blog post: Why Create a New Unix Shell?

                                                        The build time and runtime descriptions of distributed systems are pretty disjoint now, but I think they could be moved closer together. Build time is a significant problem, not just a detail.

                                                        I mentioned a couple books on the philosophy of systems here: Philosophy of Systems


                                                        I like the OP’s framing of things. I agree that “systems programming” is overloaded and there should be another word for describing the architecture of distributed systems.

                                                        Although I guess I totally disagree with the conclusion about OCaml and Haskell. I’m pretty sure we are talking about the same thing, but maybe not exactly.

                                                        I guess he defining a “system” with the 5 qualities, which I agree with, but I am picking out “distributed systems” as an important subset of systems that have those 5 qualities.

                                                        My basic thesis is that Shell should be the language for describing the architecture of distributed systems. The architecture is essentially a set of processes and ports, and how they are wired together. And how they can be applied to a particular hardware/cluster configuration.

                                                        Right now those two things are heavily entangled. We’re basically still in the era where you have to modify your (distributed) program to run it on a different computer (cluster).

                                                        Concretely, I think a cleaner shell mostly needs Ruby-like blocks, and it can express a lot of things, to do stuff like this:

                                                        https://www.terraform.io/docs/configuration/syntax.html

                                                        … which looks pretty similar to Google’s (internal) Borg configuration language. I described that language to someone as roughly “JSON with map/filter/cond/inheritance” :) It evaluates to protocol buffers that get send to the Borg master, and then flags get sent to the Borg “slaves” on each machine.

                                                        Kubernetes has almost exactly the same architecture as far as I can tell, but everybody seems to use Go templates to generate YAML. That is not a good description language! :-( Things seem to have gone backward in this respect when the tech made its way out of Google (where I used all this cluster / big data stuff for many years).


                                                        Also, I’m not the only one who thinks this. I link to Why Next Generation Shell? in my FAQ. He uses the term “systems engineer” too, which is also overloaded.


                                                        [1] And they really are Unix processes; it’s hard to think of any significant non-Unix distributed systems. I guess there are still many mainframes in similar roles, but I imagine the number of nodes is fairly small compared to Unix-based systems.

                                                        1. 4

                                                          Feature request for oil. currying of commands…

                                                          let targz = tar args… | gzip

                                                          I think a proper ‘systems language’ would let composing OS processes be trivial while looking a bit like ocaml or reason ml. If I designed something i would consider having a clean distinction between functions and processes. Then let processes be first class things like functions, that can be curried, passed as arguments generated on the fly, just like closures in functional languages.

                                                          The difference between ‘proc’ and ‘func’ would be procs can interop with OS processes and can die or be cancelled.

                                                          and a static type system…

                                                          1. 3

                                                            “I think a proper ‘systems language’ would let composing OS processes be trivial while looking a bit like ocaml or reason ml.”

                                                            I’ve said this, too, but in the context of getting rid of pipes. I thought modularity and composition were good but defaulting on the processes and pipes weren’t. At the least, we should have a choice. It seems we can specify what’s supposed to happen at a high level with the implementation (eg pipes, function calls) being generated later. The developer enters the criteria for that. So, we start getting the benefits of high-level languages plus can get close to specific functionality like how UNIX works. Might avoid maintenance and security issues, too.

                                                            1. 3

                                                              Oil will have proc and func, with exactly those keywords:

                                                              http://www.oilshell.org/blog/2017/02/05.html

                                                              proc is identical to current shell functions, which take argv and return an exit code, and can also be transparently put inside a pipeline or run in a subshell/command sub. It’s a cross between a procedure and a process.

                                                              func is basically like Python or JavaScript functions.

                                                              As for currying, I’d have to see some examples. I don’t see why a normal function syntax doesn’t solve the problem.

                                                              By the way, a complex service at Google can have over 10K lines of the Borg config language, briefly mentioned here:

                                                              https://research.google.com/pubs/pub43438.html?hl=es

                                                              It is actually a functional language – it’s sort of like JSON with map/filter/cond/lambda/inheritance. And practically speaking, the functional style doesn’t really add anything. Most people seem to complain about masses of highly nested curly braces and awkward lambdas of map and filter. Plenty of people at Google have background in functional languages and they don’t even like it as a “config” syntax.

                                                              Some teams went as far as to develop and alternate Python-derived language for describing service configs instead. A coworker actually made a statically typed variant which was abandoned.

                                                              I basically think there is a confusion between functional-in-the-small and functional-in-the-large. I don’t care about functional-in-the-small – map/filter/cond etc. can be written in an imperative style. However functional-in-the-large is very important in distributed systems. It lets you compose processes and reason about them.

                                                              And you can write pure functions in an imperative style! In fact that is how most of my programs including Oil are written. They use zero mutable globals, and pure dependency injection of I/O and state, which is essentially equivalent to functional programming.

                                                              More concretely, I want Oil to be familiar to both existing shell users, and users of common languages like Python, JavaScript, Go, etc. I’m trying not to invent any new syntax – it should all be borrowed from a popular language. I think Reason ML is great and they make some good critiques of the inconsistency of OCaml syntax, and they bring it closer to JavaScript. So Oil might look more like Reason ML than OCaml.

                                                              1. 1

                                                                Main advantage of functional vs imperative for me is the first is easier get get correct the first time or during refactorings. This article has a good explanation why functional programs are easier to verify than imperative.

                                                              2. 2

                                                                Whilst it’s a nice idea, for that example I would use a bash function:

                                                                function targz {
                                                                  tar "$@" | gzip
                                                                }
                                                                

                                                                Also, I’ve become less fond of command line arguments over time, and tend to prefer env vars instead for key/value things. That way we don’t have to care about their order, we don’t need to match up keys with values ourselves, they’re automatically propagated through wrapper scripts, etc. I tend to only use command lines for inherently sequential things, like a list of filenames to act on.

                                                                I’m not sure if ‘currying environments’ makes sense, although something like Racket’s parameterize for env vars in the shell would be nice. I’ve actually written some Racket machinery which lets me set env vars this way when invoking subprocesses :)

                                                                1. 1

                                                                  I am interested in another example of use case for this.

                                                                  1. 3

                                                                    Try writing a Go program that does a lot of shelling out to commands like ssh, tar, gzip, gsutil or awscli , then try the same in bash. In bash you will have crappy programming experience ‘in the large’, with go you will have an overly verbose mess ‘in the small’.

                                                                    1. 2

                                                                      Yeah writing shell scripts in Go seems to be more and more common these days, because Go is a common “cloud service” language, and a lot of devs don’t know shell, or (understandably) want to avoid it.

                                                                      My friend sent me a shell script rewritten in Go. I get why. It works, but it’s clunky.

                                                                      Here’s another example:

                                                                      https://jvns.ca/blog/2017/07/30/a-couple-useful-ideas-from-google/

                                                                      Here’s a post about rewriting shell in Python that I link from my FAQ. IMO it inadvertenty proves the opposite point: Python is clunky for this use case!

                                                                      https://medium.com/capital-one-developers/bashing-the-bash-replacing-shell-scripts-with-python-d8d201bc0989

                                                                2. 1

                                                                  Personally, I’m keeping my fingers crossed hard for Luna language to (eventually) become the language of distributed computing… and of scaling the ladder of abstraction in both ways…

                                                                3. 4

                                                                  Agreed. But don’t expect the languages to come before the thinking and theories!

                                                                  1. 2

                                                                    VDM with code generator? ;) Also, ASM’s have been used to model everything. One, Asmeta, is a programming language, too. So, it seems doable. I’m not going to say pragmatic, though.

                                                                  1. 5

                                                                    While functional programming languages like Haskell are conducive to modularity and otherwise generally good software engineering practices, they are unfit as implementation languages for what I will call interactive systems. These are systems that are heavily IO bound and must provide some sort of guarantee with regard to response time after certain inputs. I would argue that the vast majority of software engineering is the engineering of interactive systems, be it operating systems, GUI applications, control systems, high frequency trading, embedded applications, databases, or video games. Thus Haskell is unfit for these use cases. Haskell on the other hand is a fine implementation language for batch processing, i.e. non-interactive programs where completion time requirements aren’t strict and there isn’t much IO.

                                                                    It’s not a dig at Haskell, this is an intentional design decision. While languages like Python/Java remove the need to consider memory allocation, Haskell takes this one step further and removes the need to consider the sequential steps required to execute a program. These are design trade-offs, not strict wins.

                                                                    1. 5

                                                                      While languages like Python/Java remove the need to consider memory allocation, Haskell takes this one step further and removes the need to consider the sequential steps required to execute a program.

                                                                      Haskell makes it necessary to explicitly mark code which must be performed in sequence, which, really, is a friendlier way of doing things than what C effectively mandates: In C, you have to second-guess the optimizer to ensure your sequential code stays sequential, and doesn’t get reordered or removed entirely in the name of optimization. When the IO monad is in play, the Haskell compiler knows a lot of its usual tricks are off-limits, and behaves itself. It’s been explicitly told as much.

                                                                      Rust made ownership, previously a concept which got hand-waved away, explicit and language-level. Haskell does the same for “code which must not be optimized as aggressively”, which we really don’t have an accepted term for right now, even though we need one.

                                                                      1. 8

                                                                        The optimiser in a C implementation absolutely won’t change the order in which your statements execute unless you can’t observe the effect of such changes anyway. The definition of ‘observe’ is a little complex, but crucially ‘my program is faster’ isn’t an observation that counts. Your code will only be reordered or removed in the name of optimisation if such a change is unobservable. The only way you could observe an unobservable change is by doing things that have no defined behaviour. Undefined behaviour exists in Haskell and Rust too, in every language.

                                                                        So I don’t really see what this has to do with the concept being discussed. Haskell really isn’t a good language for expressing imperative logic. You wouldn’t want to write a lot of imperative logic in Haskell. It’s very nice that you can do so expressively when you need to, but it’s not Haskell’s strength at all. And it has nothing to do with optimisation.

                                                                        1. 3

                                                                          What if you do it using a DSL in Haskell like Galois does with Ivory? Looks like Haskell made their job easier in some ways.

                                                                          1. 1

                                                                            Still part of Haskell and thus still uses Haskell’s awful syntax. Nobody wants to write a <- local (ival 0). or b' <- deref b; store a b' or n `times` \i -> do when they could write int a = 0;, a = *b; or for (int i = 0; i < n; i++).

                                                                            1. 8

                                                                              “Nobody wants to”

                                                                              You’re projecting your wishes onto everybody else. There’s piles of Haskell code out there, many DSL’s, and some in production. Clearly, some people want to even if some or most of us don’t.

                                                                              1. 1

                                                                                There is not ‘piles of Haskell code out there’, at least not compared to any mainstream programming language. Don’t get confused by its popularity amongst people on lobsters, hackernews and proggit. It’s an experimental research language. It’s not a mainstream programming language. It has piles of code out there compared to Racket or Idris or Pony, but compared to Python or C or C++ or Ruby or Java or C# or god forbid Javascript? It might as well not exist at all.

                                                                                1. 2

                                                                                  Im not confused. Almost all languages fail getting virtually no use past their authors. Next step up get a few handfuls of code. Haskell has had piles of it in comparison plus corporate backing and use in small scale. Then, there’s larger scale backings like Rust or Go. Then, there’s companies with big market share throwing massive investments into things like .NET or Java. There’s also FOSS languages that got lucky enough to get similarly high numbers.

                                                                                  So, yeah, piles of code is an understatement given most efforts didnt go that far and a pile of paper with source might not cover the Haskell out there.

                                                                                  1. 1

                                                                                    I don’t care how popular Haskell is compared to the vast majority of languages that are used only by their authors. That’s completely irrelevant to the discussion at hand.

                                                                                    Haskell is not a good language for expressing imperative concepts. That’s plainly and obviously true. Defending it on the basis that it’s widely used ignores that firstly languages aren’t better simply because they’re widely used, secondly that languages can be widely used without necessarily being good at expressing imperative concepts, and thirdly that Haskell isn’t widely used.

                                                                              2. 4

                                                                                int a = 0 is okay, but not great. a = *b is complete gobbledygook that doesn’t look like anything unless you already know C, but at least it’s not needlessly verbose.

                                                                                for (int i = 0; i < n; i++) is needlessly verbose and it looks like line noise to anyone who doesn’t already know C. It’s a very poor substitute for actual iteration support, whether it’s n.times |i| or for i in 0..n or something else to express your intent directly. It’s kind of ridiculous that C has special syntax for “increment variable by one and evaluate to the previous value”, but doesn’t have special syntax for “iterate from 0 to N”.

                                                                                All of that is kind of a minor nit pick. The real point is that C’s syntax is not objectively good.

                                                                                1. 2

                                                                                  How in the world are people unfamiliar with ruby expected to intuit that n.times|i| means replace i with iterative values up to n and not multiply n times i?

                                                                                  1. 2

                                                                                    A more explicit translation would be 0.upto(n) do |i|.

                                                                                  2. 0

                                                                                    You do know C. I know C. Lots of people know C. C is well known, and its syntax is good for what it’s for. a = *b is not ‘gobbledygook’, it’s a terse way of expressing assignment and a terse way of expressing dereferencing. Both are very common in C, so they have short syntax. Incrementing a variable is common, so it has short syntax.

                                                                                    That’s not ridiculous. What I am saying is that Haskell is monstrously verbose when you want to express simple imperative concepts that require a single character of syntax in a language actually designed around those concepts, so you should use C instead of Haskell’s weird, overly verbose and syntactically poor emulation of C.

                                                                            2. 3

                                                                              How does Haskell allow you to explicit mark code that must be performed in sequence? Are you referring to seq? If you’re referring to the IO Monad, it’s a fair point, but I think generally it’s considered bad practice to default to using the IO monad. This sort of thing creates a burden when programming Haskell, at least for me. I don’t want to have to constantly wonder if I’ll need to port my elegant functional code into sequential IO Monad form in the future. C++/Rust address this sort of decision paralysis via “zero-cost abstractions,” which make them both more fit to be implementations languages, according to my line of reasoning above.

                                                                              1. 5

                                                                                Personally, I dislike discussions involving “the IO Monad”. The key point is that Haskell uses data flow for control flow (i.e. it’s lazy). We can sequence one thing after another by adding a data dependency (e.g. making bar depend on the result of foo will ensure that it runs afterwards).

                                                                                Since Haskell is pure, compilers can understand and optimise expressions more thoroughly, which might remove ‘spurious’ data dependencies (and therefore sequencing). If we want to prevent that, we can use an abstract datatype, which is opaque to the compiler and hence can’t be altered by optimisations. There’s a built-in datatype called IO which works well for this (note: none of this depends at all on monads).

                                                                                1. 3

                                                                                  The trouble is that oftentimes when you’re building time-sensitive software (which is almost always), it’s really inconvenient if the point at which a function is evaluated is not clear from the source code. Since values are lazy, it’s not uncommon to quickly build up an entire tree of lazy values, and then spend 1-2 seconds waiting for the evaluation to complete right before the value is printed out or displayed on the screen.

                                                                                  You could argue that it’s a matter of setting correct expectations, and you’d be right, but I think it defeats the spirit of the language to have to carefully annotate how values should be evaluated. Functional programming should be about functions and pure computation, and there is no implicit notion of time in function evaluation.

                                                                                  1. 4

                                                                                    I agree that Haskell seems unsuitable for what is generally called “systems programming” (I’m currently debugging some Haskell code that’s been over-complicated in order to become streaming). Although it can support DSLs to generate suitable code (I’ve not experience with that though).

                                                                                    I was just commenting on using phrases like “the IO Monad” w.r.t. evaluation order, etc. which is a common source of confusion and hand-waving for those new to Haskell, or reading about it in passing (since it seems like (a) there might be something special about IO and (b) that this might have something to do with Monads, neither of which are the case).

                                                                                    1. 2

                                                                                      building time-sensitive software (which is almost always)

                                                                                      Much mission-critical software is running in GC’d languages whose non-determinism can kick in at any point. There’s also companies using Haskell in production apps that can’t be slow. At least one was using it specifically due to its concurrency mechanisms. So, I don’t think your “almost always” argument holds. The slower, less-predictable languages have way too much deployment for that at this point.

                                                                                      Even time-sensitive doesn’t mean what it seems to mean outside real-time since users and customers often tolerate occasional delays or downtime. Those they don’t might also be fixed with some optimization of those modules. Letting things be a bit broken fixing them later is default in mainstream software. So, it’s not a surprise it happens in lots of deployments that supposedly are time-critical as a necessity.

                                                                                      In short, I don’t think the upper bounds you’ve established on usefulness match what most industry and FOSS are doing with software in general or timing-sensitive (but not real-time).

                                                                                      1. 2

                                                                                        Yeah it’s a good point. There certainly are people building acceptably responsive apps with Haskell. It can be done (just like people are running go deployments successfully). I was mostly speaking from personal experience on various Haskell projects across the gamut of applications. Depends on cost / benefit I suppose. For some, the state of the art type system might be worth the extra cycles dealing the the occasional latency surprise.

                                                                                        1. 2

                                                                                          The finance people liked it because it was both closer to their problem statements (math-heavy), the apps had lower defects/surprises vs Java/.NET/C, and safer concurrency. That’s what I recall from a case study.

                                                                                  2. 1

                                                                                    If you’re referring to the IO Monad, it’s a fair point, but I think generally it’s considered bad practice to default to using the IO monad

                                                                                    Lmao what? You can define >>= for any data type effectively allowing you to create a DSL in which you can very precisely specify how the elements of the sequence combine with neat do notation.

                                                                                    1. 2

                                                                                      Yes that’s exactly the problem to which I’m referring: Do notation considered harmful. Also do notation isn’t enough to specify evaluation sequencing since values are lazy. You must also carefully use seq

                                                                                      1. 1

                                                                                        Ah well I use a Haskell-like language that has strict-by-default evaluation and seems to be able to address a lot of those other concerns at least by my cursory glance:)

                                                                                        Either way the benefits of do, in separating the logic and execution of procedures, look great to me. But I may be confusing them with the benefits of dependent typing, nevertheless the former facilitates the latter when it comes to being able to express various constraints on a stateful system.

                                                                                2. 3

                                                                                  For systems Haskell, you might like Habit from the people behind House, a Haskell OS. I just found some answers to timing part that I’ll submit in morning.

                                                                                  1. 1

                                                                                    The House website seems incredibly out of date!

                                                                                    1. 3

                                                                                      Oh yeah. It’s mostly historical. They dropped the work for next project. Then dropped that for even better one. We get some papers and demos out of it.

                                                                                      1. 2

                                                                                        But so damn cool.

                                                                                        1. 2

                                                                                          Exactly! Even more so, there’s a lot of discussion of how to balance the low-level access against Haskell’s high-level features. They did this using the H Layer they describe in some of their papers. It’s basically like unsafe in Rust where they do the lowest-level stuff in one way, wrap it where it can be called by higher-level Haskell, and then do what they can of the rest in Haskell. I figured the concepts in H Layer might be reusable in other projects, esp safe and low-level. The concepts in Habit might be reusable in other Haskell or non-Haskell projects.

                                                                                          It being old doesn’t change that. Good example is how linear logic was in the 1980’s, That got used in ML first I think years later, then them plus singleton types in some safer C’s in the 2000’s, and an affine variant of one of them in Rust. They make a huge splash with “no GC” claim. Now, linear and affine types are being adapted to many languages. The logic is twenty years old with people talking about using it for language safety for 10-20 years. Then, someone finds it useful in a modern project with major results.

                                                                                          Lots of things work that way. It’s why I submit older, detailed works even if they have broken or no code.

                                                                                    2. 1

                                                                                      none of the examples of “interactive systems” you mention are nomally io bound. sub-second response time guarantees, otoh, are only possible by giving up gc, and use a real-time kernel. your conclusion that Haskell is unusable for “these use cases” seems entirely unfounded. of course, using Haskell for real time programming is a bad idea, but no less bad than anything that’s, essentially, not C.

                                                                                      1. 2

                                                                                        I’ve had a few personal experiences writing large Haskell applications where it was more trouble than I thought it was worth. I regularly had to deal with memory leaks due to laziness and 1-5 second stalls at io points where large trees of lazy values were evaluated last minute. I said this in another thread: it can be done, it just requires a bit more effort and awareness. In any case, I think it violates the spirit of Haskell programming to have to carefully consider latency issues, GC times, or lazy value evaluation when crafting pure algorithms. Having to trade off abstraction for performance is wasteful IMO, i think Rust and C++ nail this with their “zero cost abstractions.”

                                                                                        I would label most of those systems IO bound. My word processor is normally waiting on IO, so is my kernel, so is my web app, so is my database, so is my raspberry pi etc.

                                                                                        1. 1

                                                                                          I guess I’m picking nits here, but using lots of working memory is not “memory leaks”, and a program that is idling due to having no work to perform is not “io bound”. Having “to carefully consider latency issues, GC times, [other tradeoffs]” is something you have to do in every language. I’d venture that the ability to do so on a subconcious level is what distinguishes a skilled developer from a noob. This also, I think, plays a large part in why it’s hard for innovative/weird languages to find adoption; they throw off your sense of how things should be done.

                                                                                          1. 1

                                                                                            Yes you have to consider those things in all languages which is precisely my point. Haskell seeks to abstract away those details but if you want to use Haskell in any sort of “time-sensitive” way, you have to litter your pure, lazy functional code with annotations. That defeats the purpose of the language being pure and lazy.

                                                                                            And yes, waiting on user input does make your program IO bound. If your program is spending more time waiting on IO and less time churning the CPU, it is IO bound. IO bound doesn’t simply mean churning the disk.

                                                                                          2. 1

                                                                                            I brought that up before as a counterpoint to using Haskell. A Haskeller gave me this link which is a setting for making it strict by default. Might have helped you out. As a non-Haskeller, I can’t say if it makes the language harder to use or negates its benefits. Worth looking into, though, since it was specifically designed to address things like bang patterns that were cluttering code.

                                                                                      1. 3

                                                                                        Formal methods do not require any setup code because they are not tied to the actual components under test.

                                                                                        This is what makes me a bit wary of verifying things outside of the program itself (unlike e.g. types or contracts). We have a bunch of tools like doctest because we can’t even manage to keep docstring examples in sync with the implementation that occurs right below them.

                                                                                        Are there ways to bridge such gaps, that could be added to a CI pipeline alongside other tests and verifiers (i.e. checking that the implementation is making transitions that the model allows, checking that the model’s transitions are actually possible in the implementation)?

                                                                                        1. 3

                                                                                          This is the main concern that’s come up as I’ve been sharing TLA+ with coworkers.

                                                                                          The purpose of doc strings is to give meaning to your code (i.e explain the implementation). To fulfill their purpose, docs have to stay in sync. The problem is there is no automated way to evaluate if they are in sync because they are words and, as you pointed out, we’re bad at checking them.

                                                                                          What I find interesting about lightweight formal methods is that they are completely useful outside of the code. If the code doesn’t match the spec at all that doesn’t diminish the value of the spec. They are a way of describing your thinking and then checking it. So I guess my response is maybe we don’t need to automate checking the model vs the implementation.

                                                                                          For a better idea of how this can all fit together, I highly recommend reading Towards Zero Defect Programming by Allan M. Stavely. It walks through the Cleanroom methodology which is a way of incorporating formal methods into the software dev cycle. My main take way is that there’s a lot of value in keeping the spec separate from the implementation. Decoupling the two allows teams to understand the program better together.

                                                                                        1. 3

                                                                                          This will make it incompatible with GPL:ed projects – right? As GPL does not allow any additional limitations?

                                                                                          Reminds me of the classic JSLint license: https://en.wikipedia.org/wiki/JSLint

                                                                                          That license had “The Software shall be used for Good, not Evil.” in it – which caused quite a few problems.

                                                                                          1. 6

                                                                                            Not just GPL; it violates the FSF’s definition of Free Software:

                                                                                            The freedom to run the program as you wish, for any purpose (freedom 0).

                                                                                            It violates the Open Source Initiative’s definition of open source:

                                                                                            1. No Discrimination Against Persons or Groups

                                                                                            The license must not discriminate against any person or group of persons.

                                                                                            1. No Discrimination Against Fields of Endeavor

                                                                                            The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.

                                                                                            It violates the Debian Free Software Guidelines:

                                                                                            1. No Discrimination Against Persons or Groups

                                                                                            The license must not discriminate against any person or group of persons.

                                                                                            1. No Discrimination Against Fields of Endeavor

                                                                                            The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.)

                                                                                            In other words, it’s proprietary software (with source available)

                                                                                          1. 1

                                                                                            I think rel="me" is a neat little concept. Had no idea that anyone was misunderstanding it as some sort of author attribution.

                                                                                            I’m glad the author makes the point that the purpose of markup isn’t to appease Google. Still, I find it strange that so much supposedly “indie” web stuff seems to revolve around silos, and Twitter in particular. I’ve implemented some of it on my own site, but now that Mozilla Persona is dead their auth ideas all seem to rely on “log in via twitter” (or some other proprietary service) :(

                                                                                            1. 1

                                                                                              But the firm’s latest rankings, released last week, show Swift dropping to 11th place. Kotlin slipped from 27th place to 28th place. Though notable, slipping one place in one quarter doesn’t mean that Swift and Kotlin are in decline or even that they’ve peaked. “In general, we caution readers not to assign too much weight to small changes in the rankings; the differences between one spot or another, in general, tend to be slight,” RedMonk co-founder Stephen O’Grady wrote in a blog post analyzing the findings.

                                                                                              They literally quote someone who says that articles like this don’t mean anything…

                                                                                              1. 1

                                                                                                I’ve always stuck with cron for scheduling. On my own laptop I use Task Spooler to queue up expensive commands so that they run one at a time in the background (e.g. large downloads and network transfers; as well as commands which may interfere if run concurrently)

                                                                                                1. 4

                                                                                                  Never heard of it, but it seems like a super interesting approach to interactive environment. I cannot help but remember this Bret Victor’s talk about how we have been programming in almost-anachronistic ways with no innovation in the interfaces.

                                                                                                  1. 8

                                                                                                    There’s nothing obsolete about text. Visual languages don’t work. They’ve been tried hundreds of times, to no avail, because GUIs are fundamentally bad user interfaces for experienced users. Text is a better interface for power users, and programming languages are for power users.

                                                                                                    1. 14

                                                                                                      Why can’t I re-sort the definitions in my source instead of scrolling around then? Why is it hard to see a dependency graph for all my functions? Why do I have to jump between files all the time? Text - an interface for linear presentation of information - is fundamentally a kludge for code, which is anything but linear.

                                                                                                      1. 1

                                                                                                        Why can’t I re-sort the definitions in my source instead of scrolling around then?

                                                                                                        Sort them by what? It wouldn’t be difficult to write a script using the compiler module of Python to reorder the declarations in your file in an order you chose, which you could then use to replace the text of a buffer in your text editor. But usually I’d suggest what you want is to see a list of definitions in a particular order, which you could then use to jump to the definitions.

                                                                                                        In every case that I’ve seen of not using plain text, it inevitably become inscrutable. What is actually in my Smalltalk/Lisp image? What is actually there? What can people get out of it later when I deploy it?

                                                                                                        Why is it hard to see a dependency graph for all my functions?

                                                                                                        Because nobody has written something that will take your source files, determine their dependencies, and produce the DOT output (a very popular text-based format for graphs, far superior in my opinion to any binary graph description format) for that graph? It’s not like it’s particularly difficult.

                                                                                                        Why do I have to jump between files all the time?

                                                                                                        Because it turns out it’s useful to organise things into parts. Because it turns out it’s useful to be able to parallelise compilation and not reparse every bit of code you’ve ever written every time you change any part of it.


                                                                                                        I think that it’s definitely a requirement of any decent programming language to have a way to easily take the source code of that programming language and reify it into a syntax tree, for example. That’s very useful to have in a standard library. In Lisp it’s just read, Python has more complex syntax and requires more machinery which is in a standard library module, other languages have similar things.

                                                                                                        One point might be: maybe you don’t need a dependency graph if you can just make your code simpler, maybe you don’t need to jump around files much if your code is properly modularised (and you have a big enough screen and narrow enough maximum line length to have multiple files open at once), maybe sorting your definitions is wrong and what you want is a sortable list of declarations you can jump to the definitions.

                                                                                                        Not to mention that version control is important and version controlling things that aren’t text is a problem with conventional version control tools. Might not be an issue, you have your own VCS, but then you enter the land of expecting new users of your language to not only not use their standard editor, but also to not use their standard VCS, not use their standard pastebin, etc. How do you pastebin a snippet of a visual language so someone on an IRC channel can see it and give you help? How do you ask questions on StackOverflow about a visual language?

                                                                                                        It’s not even an issue of them being unusual and unsupported. By their very nature, not using text means that these languages aren’t compatible with generic tools for working with text. And never will be. That’s the thing about text, rather than having many many many binary formats and few tools, you have one binary format and many many tools.

                                                                                                        1. 8

                                                                                                          Hey Miles, thanks for elaborating. I think we could have more interesting discussions if you give me a bit more credit and skip the trivial objections. You’re doing the same thing you did last time with C++ compilers. Yes, I know I could write a script, it’s not the point. I’m talking about interactive tools for source code analysis and manipulation, not a one-off sort.

                                                                                                          I don’t agree with your objections about parallel compilation and parsing. It seems to me that you’re just thinking about existing tools and arguing from the status quo.

                                                                                                          Further down, you make a suggestion which I interpret as “better languages could mitigate these issues” which is fair, but again I have to disagree because better languages always lead to more complex software which again requires better tools, so that’s a temporary solution at best.

                                                                                                          You also raise a few objections, and here I should clarify that what I have in mind is not some kind of visual flowchart editor. What I’m claiming is that the conflation of internal representation and visual representation for code is counterproductive, but I think that a display representation that mostly looks like text is fine (as long as it’s actually within a structured editor). What I’m interested in is being able to manipulate symbols and units of code as well as aspects of its structure rather than individual characters.

                                                                                                          Consequently, for pastebin or StackOverflow, you could just paste some text projection of the code, no problem. When it comes to VCS, well, the current situation is quite poor, so I’d welcome better tools there. For example, if there was a VCS that showed me diffs that take into account the semantics of the language (eg like this: https://www.semanticmerge.com), that would be pretty cool.

                                                                                                          For the rest of your objections, I offer this analogy: imagine that we only had ASCII pictures, and none of this incompatible JPG/PSD/PNG nonsense with few complicated tools. Then we could use generic tools for working with text to manipulate these files, and we wouldn’t be constrained in any way whether we wanted to create beautiful paintings or complex diagrams. That’s the thing about text!

                                                                                                          I think the practitioners and particularly academics in our field should have more sense of possibilities and less affection for things the way they are.

                                                                                                          1. 1

                                                                                                            When it comes to VCS, well, the current situation is quite poor, so I’d welcome better tools there.

                                                                                                            Existing VCS could work reasonably well if the serialisation/“text projection” was deterministic and ‘stable’, i.e. minimising the amount of spurious changes like re-ordering of definitions, etc. As a first approximation I can imagine an s-expression language arranging the top-level expressions into lexicographic order, spreading them out so each sub-expression gets its own line, normalising all unquoted whitespace, etc. This would be like a very opinionated gofmt.

                                                                                                            If users wan’t to preserve some layout/etc. then the editor can store that as metadata in the file. I agree that semantics-aware diffing would be great though ;)

                                                                                                            1. 1

                                                                                                              So you always end up separating the storage format from display representation in order to create better tools, which is exactly my point.

                                                                                                              1. 1

                                                                                                                Yes, I agree with your points. Was just remarking that some of these improvements (e.g. VCS) are easier to prototype and experiment with than others (e.g. semantics-aware queries of custom file formats).

                                                                                                            2. 1

                                                                                                              The way I see it is that there are tools for turning text into an AST and you can use them to build the fancy things you want. My point wasn’t ‘you can write that sort as a one-off’. You can edit code written in a text-based programming language with a really fancy editor that immediately parses it to an AST and works with it as an AST, and only turns it into text when written to disk. I have no problem with that. But really you’re still editing text when using something like paredit.

                                                                                                              Something like vim but where the text objects are ‘identifier’, ‘ast node’, ‘expression’, ‘statement’, ‘logical line of code’, ‘block’, etc. rather than ‘text between word separators’, ‘text between spaces’, ‘line’, etc. would be a useful thing. In fact, you could probably do this in vim. I have an extension I use that lets you modify quotes around things taking into account escaped quotes within, etc. That’d probably work way better if it had that default structure for normal text and then could be customised to actually take into account the proper grammar of particular programming languages for which that is supported.

                                                                                                              What I’m concerned about is the idea that it’s a good idea to store code in a proprietary binary file format that’s different for every language, where you can’t use the same tools with multiple languages. And then having to reimplement the same basic functionality for every single language in separate IDEs for each, where everything works slightly differently.

                                                                                                              I do find it useful that I can do ci( and vim will delete everything inside the nearest set of parentheses, properly taking into account nesting. So if I have (foo (hello 1 2 3) bar) and my cursor is on the a, it’ll delete everything, even though the nearest ( and ) are beside hello and not foo. That kind of thing, more structured editing? I’m all for that.

                                                                                                              Consequently, for pastebin or StackOverflow, you could just paste some text projection of the code, no problem. When it comes to VCS, well, the current situation is quite poor, so I’d welcome better tools there. For example, if there was a VCS that showed me diffs that take into account the semantics of the language (eg like this: https://www.semanticmerge.com), that would be pretty cool.

                                                                                                              Ultimately I think if you have a recognised standardised text projection of your code, you might as well just make that the standardised format for it, then your fancy editor or editor plugin can parse it into the structures it needs. This helps ensure you can edit code over SSH, and have a variety of editors compatible with it, rather than just the single language-designer-provided IDE.

                                                                                                              One of the nice things about git is that it stores snapshots internally rather than diffs. So if you have a language-specific tool that can produce diffs that are better due to being informed by the grammar of the language (avoiding the problem of adding a function and the diff being ‘added a new closing brace to the previous function then writing a new function except for a closing brace’, for example), then you can do that! Change the diff algorithm.

                                                                                                              For the rest of your objections, I offer this analogy: imagine that we only had ASCII pictures, and none of this incompatible JPG/PSD/PNG nonsense with few complicated tools. Then we could use generic tools for working with text to manipulate these files, and we wouldn’t be constrained in any way whether we wanted to create beautiful paintings or complex diagrams. That’s the thing about text!

                                                                                                              Well I mean I do much prefer creating a graph by writing some code to emit DOT than by writing code to emit PNG. I did so just the other day in fact. http://rout.nz/nfa.svg. Thank god for graphviz, eh?

                                                                                                              Note that there’s also for example farbfeld, and svg, for that matter: text-based formats for images. Just because it’s text underneath doesn’t mean it has to be rendered as ASCII art.

                                                                                                              1. 1

                                                                                                                Cool, I’m glad we can agree that better tools would be good to have.

                                                                                                                As far as the storage format, I don’t actually have a clear preference. What’s clearly needed is a separation of storage format and visual representation. If we had that, arguments about tabs vs spaces, indent size, let/in vs where, line length, private methods first or public methods first, vertical vs horizontal space (and on and on) could be nullified because everybody could arrange things however they like. Why can’t we have even such simple conveniences? And that’s just the low hanging fruit, there are far more interesting operations and ways of looking at source that could be implemented.

                                                                                                                The other day there was a link to someone’s experiment (https://github.com/forest-lang/forest-compiler) where they use one of the text projections as the storage format. That might work, but it seems to me that the way parsing currently happens, there’s a lot of unnecessary work as whole files are constantly being reparsed because there is no structure to determine the relevant scope. It seems that controlling operations on the AST and knowing which branches are affected could be a lot more efficient. I’m sure there’s plenty of literature of this - I’ll have to look for it (and maybe I’m wrong about this).

                                                                                                                What I’m concerned about is the idea that it’s a good idea to store code in a proprietary binary file format that’s different for every language, where you can’t use the same tools with multiple languages. And then having to reimplement the same basic functionality for every single language in separate IDEs for each, where everything works slightly differently.

                                                                                                                I understand your concern, but this sounds exactly like the current state of affairs (other than really basic stuff like syntax highlighting maybe). There’s a separate language plugin (or plugins) for every combination of editor/IDE and language, and people keep rewriting all that stuff every time a new editor becomes popular, don’t they?

                                                                                                                One of the nice things about git is that it stores snapshots internally rather than diffs.

                                                                                                                Sure, we can glean a bit more information from a pair of snapshots, but still not much. It’s still impossible to track a combination of “rename + change definition”, or to treat changes in the order of definitions as a no-op, for example. Whereas if we were tracking changes in a more structured way (node renamed, sub-nodes modified etc.), it seems like we could say a lot more meaningful things about the evolution of the tree.

                                                                                                                Thank god for graphviz, eh?

                                                                                                                Perhaps the analogy was unclear. Being able to write a set of instructions to generate an image with a piece of software has nothing to do with having identical storage format and visual representation. If we approached images the same way we approach code, we would only have ASCII images as the output format, because that’s what is directly editable with text tools. Since you see the merits of PNG and SVG, you’re agreeing that there’s merit in separating internal/storage representation from the output representation.

                                                                                                                1. 1

                                                                                                                  What I’m concerned about is the idea that it’s a good idea to store code in a proprietary binary file format that’s different for every language

                                                                                                                  I might have missed something, but I didn’t see anyone proposing this.

                                                                                                                  In particular, my understanding of Luna is that the graphical and textual representations are actually isomorphic (i.e. one can be derived if given the other). This means we can think of the textual representation as the being both a traditional text-based programming language and as a “file format” for serialising the graphical programming language.

                                                                                                                  Likewise we can switch to a text view, use grep/sed/etc. as much as we like, then switch back to a graphical view if we want (assuming that the resulting text is syntactically valid).

                                                                                                            3. 1

                                                                                                              Tools that improve navigation within textual source have existed for a long time. I’ve been using cscope to bounce around in C and Javascript source bases for as long as I can remember. The more static structure a language has, the easier it is to build these tools without ambiguity. The text source part isn’t really an issue – indeed it enables ad hoc tooling experiments to be built with existing text management tools; e.g., grep.

                                                                                                              1. 4

                                                                                                                Those tools aren’t text, though. They’re other things the augment the experience over just using text which becomes an incidental form of storage. Tools might also use AST’s, objects, data flows, constraints, and so on. They might use anything from direct representation to templates to synthesis.

                                                                                                                I think the parent’s point was just text by itself is far more limited than that. Each thing I mentioned is available in some programming environment with an advantage over text-driven development.

                                                                                                                1. 1

                                                                                                                  I think it’s wrong to say that the text storage is incidental. Line-oriented text files are about the lowest common denominator way we have to store data like this.

                                                                                                                  For starters, it’s effectively human-readable – you can lift the hood up and look at what’s underneath, understanding the effect that each individual character has on the result. Any more complicated structure, as would be generally required to have a more machine-first structured approach to program storage, is not going to have that property; at least not to the same extent.

                                                                                                                  If this thread demonstrates anything, it’s that we all have (at times, starkly!) different preferences for software engineering tools. Falling back on a textual representation allows us to avoid the need to seek consensus on a standard set of tools – I can use the editor and code manipulation tools that make sense to me, and you can stick to what makes sense to you. I think a lot of the UNIX philosophy posturing ends up being revisionist bunk, but the idea that text is a pretty universal interface for data interchange isn’t completely without merit.

                                                                                                                  1. 6

                                                                                                                    The under-the-hood representation is binary-structured electricity that gets turned into human-readable text by parsing and display code. If already parsing it and writing display code, then one might just as well use a different encoding or structure. Text certainly has advantages as one encoding of many to have available. Plugins or input modules can take care of any conversions.

                                                                                                                    Text does often have tooling advantages in systems like UNIX built with it in mind, though.

                                                                                                                    1. 1

                                                                                                                      I think it’s a reductionist argument for the good-enough, hard earned status quo. I think it can be valid, but only within a very narrow perspective - operational and short term.

                                                                                                                      To my mind, your position is equivalent to this: we should only have ASCII images, and we don’t need any of that PNG/JPG/PSD stuff with complicated specialised tools. Instead, we can use generic text tools to make CAD drawings, diagrams, paintings - whatever. All of those things can be perfectly represented in ASCII, and the text tools will not limit us in any way!

                                                                                                                  2. 2

                                                                                                                    I want to search my code like a database, e.g. “show my where this identifier is used as a parameter to a function” - the tooling for text doesn’t support this. Structured tooling would be super useful.

                                                                                                                    1. 2

                                                                                                                      Many things can be “queried” with grep and regular expressions. Which is also great to find “similar occurrences” that need to be checked but are only related by some operators and function calls following another. But on the other hand I’d definitely argue that IDEs at least have a tiny representation of the current source file for navigation or something and that you can click some token and find its uses, definitions, implementations … But it only works if I disable the low power mode. And with my 8Gb RAM MacBook I sometimes have to kill the IDE before running the program to make sure I can still use it at the same time.

                                                                                                                      1. 7

                                                                                                                        Maybe if it wasn’t parsing and re-parsing massive amounts of text all the time, it would be more energy efficient…

                                                                                                                      2. 1

                                                                                                                        Exactly. And it could extend beyond search; code could be manipulated and organised in more powerful ways. We still have rudimentary support for refactoring in most IDEs, and so we keep going through files and manually making structurally similar changes one by one, for no reason other than the inadequate underlying representation used for code.

                                                                                                                        I could be wrong and maybe this is impossible to implement in any kind of general way beyond the few specific examples I’ve thought of, but I find it strange that most people dismiss the very possibility of anything better despite the fact that it’s obviously difficult and inconvenient to work with textual source code.

                                                                                                                        1. 1

                                                                                                                          The version of cscope that I use does things of that nature. The list of queries it supports:

                                                                                                                          Find this C symbol:
                                                                                                                          Find this global definition:
                                                                                                                          Find functions called by this function:
                                                                                                                          Find functions calling this function:
                                                                                                                          Find this text string:
                                                                                                                          Change this text string:
                                                                                                                          Find this egrep pattern:
                                                                                                                          Find this file:
                                                                                                                          Find files #including this file:
                                                                                                                          Find assignments to this symbol:
                                                                                                                          

                                                                                                                          I use Find functions calling this function a lot, as well as Find assignments to this symbol. You could conceivably add more query types, and I’m certain there are other tools that are less to my admittedly terminal-heavy aesthetic preference that offer more flexible code search and analysis.

                                                                                                                          The base structure of the software being textual doesn’t get in the way of this at all.

                                                                                                                          1. 3

                                                                                                                            Software isn’t textual. We read the text into structures. Our tools should make these structures easier to work with. We need data structures other than text as the common format.

                                                                                                                            Can I take cscope’s output and filter down to “arguments where the identifiers are of even length”?

                                                                                                                            1. 5

                                                                                                                              Compilers and interpreters use structured representations because those representations are more practical for the purposes of compiling and interpreting. It’s not a given that structured data is the most practical form for authoring. It might be. But what the compiler/interpreter does is not evidence of that.

                                                                                                                              1. 1

                                                                                                                                Those representations are more practical for searching and manipulating. Try it!

                                                                                                                              2. 1

                                                                                                                                I would also be interested on your thoughts about Lisp where the code is already structured data. This is an interesting property of Lisp but it does not seem to make it clearly easier to use.

                                                                                                                                1. 2

                                                                                                                                  but it does not seem to make it clearly easier to use.

                                                                                                                                  Sure it does: makes macros easier to write than a language not designed like that. Once macros are easy, you can extend the language to more easily express yourself. This is seen in the DSL’s of Common LISP, Rebol, and Racket. I also always mention sklogic’s tool since he DSL’s about everything with a LISP underneath for when they don’t work.

                                                                                                                          2. 2

                                                                                                                            Sure, but all of these tools (including IDEs) are complicated to implement, error-prone, and extremely underpowered. cscope is just a glorified grep unless I’m missing something (I haven’t used it, just looked it up). The fact that you bring it up as a good example attests to the fact that we’re still stuck somewhere near mid-twentieth century in terms of programming UI.

                                                                                                                            1. 4

                                                                                                                              I bring it up as a good example because I use it all the time to great effect while working on large scale software projects. It is relatively simple to understand what it does, it’s been relatively reliable in my experience, and it helps a lot in understanding the code I work on. I’ve also tried exuberant ctags on occasion, and it’s been pretty neat as well.

                                                                                                                              I don’t feel stuck at all. In fact, I feel wary of people attempting to invalidate positive real world experiences with assertions that merely because something has been around for a long time that it’s not still a useful way to work.

                                                                                                                        2. 2

                                                                                                                          Have you noted, that the Luna language has dual representation? Where each visual program has an immediate and easily editable text representation, and the same is true in the other direction as well? This is intended to be able to keep the benefits of the text interface, while adding the benefits of a visual representation! That’s actually the main idea behind Luna.

                                                                                                                          1. 1

                                                                                                                            What about the power users who use things like Excel or Salesforce? These are GUIs perfectly tailored to specific tasks. A DJ working with a sound board certainly wouldn’t want a textual interface.

                                                                                                                            Textual interfaces are bad, but they are generic and easy to write. It’s a lot harder to make an intuitive GUI, let alone one that works on something as complex as a programming language. Idk if Luna is worthwhile, but text isn’t the best user interface possible imho

                                                                                                                            1. 3

                                                                                                                              DJs use physical interfaces, and the GUIs emulation of those physical interfaces are basically all terrible.

                                                                                                                              I’ve never heard of anyone liking Salesforce, I think that must be Stockholm Syndrome. Excel’s primary problem in my opinion is that it has essentially no way of seeing how data is flowing around. If something had the kind of ‘reactive’ nature of Excel while being text-based I’d much prefer that.

                                                                                                                              Textual interfaces are excellent. While there are tasks that benefit from a GUI - image editing for example - in most cases GUIs are a nicer way of representing things to a new user but are bad for power users. I wouldn’t expect first year computer science students to use vim, as it’s not beginner-friendly, but it’s by far the best text editor out there in the hands of an experienced user.

                                                                                                                              1. 2

                                                                                                                                I wouldn’t expect first year computer science students to use vim, as it’s not beginner-friendly, but it’s by far the best text editor out there in the hands of an experienced user.

                                                                                                                                I’d call myself an “experienced user” of vim. I’ve written extensions, given workshops, and even written a language autoindent plugin, which anyone who’s done it knows is like shoving nails through your eyeballs. About once a year I get fed up with the limitations of text-only programming and try to find a good visual IDE, only to switch back when I can’t find any. Just because vim is the best we currently have doesn’t mean it’s actually any good. We deserve better.

                                                                                                                                (For the record, vim isn’t beginner-unfriendly because it’s text only. It’s beginner-unfriendly because it’s UI is terrible and inconsistent and the features are all undiscoverable.)

                                                                                                                                1. 2

                                                                                                                                  Most people don’t bother to learn vimscript properly, treating it much like people treated Javascript for years: a bunch of disparate bits they’ve picked up over time, with no unifying core. But once you actually learn it, it becomes much easier to use and more consistent. The difference between expressions and commands becomes sensible instead of seeming like an inconsistency.

                                                                                                                                  I never get fed up with the limitations of text-only programming, because I don’t think they exist. Could you elaborate on what you are saying those limitations are?

                                                                                                                                  And I totally, 100% disagree with any claim that vim’s UI is bad or inconsistent. On the contrary, it’s extremely consistent. It’s not a bunch of little individual inconsistent commands, it’s motions and text objects and such. It has extensive and well-written help. Compared to any other IDE I’ve used (a lot), it’s way more consistent. Every time I use a Mac program I’m surprised at how ad-hoc the random combinations of letters for shortcuts are. And everything requires modifier keys, which are written with ridiculous indecipherable symbols instead of ‘Ctrl’ ‘Shift’ ‘Alt’ etc. Given that Mac is generally considered to be very easy to use, I don’t think typical general consensus on ease of use is very instructive.

                                                                                                                          2. 3

                                                                                                                            Bret Victor explains the persistence of textual languages as resistance to change, drawing an equivalence between users of textual languages now and assembly programmers who scoffed at the first higher-level programming languages. But this thread is evidence that at least some people are interested in using a language that isn’t text-based. Not everyone is fairly characterized by Bret Victor’s generalization. So then why hasn’t that alternative emerged? There are plenty of niche languages that address a minority preference with reasonable rates of adoption. With the exception of Hypercard, I can’t think of viable graphical programming language. Even Realtalk, the language that runs Dynamicland (Bret Victor’s current focus), is text-based, being a superset of Lua. I keep hearing about how text-based languages are old-fashioned and should die out, but I never hear anything insightful about why this hasn’t happened naturally. I’m not denying that there are opportunities for big innovation but “make a visual programming language” seems like an increasingly naive or simplistic approach.

                                                                                                                            1. 4

                                                                                                                              I think it has to do with the malleability of text. There’s a basic set of symbols and one way to arrange them (sequentially.) Almost any problem can be encoded that way. Emacs’ excellent org-mode is a testament to the virtue of malleability.

                                                                                                                              Excel also has that characteristic. Many, many kind of problems can be encoded in rectangles of text with formulas. (Though I might note that having more ways to arrange things allows new kinds of errors, as evidenced by the growing cluster of Excel features for tracing dependencies & finding errors.)

                                                                                                                              Graphical languages are way less malleable. The language creator decides what elements, relations, and constraints are allowed. None of them let me redefine what a rectangle represents, or what relations are allowed between them. I think that’s why these languages can be great at solving one class of problem, but a different class of problem seems to require a totally different graphical language.

                                                                                                                              1. 1

                                                                                                                                My suspicion is that it’s because graphical languages merge functionality and aesthetics, meaning you have to think very, VERY hard about UI/UX and graphic design. You need to be doing that from the start to have a hope of it working out.

                                                                                                                            1. 1

                                                                                                                              I think there are actually two distinct forms of “reusability” in the wild, which I like to distinguish between.

                                                                                                                              Some things are reusable because they’re so small and simple that they can slot into solutions for all sorts of problems. Examples include unix utilities, JSON/protobuf/etc., arrays, linked lists, filesystems, SQlite (RDBMS servers like Postgres are debatable, I personally wouldn’t call them “simple”), key/value stores, etc.

                                                                                                                              This is a good form of reusability, since this sort of code tends to be solving a well-specified (although abstract) problem, and can often be considered “finished”. Pulling out such functionality from a larger program hence reduces the amount of stuff we need to care about.

                                                                                                                              Other things are reusable because they have so many hooks/options that we can configure them to plug into all sorts of solutions. Examples include stereotypical “enterprise ready” software, towering monoliths of code like GCC and GHC, “frameworks” and “ecosystems” like Symfony, Drupal, Wordpress, etc.

                                                                                                                              This second form of “reusability” is often underisable. We need to go out of our way to support other use-cases, which may not even exist. This sort of software is often modelling some concrete, domain-specific concern like “users” or “reports”, which are informal enough that the decisions encoded in this implementation are probably unsuitable for other applications, and hence the need to keep adding more and more parameters and overrides.

                                                                                                                              I think there’s a correlation with how functional programming and object oriented programming tend to be practiced too. FP libraries usually provide some abstract entity, which may be simple, self-contained and reusable in the first sense. Whilst OOP libraries often provide interfaces to some concrete entity, like a Web service, with hooks and parameters to make it reusable in the second sense.

                                                                                                                              I think this may be a consequence of their approaches to code layout, the expression problem, etc. OOP limits what we can do (the set of methods), but subclassing makes it easy to override and alter how those things are done. FP limits what there is (the data constructors), but makes it easy use those things in new ways.

                                                                                                                              1. 6

                                                                                                                                Extensibility and re-usability are potential goals for system boundaries only. ..creating something that’s re-usable is pretty inherently about creating something that’s a system boundary, to some degree or another. And if you’re knowingly working on a system boundary… you’re knowingly working on something that supposed to be re-usable already. It’s pretty redundant to hail it as a design goal.

                                                                                                                                This is great. I’ve been collecting anti-reuse, anti-abstraction links. Everytime I add one to my collection I’m going to share the whole thing.

                                                                                                                                1. 4

                                                                                                                                  The part you quoted is also followed by:

                                                                                                                                  System boundaries are the danger zones of design. To whatever extent possible, we don’t want to create them unnecessarily. Any mistakes we make there are frozen, under threat of expensive breaking changes to fix.

                                                                                                                                  This is a nice counter to the approach of “all classes should be isolated”, “inject everything”, “never use ‘new’”, “mock everything”, etc. that I’ve encoutered in old jobs. It turns the implementation detail of internal code organisation boundaries (i.e. classes) into faux system boundaries, which makes them much more rigid and burdensome to change, slowing us down and discouraging refactoring.

                                                                                                                                1. 1

                                                                                                                                  Sounds like a pleasant experience. One thing I wanted to point out is that many FP practices/principles (single assignment, returning fresh data instead of mutating, etc.) can be followed in almost any language, in case any JS or TypeScript users didn’t want to switch to something like PureScript. I find that they tend to be good ideas even on their own, as long as it’s not going against the grain of surrounding code.