Threads for c-cube

  1. 15

    I’m (positively) surprised by the JSON5 support! It’s the most comprehensive extension to JSON that aims at making it more usable to people, especially for configuration files and such. Adoption in SQLite is a significant endorsement, so maybe JSON5 will become more and more popular and compete with yaml. I doubt yaml will ever end up in SQLite :-).

    1. 1

      I used to have to hand-write a lot of JSON for unit tests — in C++ string literals, so no fancy IDE syntax checking — and JSON5 is a lifesaver for that.

      1. 1

        The JSON library that xmake uses supports at least some of these extensions and being able to specify numbers in hex is incredibly useful. We use it in CHERIoT to describe the memory layout of a target SoC and having to translate all of those numbers to decimal would be a lot of pain. Not sure if it supports comments, but that would be nice too.

    1. 21

      A problem with LTV for some formats, I think, is that the type influences how the length is represented.

      For example, in CBOR (or msgpack, they’re pretty close) the first byte determines the type and, depending on the type, contains part of the value or part of the length. The type is the 3 most significant bits; the 5 least significant might contain a small integer or special value, or parts of the length which then follows. You simply cannot parse the length without parsing the type first.

      In protobuf, messages are composed of (tag+type), length, value triples. The first byte contains the tag (record field) along with the wire type. But there too, length is not always an actual thing: integers are directly encoded after the first byte via a varint. What would that even look like with LTV?

      1. 3

        The article didn’t convince me that LTV is better than TLV, but your comment did.

        If we lead with the length, it can’t be specialized based on the type. It should always be the number of bytes following, not the number of entries of the type specified.

      1. 5

        Author here! Let me know if you have any questions :)

        1. 6

          Why have variables? It seems like you’ve given yourself an alpha-equivalence problem.

          Edit: Another question: Are scrapyards monoidal? Is it possible to add two scrapyards together?

          1. 2

            Why have variables? It seems like you’ve given yourself an alpha-equivalence problem.

            In a previous implementation, I canonicalized variable names before hashing. Turns out it wasn’t a very useful feature. So now, for scraps to have the same hash, they have to be exactly the same

            In practice, I think community preprocessors (linters, formatters, etc) will push many programs to fall into the same buckets.

            Edit: Another question: Are scrapyards monoidal? Is it possible to add two scrapyards together?

            If you read this comment, this is something I’m still working on.

            The naming of scraps is unique to a scrapyard, but the repository of scraps themselves are interchangeable if you don’t consider collisions.

            1. 2

              Oh, I mean, why have any variables at all? In my Cammy experimental language, there are no variables or lambda-binders; instead, functions are composed directly from smaller functions. This lets me skip over alpha- and eta-equivalence entirely! Unlike in Unison, Dark, and apparently Scrapscript; Cammy code does not allow naming types or creating custom encapsulation.

              Additionally, Cammy’s hives are monoidal. They’re JSON, and there is an algorithm (not yet implemented) for merging two hives into one larger hive. The upshot is that namespaces are always local to each hive; there is no global namespace. (I totally understand your approach of making your namespace into the global namespace; I did something similar with Tahoe-LAFS once upon a time.)

              1. 2

                Super interesting idea! It would solve a certain set of problems, but I think it would make the language overall less accessible.

                I could definitely see a Cammy-like embedded DSL being a fun project :)

                Btw I think you’d really enjoy Smullyan’s To Mock A Mocking Bird, if you haven’t read it already. It’s an entertaining puzzle approach to combinators.

              2. 1

                Please keep variables, they’re very useful for many programmers. Not everyone wants to write in a concatenative or pointfree style all the time!

                Unrelatedly, do you expect scrapscript to be defined by a spec and possibly have several implementations (including some that are to be embedded into other languages), or to be defined by your canonical implementation?

                1. 2

                  Great great question.

                  I’m hoping to publish the full spec this month.

                  I am not sure yet, but I think there will be a canonical implementation hosted in the scrapyard itself. But in general, I think the ecosystem will look more like JSON, with many non-canonical implementations in many languages.

            2. 3

              Is scrapscript statically typed? If so, what type system are you using?

              1. 1

                Yes, statically typed. I’m pretty sure it’s basic Hindley-Milner, but I’m not an expert.

                I think the type system is 1:1 with Elm right now, which is to say pretty simple.

              2. 2

                Do you have any concrete documentation? The pitch sounds great but there’s only so much I can glean from trying to read between the lines of cryptic little snippets of a language I don’t know.

                1. 2

                  Concrete documentation is my next goal :) I wanted to see if people were interested in the idea before committing more time to it, and it looks like I’m getting way more support than I could’ve ever imagined!

                  1. 2

                    I second that, the pitch worked pretty well, now you have curious people who want to know more :)

                    1. 1

                      Yes, it seems many of us would love to see gory details! At least just an explanation of the syntax.

                  2. 2

                    If two values have the same hash, are they equal? Can you compare functions for equality? Is 1 == 1.0? How many equality operators do you have?

                    1. 4

                      If two values have the same hash, are they equal?

                      #sha1$e4caecf0d6f84d4ad72e228adce6c2b46a0328f9$0
                      

                      The little $0 on the end is an extra ID for the scrapyard to uniquely identify scraps with colliding hashes, if needed.

                      So yes, two hashes always means equal scraps within the same scrapyard.

                      Can you compare functions for equality?

                      Yes, but variable names matter. They might also have to have the same arity, but I haven’t explored it that deeply.

                      This feature may have to be removed if it results in unpredictable or inconsistent behaviors.

                      Is 1 == 1.0?

                      No. Integers and floats are separate types.

                      I might also include complex numbers in the final spec, but I think I’m going to get community feedback first.

                      How many equality operators do you have?

                      Just one equality operator, which compares any two values of the same type.

                      1. 5

                        The little $0 on the end is an extra ID for the scrapyard to uniquely identify scraps with colliding hashes, if needed.

                        That means these things aren’t hashes, they are IDs that are interpreted relative to a particular scrapyard. So your system is not content addressible and doesn’t use IPFS. The blog post gives a different impression.

                        1. 2

                          Sorry if I misrepresented myself.

                          $0/$1/$2/… seems like a simple way to solve collision problems. If you have other suggestions, I’d love to hear them.

                          I do hope to decentralize scrapyards at some point, but distributed systems are too hard to tackle from the start. One strategy is to take the biggest scrapyard and decentralize it by using IPFS or a custom backend. Another strategy would be to federate all scrapyards together by creating an exchange protocol.

                          1. 3

                            I may be misunderstanding, but: if you’re going to use digests as proxies for scripts, then you’re committed to the belief that there is a 1::1 mapping from digest ⟶ script. This is of course a false belief, but with a good digest function it’s true enough for the real world, and lots of software believes it.

                            If you accept the possibility of hash collisions, then you can’t use digests to represent scripts anymore. You can tack on integers to make the digests locally unique, but how do you make that globally unique without a central registry? And with a registry, you don’t need the digests anymore.

                            1. 1

                              Thanks! It sounds like you fully understand the problem :)

                              We’re early enough in scrapscript to change some of these things, so your thoughts are helpful.

                              Here’s the tradeoffs I see for having collision integers:

                              • Trusting the digest: By trusting the digest, the only sensible way to handle collisions would be to randomly sample all the matching scraps until one matches by type. This is honestly not a bad option, because it makes a lot of the system simpler and makes everything more distributable.
                              • Expecting collisions: Explicit is often better than implicit. By specifying a version, it prevents a certain class of (very very unlikely) attacks from ever occurring. This is probably the worse choice, but it’s the more reversible choice. It’s easy to ignore the $0 in existing identifiers, but it’s difficult to add them.

                              I very much expect to adopt the $0/$1/… syntax in the beginning, and then drop it when we have more confidence in the system. I’m open to ideas though, and will be fielding more feedback from the community

                              1. 4

                                In general, every content-addressible store I’ve ever heard of assumes collisions don’t happen, as do blockchains, and git. The keyspace is too massive (assuming the hash function isn’t broken, and if somebody figured out how to make sha1 collusion on demand, a lot of systems will suddenly have a very big problem) SHA1 has a uniformly distributed 160 bit digest. Calculate the collision probability and multiply it by the number of scraps you expect to exist (feel free to make absurd assumptions like “there will be 3 scraps for every atom in the observable universe”) figure out what the real probability of a collision is (edit: I think this is wrong, it’s not a simple multiplication because of the birthday paradox. I don’t remember the right formula, but the number is still very small), and if you’re still worried, (i.e. it’s above 5%, or 1%, or 0.00001%, whatever), switch to SHA256, if that’s not satisfactory, there’s SHA512, if that’s not satisfactory, make sure you haven’t dropped a few hundred zeros somewhere in your calculations 😉 (and use a sponge function, which can make the key arbitrarily long)

                                (Actually, probably switch to sha256 anyway, it’s thought to be more secure, and it’s I think what IPFS and Docker Registry use, which are both currently extant, large, content stores)

                                Explicit may be better than implicit*, but sticking a $0 at the end of everything and building contingencies for hash collisions is just extra work and cognitive load, and kind of undermines the concept of a content-addressable store.

                                **IMO in many cases they’re dual, and regardless, tacit is better than both. Often times I hear that rule repeated, it feels like there’s some conspiracy of keyboard manufacturers who profit when developers need to press more buttons to say the same thing.

                                1. 3

                                  Wow, I think you pretty solidly convinced me.

                                  Thank you so much for the help!

                                  1. 2

                                    No problem! It’s not every day someone walks by with a question I’ve spent time researching in the last week :)

                                    This project looks really cool and I’m looking forward to trying it out, godspeed.

                            2. 1

                              Could you add a content length field to the ID to prevent (many) collisions? My understanding is that most known collision attacks require adding arbitrary data. If you have the constraint that both the hash and the content length must be equal, I think deliberate collision attacks become a lot harder. But you should check with a cryptographer.

                      2. 2

                        The first example defines a type person. What happens when you distribute this code across the internet, and the type person conflicts with another type with the same name in the code that dereferences the hash? Is there a module system to manage namespaces and prevent name conflicts?

                        In an expression like spaceq/is-planet, it looks like spaceq is a module and is-planet is a member of this module. Is spaceq a globally unique module name? If someone, somewhere else on the internet, makes a module called spaceq, is there now a name conflict that will confuse any code that tries to use spaceq? How do you manage namespaces on an internet scale, guarantee uniqueness and prevent conflicts?

                        1. 2

                          Great questions!

                          The first example defines a type person. What happens when you distribute this code across the internet, and the type person conflicts with another type with the same name in the code that dereferences the hash? Is there a module system to manage namespaces and prevent name conflicts?

                          The scrapyard is the namespace. I suspect that everybody will want to use the same scrapyard, but it’s not required.

                          I’m not 100% sure, but I think we have to replace all the names with their hashes before sending them over the wire. We don’t want cases where they arrive at the destination and use an updated version, or something like that.

                          I’m also not 100% sure about this, but I think we need to put the expected scrapyard in the message header, so that the recipient knows where to find all the referenced scraps.

                          In an expression like spaceq/is-planet, it looks like spaceq is a module and is-planet is a member of this module. Is spaceq a globally unique module name? If someone, somewhere else on the internet, makes a module called spaceq, is there now a name conflict that will confuse any code that tries to use spaceq? How do you manage namespaces on an internet scale, guarantee uniqueness and prevent conflicts?

                          In that example, spaceq is actually the user and is-planet is their function within the scrapyard.

                          It’s just a simple key-value store, so users can use arbitrary paths after their username. e.g. spaceq/planets/revised/is-planet haha

                          Whoever manages the scrapyard can manage their namespace however they want. The scrapyard I’ll be managing will probably be the “worldwide” one, and I’m still deciding how to fairly distribute user accounts. I want to avoid domain squatting and other legacy problems of the existing internet

                          The public scrapyard will essentially be two key-value stores: (1) name -> list (hash,timestamp), (2) hash -> scrap. An authentication layer will ensure that only you can edit your keyspace.

                          1. 1

                            Have you ever looked at SDSI/SPKI? It was a system for certificates, identification, authentication and authorization based on S-expressions. Your language seems more than powerful enough to do something similar.

                            I urge you not to design for a centralized system. It makes some stuff easier, but in the long run it creates all sorts of problems, esp if it has to handle user accounts, and esp^2 if it becomes a social network. Just look at what’s happening to other centralized social networks.

                            1. 2

                              I’ll definitely look into that! Sounds right down my alley. I’ve been having trouble with finding an elegant way to do the auth side of things.

                              There are different levels of centralization. Facebook, email, and bittorrent offer different tradeoffs.

                              I think I’m going to start with an email-like model where anybody can host their own interoperable server. But I suspect that most people will pick a popular default, much like GitHub or Gmail. If power gets too consolidated, I’ll figure out how to decentralize for robustness.

                              In case it matters, I just want to reiterate that scrapscript is not a money-making endeavor for me. I’d like to build profitable side-businesses on top of scrapscript, but the core language and ecosystem must remain open and equitable :) the centralization question is more of a matter of technical tradeoffs right now

                      1. 30

                        Well, I understand it’s frustrating to have to use a language you don’t like at work. I mainly program in OCaml so this might come out as defensive in places, but I do think so some of the criticism is valid!

                        For a start, yeah, the syntax has known flaws (especially wrt nesting; it’s also not great for formatting imho). The lack of line comments is annoying. However, lexing inside comments is an actual feature that allows for nested comments (so you can comment out something with comments in it, no problem).

                        The interface issue is kind of a thing, but you can either use objects, first-class modules, or go C-style with a record of functions. This is a lot easier than C because OCaml has closures, so it’s just a bit manual, not difficult. I have my gripes with some standard types not being interfaces (in particular, IO channels…). But overall this is a bit exaggerated as it’s not hard to implement your own. The thing about Int.abs puzzles me, what else do you expect Int.abs min_int to return anyway?

                        And of course, the ecosystem. Yeah, there are multiple standard libraries (although I’d consider batteries to have lost most momentum). It’s painful. It’s made a lot of progress in the past years (e.g. ocaml-lsp-server is a pretty solid LSP server now!) but it’s true that OCaml remains a small language. This part is pretty accurate.

                        However, the last paragraph annoys me more. Yes, OCaml is a lot more functional than Java or Rust. It’s not Haskell, but: in OCaml, immutable structures are both efficient and idiomatic. In Java, they’re not idiomatic; in Rust, I know of one library (im) that explicitly relies on that, everything in the stdlib is mutable, and most of the ecosystem embraces mutability. In addition, Java does not actually have functions; and Rust has a strong distinction between functions and closures. In OCaml all functions are closures and it’s seamless to create closures thanks to the GC. Try writing CPS code in Rust and see if it’s as trivial as in OCaml! Tailcalls are not easy to emulate…

                        So there are reasons to use OCaml in 2023. If you work on some algorithm-heavy, symbolic domain like compilers or logic tools (historically a strength of OCaml) it’s still one of the best languages; it’s not tied to a big corporation, compiles to native code with reasonable resource requirements.

                        1. 7

                          However, lexing inside comments is an actual feature that allows for nested comments (so you can comment out something with comments in it, no problem).

                          You don’t need to actually lex the contents of comments for that, just skim through the character stream looking for opening and closing delimiters and counting how many you find.

                          Agree with your second-to-last paragraph though; a lot of me learning Rust was learning how not to try to write it as if it were OCaml. Inconvenient closures and lack of partial evaluation were particularly painful.

                          1. 7

                            You don’t need to actually lex the contents of comments for that, just skim through the character stream looking for opening and closing delimiters and counting how many you find.

                            Yes, that’s what OCaml’s lexer does, but it has to account for string literals as well, to handle these:

                            (* "this is what a comment opening in OCaml looks like: (*" *)
                            

                            If you don’t look at " you would think that a “*)” is missing. That’s why (* " *) is not valid.

                            1. 5

                              Nope. Why would your comments care whether or not there’s an unclosed string delimiter in them? Rust works this way for example: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=973d237c97542b8e34623fdad0423f9c Various langs do various things with various flavor of “raw” strings, but those are a nice-to-have, not an essential feature.

                              You can do it by having your lexer state machine have entirely separate sub-lexers for strings and block comments, like this pseudocode:

                              fn lex_token(in: &mut Lexer) -> Token {
                                  loop {
                                      let next_char = in.advance_one_char();
                                      if next_char == '(' && in.peek() == '*' {
                                          skip_comment(in, 1);
                                          continue;
                                      } else if next_char == '"' {
                                          return lex_string(in);
                                      } else {
                                          // the rest of the lexer
                                      }
                                  }
                              }
                              
                              fn skip_comment(in: &mut Lexer, depth: usize) {
                                  while depth > 0 {
                                      let next_two_chars = in.peek_two();
                                      if next_two_chars == "(*" { depth += 1; }
                                      else if next_two_chars == "*)" {depth -= 1; }
                                      in.advance_one_char();
                                  }
                              }
                              
                              fn lex_string(in: &mut Lexer) -> String {
                                  let mut accumulator = ...;
                                  loop {
                                      let next_char = in.peek();
                                      if next_char == '\\' { in.advance_one_char(); ...handle escapes... }
                                      else if next_char == '"' { break; }
                                      else { accumulator.push(next_char); in.advance_one_char() }
                                  }
                                  accumulator
                              }
                              
                              1. 7

                                The Rust code you linked is a great example of why OCaml does this. In OCaml, you can surround any valid code with (* *) to comment out that code (and only that code). That is not the case in Rust – as you demonstrated, surrounding valid code with /* */ commented out the rest of the file, because Rust treats the /* inside the string literal as the start of a nested comment.

                                1. 5

                                  As opposed to not allowing a comment to have an un-matched " in it? You can have one or the other but not both, it seems; comments can invisibly contain string delimiters or strings can invisibly contain comment delimiters. I guess I personally find the Rust method less surprising.

                          2. 3

                            in Rust, I know of one library (im) that explicitly relies on that,

                            This is a bit more subtle. In Rust, there’s basically zero semantic difference between immutable and mutable collections, so there’s simply no need to have them. This is qualitatively different from languages with Java-like semantics.

                            But the bit about closures is spot on. Rust punishes you hard if you try to write high order code, mostly because there’s no “closure type”.

                            1. 9

                              I know what you mean, but the stdlib collections, for examples, are mutable, because the only way to add or remove elements is literally to mutate them in place. The basic feature of functional collections is that you can cheaply “modify” them by creating a new version without touching the old one. In Rust, im does that, but Vec or HashMap certainly don’t, and to keep both the new and old versions you need a full copy.

                              OCaml has pretty decent support for immutable collections, as do Clojure, Scala, Erlang/Elixir, Haskell, and probably a few others. It’s not that common otherwise and it’s imho a big part of what makes a language “functional”.

                              1. 6

                                The basic feature of functional collections is that you can cheaply “modify” them by creating a new version without touching the old one

                                I am not sure. My gut feeling is that in the vast majority of cases, functional programs do not retain historical, old versions of collections. Cheap modification is a spandrel, the actually important bit is value semantics: collections cannot be modified by someone else from under your feet.

                                1. 9

                                  You’d be surprised! A use case that I think the compiler has, for example, is to keep the scoping environment stored in each node of the post-typechecking AST. As a pure functional structure, each environment is only slightly different from the previous one, and you can store thousands of variants in the AST for one file. If you do incremental computations, the sharing is also pretty natural between incrementally modified versions.

                                  1. 4

                                    This updates me a bit, as, indeed, compilers are fairly frequent use-case for FP, and indeed in compilers you want scopes to be immutable. But I think scopes usually are a specialized data structure, rather than an off-the-shelf map? Looking at OCaml’s own compiler:

                                    It seems that they have have “scope which points at the parent scope” plus an immutable map, and the map is hand-coded. So this looks closer to “we have a specialised use-case and implement a specialized data structure for it”, rather than “we re-use facilities provided by the language because they are a great match”.

                                    1. 3

                                      Sure, this is also for bootstrapping reasons I think. A lot of things in the compiler don’t directly depend on the stdlib because they come too early, or that’s my understanding anyway. Parts are also very old :-).

                                      You can also find some instances of functional structures being used in backtracking algorithms (pretty sure alt-ergo uses them for that, maybe even CVC5 with custom immutable structures in C++). You just keep the old version and return to it when you backtrack.

                                      I personally have some transaction-like code that modifies state (well… an environment) and rolls back to the previous env if some validation failure occurs. It’s also much simpler when everything is immutable.

                                      Granted these are relatively niche but it shows how convenient the ability to keep old versions can be. And the language does facilitate that because the GC is optimized for allocating a lot of small structures (like tree nodes), and there’s easy functional updates of records, etc.

                                  2. 3

                                    collections cannot be modified by someone else from under your feet.

                                    The other edge case is if you have errors and need to back out of the current operation. Immutable data means you will always have the original copy to fall back to if you want to retry the operation. If you are using single owner mutability, have partially modified the collection and need to back out of the operation then it’s a bit trickier to retry because you need to undo all of the operations to get back to the original state.

                                    I think that they are nearly equivalent, but still different enough to have some tradeoffs. For example, Erlang style let it crash style languages might be better off with full immutability so that they don’t corrupt state on crashes (or things will get complicated).

                                    It’s interesting because I don’t see this distinction being drawn any where.

                                    1. 2

                                      FWIW, in AccessKit, I previously used im, so that when updating the accessibility tree (which is implemented as a hash map of nodes keyed by node ID), I could have a complete snapshot of the tree before applying the update, without paying the cost of copying all nodes (as opposed to just the ones that were changed or removed). I stopped using im because its license (MPL) is incompatible with my goals for AccessKit, and to avoid compromising performance by copying the whole map on every update, I had to compromise my design; I no longer have a complete snapshot of the tree as it was before the update.

                                      1. 4

                                        Yup, sometimes you do want to have historical snapshots, and immutable collections are essential for those cases as well. Another good example here is Cargo: it uses im to implement dependency solving using trial&backtracking method.

                                        My point is that’s not the primary reason why immutable collections are prevalent in functional programs. It’s rather that FP really wants to work with values, and immutability is one way to achieve that.

                                        More generally, I’ve observed that “Rust doesn’t have/default to immutable collections, so FP style in rust is hard” is a very common line of reasoning, and I believe it to be wrong and confused (but for a good reason, as that’s a rather subtle part about Rust). In Rust, ordinary vectors are values (http://smallcultfollowing.com/babysteps/blog/2018/02/01/in-rust-ordinary-vectors-are-values/), you really don’t need immutable data structures unless you specifically want to maintain the whole history.

                                        1. 2

                                          I stopped using im because its license (MPL) is incompatible with my goals for AccessKit

                                          Would https://lib.rs/crates/hamt-rs work?

                                1. 3

                                  Really interesting, but it could use a [2015] in the title. Now I’m curious if this has become even worse!

                                  1. 2

                                    Too large, hard to use (C++). Too small, hard to use (Turing tarpit).

                                    1. 6

                                      There’s another axis: too small, hard to optimise. Complexity has to live somewhere. There are three place something like a complex control-flow structure can live:

                                      • The language
                                      • The standard library
                                      • User code

                                      As you go down the list, it becomes harder for an implementation to optimise. Smalltalk is the extreme example of a tiny language. The entire spec fits on one piece of paper, most control flow is in the standard library. Even if statements are just messages sent to either a true or false object, which will either execute the closure passed as an argument or not. This would be painfully slow so most Smalltalk have a set of ‘primitive’ methods for these things that are not subject to dynamic dispatch, but which then end up with exciting performance cliffs (why is your custom loop construct a factor of ten slower than the standard library one?).

                                      Compilers like to have more information. A lot of modern compilers assume semantics of standard library functions because they are as well specified as language features but that then means you end up splitting the implementation in subtle and complex ways.

                                      I personally like the Smalltalk philosophy that says ‘nothing should go in the language if it can go in the standard library’. Any modern language is going to have to think hard about module versioning and supporting versions of module with different interface versions in the same program if it wants to scale to modern software engineering problems. If it solves these problems, putting most features in the standard library provides a way of introducing breaking changes without breaking old programs, but it does require a lot of care to enable optimisation. Higher-level control flow structures, for example, often come with more information about aliasing and interference than raw loops, but if they’re in the standard library and the compiler just sees a loop then they’re harder to optimise.

                                      1. 1

                                        ‘nothing should go in the language if it can go in the standard library

                                        That’s also kind of the philosophy of C++, in a way, and it’s why C++ has std::variant instead of something sane like proper sum types. Yikes.

                                        1. 1

                                          There’s a bit of a trade off here. You can implement variant only if you have type-unsafe unions. If you want type safety, you can’t implement variant in the library.

                                    1. 2

                                      One thing I’ve wondered about these NP-Complete problems is whether the definition of each problem is overly broad. Certainly there are many large 3SAT instances which can be easily solved in not much time at all. For a trivial example, any 2SAT instance can be written as a 3SAT instance. So there must be certain subclasses of 3SAT that actually take exponential time to solve, with a definition more precise than the basic 3SAT problem. Does anybody know what a definition of such a subclass would be? Are there any classic non-3SAT NP-Complete problems where every instance requires exponential time for a solution?

                                      Does this hardness variability keep NP-Complete problems from being good candidates for use in cryptographic protocols?

                                      1. 2

                                        For a lot of them, there isn’t a low-complexity algorithm that can classify inputs as you desire. This is one of the problems for SAT (and SMT) solvers: very similar-looking inputs may be trivial to prove or falsify or may be insanely computationally expensive. Being able to accurately estimate the time take to solve a problem in one of these spaces would be very useful.

                                        1. 2

                                          There’s a certain “uncanny valley” of difficulty for SAT problems, if I recall. It’s at the border between sat and unsat, and typically the hard random instances used in the random category of SAT competitions tend to lie in it. If you’re out of it (many more variables than clauses, or the opposite) it tends to be easier to solve because it’s way under or over-constrained.

                                      1. 8

                                        I like this blog, the posts are very informative. I’d feel so bitter and exhausted if I wanted to push changes into the C standard, but some people are that patient, it’s admirable.

                                        Anyway, I’m not a C expert, but it seems that many C programmers want to have their cake and eat it too, depending on the topic at hand: they want compilers to optimize their code, and think C is the fastest language out there; but at the same time C is believed to be a low level, portable language, even though standard C bars you from even checking an integer overflow. If Linux can’t use standard C how can this myth stand?

                                        1. 4

                                          You can check for integer overflow, you just need to check before you encounter integer overflow and skip the integer overflow codepath in cases of overflow. Compilers are pretty good at recognising this kind of idiom and so often turn it into a branch-on-carry.

                                          To give a concrete and more interesting example, consider integer division by zero. On x86, this will usually trap and on *NIX it’s (confusingly) controlled via the floating point environment whether it traps or not and you get SIGFPE if it does. If you write a/b in your C code and b is zero, then on weird architectures like x86, once you reach that line it will explode. You don’t check this by writing a/b and then trying to check if the result is the result of integer division by zero in some weird way, you do it by writing c = b==0 ? oopsValue : a/b. Whatever the compiler does, your code will explode if you don’t do the check (on some architectures, in some modes) and so correct code will never execute the division by zero and a compiler is completely free to assume that on any path where a/b is reachable, b is non-zero.

                                          1. 2

                                            Another thing to consider is that C does not expose any way to check the overflow bit that exists in many CPUs after a calculation. It might be cognitively simpler to program in this way. But we can’t really do it in C. Assembly is always an option.

                                            1. 5

                                              no you’re not understanding the problem. Integer overflow is well defined everywhere, but C and C++ have adopted an unsupported nonsense stand that it is undefined - not implementation or specification defined, outright ub. It’s not a matter of “how do I detect that overflow happened”, the compiler writers decided to take an insane decision that if a behavior is not defined, you can optimize any code by saying the the UB operation cannot happen. So a hypothetical “did this overflow” flag is irrelevant, overflow is UB so the “did this overflow” flag is definitionally always false.

                                              For the flag (or similar) to make sense you would have to define overflow behavior. If you defined overflow behavior these nonsense “optimizations” go away: the only reason “a+b<a” isn’t a valid overflow check is because overflow is bogusly considered UB.

                                              1. 3

                                                The problem is that signed overflow isn’t defined for every machine where a C compiler exists. There are architectures out there that will trap on signed overflow, much like the x86 will trap when trying to to an integer division by 0 [1]. And even if there isn’t a trap on overflow, what will the result of INT_MAX + 1 be? You get three different answers, depending if the machine is sign magnitude (-0), 1s complement (-INT_MAX) or 2s complement (-INT_MAX-1). Of course, you’ll have to search far and wide these days for any computer that isn’t 2s complement these days [2], but they were still very popular throughout the 70s and into the 80s, when the C standard committee started its work, and every vendor wanted their C compiler to be labeled “standard” with the minimum amount of work, thus all the undefined behavior of C.

                                                Hindsight is 20/20.

                                                [1] On the VAX, a 2s complement machine, you can have the CPU trap or not on signed overflow. This can be done on a per-function basis. The MIPS, also a 2s complement machine, has instructions that will trap on overflow, and a second set of instructions that will not trap on overflow.

                                                [2] Every system I’ve ever used for the past 40 years has always been 2s complement, with defined overflow behavior.

                                                1. 5

                                                  No, signed overflow is defined on every architecture you just listed. “Not the same across all platforms” is not “this behavior is undefined”.

                                                  I did not say “this behavior must be defined by the specification” I said this has no justification for being “undefined behaviour”. Implementation defined and unspecified behavior exist specifically to handle behaviors the differ between targets, and that’s what these should be.

                                                  You need to understand there is a very real an meaningful difference here: by stating overflow is UB the compiler can incorrectly pretend overflow cannot happen, and then creates security vulnerabilities. The specification should actually say that overflow is implementation defined or unspecified - exactly which probably lands in language lawyer town, but for a developer doesn’t matter, the end result is the same: the behavior of overflow becomes consistent throughout the codegen from the compiler, and the compiler can’t just pretend that overflow does not/cannot happen.

                                                2. 1

                                                  the compiler writers decided to take an insane decision that if a behavior is not defined, you can optimize any code by saying the the UB operation cannot happen

                                                  This is not insane. Specifying something as UB means that the algorithms in a compiler can be written with the assumption that the input program does not do these things. When that is the case the code will be compiled correctly. When that is not the case there is no guarantee about the behavior of the output program. That is why it is called ‘unspecified behavior’

                                                  1. 3

                                                    The whole point of the article is that this definition of what UB means - that now the compiler got to choose what would happen, not the hardware it was running on - was introduced decades after the widespread adoption of C.

                                              2. 2

                                                It’s important to recall that when gcc introduced the definition of UB that said “I can optimize assuming the UB of overflow cannot happen” their were no builtins to test, so a whole bunch of people had to go through a lot of code working out the correct set of operations - often extremely expensive ones like division, which would also require its own UB check. This is all for the “performance” gained by pretending that overflow is not well defined behaviour on every platform ever made.

                                                1. 1

                                                  Sure, you want to check beforehand if it’s simpler. But why is C only now adding some operators for checked addition and the likes? It’s tricky to check for overflow beforehand, especially if you have a longer chain of operations, so it’s another source of errors piling up. Having standard operators would also, I suppose, help optimizers?

                                                  1. 1

                                                    Part of the goal of C has always been portability. This means that every construct in the core language is expected to be possible to lower to something simple on every architecture. On some architectures, it’s trivial to check a carry flag for overflow, on others it’s more complex (RISC-V and MIPS both lack this, though MIPSr6 finally fixed it for addition, with an encoding trick where add with the lower source operand register first does an add, with the opposite order returns the carry bit).

                                                    Most compilers have had overflow-checked builtins, and before that a lot of projects had inline assembly helpers. It’s the kind of thing that, for C, probably belongs in the standard library rather than the core language.

                                                2. 2

                                                  I like it too. It’s really well researched and thought out. But fundamentally I disagree with the conclusion.

                                                  UB is UB. you get nothing. It has to be like this. The freedom of the standards can and will be exploited to its fullest. This isn’t because implementors are malicious (except in certain situations where pathological implementations are created to try to stress these things) but it’s the nature of implementing semantics preserving transformations. We simply can’t write a compiler that “understands” the intention of code and rewrites it without breaking it.

                                                  The arguments about the difficulty of working with integers are valid though. And it is something we can and should address. I think concepts like ‘boring c’ are a realistic way to address the problem: we need the expectations to be codified into a standard before compilers can be rewritten to respect them.

                                                  We can also look at more semantic analysis that tries to identify some common examples of UB but the responsibility is fundamentally on the programmer.

                                                  1. 6

                                                    It has to be like this.

                                                    It doesn’t though, that’s the whole point. We got this way through a particular evolution of circumstances: the standard started off as a formalization of the common behaviors of what different compilers do on different systems. Compilers use the unportable parts of the standard to add optimizations that make them generate faster code, and computers are slow so the standard consciously doesn’t include things (like integer overflow being a runtime check) that will make compilers suffer. Computers grow more powerful and more uniform in functionality and compilers get better at exploiting edge cases and shit like this starts happening. Now if the standard tries to define those edge cases, the compiler implementors scream bloody murder… not because they’re malicious but because it will cause a performance regression and people judge compilers by the performance of their generated code. It’s not a technical problem, it’s a human problem.

                                                    In a language as low-level as C you will always have some UB because there are no handrails. As you say, semantic preserving transformations are hard, and some things have meaning that can only be known at runtime.. But there is zero reason your compiler can’t have, say, checked integer math that errors safely on overflow. Indeed, by the rules of UB in the standard this is totally fine! But no compiler does this… because it would be slower.

                                                    Compiler implementors value speed over UB sanity, because their users value speed over UB sanity, because it’s a lot easier to compare compiler speed than compiler UB sanity.

                                                    1. 5

                                                      Compilers use the unportable parts of the standard to add optimizations that make them generate faster code, and computers are slow so the standard consciously doesn’t include things (like integer overflow being a runtime check) that will make compilers suffer.

                                                      No, it’s not about inserting runtime checks for overflow. Overflow happens whether you check or not. The reason compiler writers are so hellbent on signed overflow specifically being UB, is that it means in the most idiomatic for loop for (int j = 0; j < N; j++) { ... } they can claim that they have proved the for loop will terminate and/or perform vectorizing optimizations if internal multiplication in the loop are guaranteed to never decrease.

                                                      In a language as low-level as C you will always have some UB because there are no handrails.

                                                      Sure, you may have some, but even in C with “no handrails” it does not need what it has done.

                                                      There are incredibly few places where you have behaviour that is UB, and the vast majority of UB in C has no reason to be UB. There is for example, no actual real justification for integer overflow being UB in C, as demonstrated by unsigned overflow not being UB - the only reason this is still up for debate is that there are compiler benchmarks that benefit from pretending overflow is not a thing.

                                                      The places where you have unavoidable UB are for example using a pointer outside of the range of the target object, use of a pointer after it has become dead, use of a garbage pointer, access to uninitialized memory*, etc. In other words places where you cannot define what will happen. I’ve *’d uninitialized memory, because this can also just be fixed by saying all memory should be initialized.

                                                      1. 1

                                                        Glad we agree.

                                                      2. 3

                                                        But no compiler does this… because it would be slower.

                                                        I find these comments extremely confusing as someone who feels very familiar with C, and has read bits of the C standard. Are the sanitizers provided by gcc, and clang insufficient for your needs? It feels to me that a lot of people are trying to standardize that C compiles by default with sanitizers enabled when (in my opinion) they chose to compile without them. I suppose, if one is targeting an obscure compiler that doesn’t support equivalents to gcc’s, and clang’s sanitizers, that I would understand your frustrations.

                                                        Could you help me out in understanding why sanitizers aren’t meeting your desires, or if there is some fallacy in my reasoning? I am hoping to clear up my confusion as I feel that I am missing something with how often I see people say they want to remove some undefined behavior from the standard.

                                                        1. 3

                                                          I’m not sure about the other implementations, but Clang’s UBSan has the same completeness problem as any dynamic analysis. For example, it converts signed addition into overflow-checked signed addition with a branch to a check on overflow. If your test workload does not trigger signed overflow then it will never be a be triggered but that doesn’t mean that it’s fine in production where a malicious adversary may provide input data that does trigger the overflow.

                                                          1. 2

                                                            but that doesn’t mean that it’s fine in production

                                                            So ship with sanitizers enabled in production code? Am I missing something here?

                                                            1. 2

                                                              If the overheads are low enough, you could ship them, though I’ve heard complaints that they’re not. Now you still have a denial-of-service vulnerability in your code. This is avoidable if a language either requires explicit checking of all potential overflows or provides integers with sub-word ranges so that overflow is statically avoidable. C has a bunch of things that make this much more of a problem than it should be. For example, the result of T* - T* is ptrdiff_t, which is a signed quantity, but array indexes are size_t, which is a signed quantity of the same size. This makes it easy to write code that won’t ever actually overflow but can’t be statically proven not to.

                                                              1. 1

                                                                The problem is not the overhead of checking for overflow. The problem is not that the compiler is not checking for overflow.

                                                                The problem is that the compiler is deciding that the overflow cannot happen and then optimizing from that. I am not, and I believe that most involved people are not, advocating for the compiler generating overflow checks or anything. What we are saying is the compiler should not be using a clearly bogus decision that overflow is undefined for signed arithmetic, and therefore it’s ok to pretend it doesn’t happen.

                                                                When this exciting definition of what a compiler could do for UB was introduced, the first thing it did was create a large an unending swathe of security bugs. Security bugs that took years to discover.

                                                                You could claim this response to UB is acceptable, but then the difference between something being classified as UB vs implementation defined or unspecified behaviour in the spec becomes security critical, and so the old habit of being lazy and classifying edge cases as UB is not acceptable, and the specification needs to be updated appropriately. But this definition of UB has been adopted solely to help compiler benchmarks, it’s why signed overflow is UB, but unsigned overflow is not, despite the exact same conditions apply to both: it’s just compiler benchmarks use for(int ...) instead of for(unsigned ..). Similarly the optimizing out null dereferences because “that’s UB” which also introduced security vulnerabilities. It’s telling that there are no optimizations that come from actual undefinable behavior (OoB, UaF, uninitialized memory, etc), but the definable cases have all resulted in “optimizations” that create security bugs.

                                                            2. 1

                                                              Are the sanitizers provided by gcc, and clang insufficient for your needs?

                                                              If you can catch this behavior at compile-time, why isn’t it part of the compiler?

                                                              If you have to insert checks at runtime, they should be opt-out instead of opt-in.

                                                      1. 12

                                                        On an episode of the On The Metal podcast, Jonathan Blow said something to the effect of “we’re trading about 10x performance by using Python, but are we really getting 10x the developer experience back on the trade?” It does seem like if being dynamic is so slow (which it is), it also ought to be an even better experience using it, but it’s generally not, at least, not yet.

                                                        1. 6

                                                          I think the problem with Blow’s argument (in my mind) is that:

                                                          a) Time to value is often way more important than performance, and the language + ecosystem have this in spades. b) Software requirements are dynamic c) No one is choosing Python for very performance critical applications

                                                          Also, Python isn’t slow because it’s dynamic. Python is slow because it’s interpreted. I bet heavily optimized, and ahead of time compiled Python could be within a factor of 2x compared to many equivalent C programs, and beat the arbitrary 10x number generally. The difficulty in getting there can’t be understated, however.

                                                          1. 12

                                                            Python is slow because it’s interpreted

                                                            This used to be a pet peeve of mine because like … Python is obviously compiled; all Python becomes bytecode before it’s interpreted. That’s what .pyc files are.

                                                            Then someone told me that Python bytecode is basically just a weird serialization format of the AST, meaning like … it’s compiled, but in a way that doesn’t hardly benefit from the compilation at all, as far as I can tell? All it does is save you having to parse? Anyway, the takeaway I got from that is that judging dynamic languages in general by how well Python does just isn’t fair, and Python should be taken as an extreme example of a design that can’t be well optimized. It’s slow because it’s Python, not because it’s dynamic. Basically every other dynamic language gets this right where Python gets it wrong.

                                                            1. 6

                                                              It’s slow because it’s Python, not because it’s dynamic. Basically every other dynamic language gets this right where Python gets it wrong.

                                                              Eeeeh. I’m not sure about this generalization? What are you arguing that Python gets wrong but other languages get it right?

                                                              It’s easy to prove that interpreted languages are slower that their compiled counter parts. But, Julia is dynamic, compiled, and fast. As is compiled Common Lisp. Dynamic languages do not imply slow. Dynamic interpreted languages, however, tend to be slow without some sort of native compilation (including JITs, for instance. See PyPy, luajit, Ruby’s jit, etc.) and even then…

                                                              1. 6

                                                                It’s not that Python is “wrong” exactly, but rather that Python made design choices that make fast interpreter hard to write.

                                                                IIRC, in particular the dynamic typing & introspection capabilies combine to make it hard to nail down the types of objects at compile time, so the bytecode is forced to be very high level - you’re effectively calling functions on objects all the time, unable to specialise down to optimised routines that can assume that a variable is a given type. The problem (for a python compiler) is that at any point the variable in question might change type, so unless the compiler can prove that this isn’t the case it has to generate generic code for every operation & can’t specialise or inline anything.

                                                                With more advanced program analysis you can deduce the types of some variables but interpreted languages that don’t allow Python’s level of dynamic flexibility are always going to have a much easier time creating fast low-level bytecode or using a JIT to generate fast machine code.

                                                                1. 4

                                                                  I think it’s also a matter of how much engineering time is thrown at the problem. JavaScript shares many of the same issues with Python, but JS engines have become quite fast due to the massive number of optimization tricks they pull. I think CPython also has a goal of being easy to understand and hack on, which might be an additional barrier to such massive improvements.

                                                                  1. 1

                                                                    It’s not that Python is “wrong” exactly, but rather that Python made design choices that make fast interpreter hard to write.

                                                                    I guess I don’t understand what choices Python made that other popular dynamic + bytecode interpreted languages didn’t? That’s my question here.

                                                                    In the Languages Benchmarks Game (where > implies faster): PHP > Perl > Python > Ruby > Lua.

                                                                    1. 3

                                                                      The biggest problem with Ruby (and I haven’t done enough Python to say whether this is true there as well but I suspect it is but maybe to a slightly lesser degree?) is that idiomatic code simply invokes dynamic dispatch everywhere. (In Ruby’s case it’s thru message passing; in Python’s it would be lookup as a hash-table access on the class’s method table instead I think?) While it’s possible to write code using lambdas, etc which never invokes message passing, such code would be considered very unusual. Ruby and Python usually get bucketed in the same class of “just about as slow as it gets” with one or the other pulling ahead in some specific area.

                                                                      Meanwhile Lua’s pervasive use of closures and locals for nearly everything results in much fewer lookups for calls. Plus in cases where you are calling a function from a table in a tight loop and a bottleneck is identified, it’s usually trivial to optimize it (in “userspace”) to a local once you’ve identified the issue, if you’re not on LuaJIT where the runtime does it for you.

                                                                      Of course, it’s easy to write Lua code that has dispatch overhead by shoving everything in a table, but at the same time the cost of such code is pretty clear, vs Ruby where the slow way is the default for everything.

                                                                      1. 2

                                                                        I’d truly like to see a benchmark of a real world script in Ruby, Python, Lua to do this comparison. Python is flexible enough that the use of classes isn’t necessarily standard, but used when it makes sense. And, from what I’ve seen with Lua the same is true. If it makes sense to use a OOP, then methods will be devised somehow. Both Lua and Python have heavily optimized core datastructures (tables, or dicts + lists), and my guess is that performance differences between them are negligible.

                                                                        Modules in both languages are a wrapper over their table / dict implementations, and local variables might go to Lua because it’s likely an array index, though it’s likely a stack index operation in Python.

                                                                        >>> def foo(x, y):
                                                                        ...     a = x + y
                                                                        ...     print(a)
                                                                        ... 
                                                                        >>> import dis
                                                                        >>> dis.dis(foo)
                                                                          2           0 LOAD_FAST                0 (x)
                                                                                      2 LOAD_FAST                1 (y)
                                                                                      4 BINARY_ADD
                                                                                      6 STORE_FAST               2 (a)
                                                                        
                                                                          3           8 LOAD_GLOBAL              0 (print)
                                                                                     10 LOAD_FAST                2 (a)
                                                                                     12 CALL_FUNCTION            1
                                                                                     14 POP_TOP
                                                                                     16 LOAD_CONST               0 (None)
                                                                                     18 RETURN_VALUE
                                                                        

                                                                        I don’t know – these look like index operations, not dict lookups. We might be splitting hairs. Gotta go LOAD_FAST

                                                                        1. 2

                                                                          They’re not index operations though. Look at the implementation of BINARY_OP_ADD in the python interpreter in https://github.com/python/cpython/blob/main/Python/generated_cases.c.h

                                                                          TARGET(BINARY_OP_ADD_INT) {
                                                                                  PyObject *right = PEEK(1);
                                                                                  PyObject *left = PEEK(2);
                                                                                  PyObject *sum;
                                                                                  assert(cframe.use_tracing == 0);
                                                                                  DEOPT_IF(!PyLong_CheckExact(left), BINARY_OP);
                                                                                  DEOPT_IF(Py_TYPE(right) != Py_TYPE(left), BINARY_OP);
                                                                                  STAT_INC(BINARY_OP, hit);
                                                                                  sum = _PyLong_Add((PyLongObject *)left, (PyLongObject *)right);
                                                                                  _Py_DECREF_SPECIALIZED(right, (destructor)PyObject_Free);
                                                                                  _Py_DECREF_SPECIALIZED(left, (destructor)PyObject_Free);
                                                                                  if (sum == NULL) goto pop_2_error;
                                                                                  STACK_SHRINK(1);
                                                                                  POKE(1, sum);
                                                                                  next_instr += 1;
                                                                                  DISPATCH();
                                                                              }
                                                                          

                                                                          and then look at the definition of _PyLongAdd() in https://github.com/python/cpython/blob/main/Objects/longobject.c which is even more complex. Then consider that this is the specialised code for ints - it’s worse for more complex types.

                                                                          Python is slow because every time it does anything at all it has to check the types & do a bunch of other unavoidable busywork because of the semantics of the language. Even LOAD_FAST is relatively slow because the interpreter is a stack machine & variables have to be pushed on to the stack in order to do anything at all with them, which means tweaking reference counts & copying object pointers.

                                                                          As I understand things it has to do all this work because the python interpreter doesn’t know that variables haven’t changed type since the last time it looked at them. In principle you could write a JIT that did proof-tool like work to satisfy itself that a given variable was always an Int, or whatever (this is the kind of thing that V8 does I believe) but the python runtime doesn’t do anything like that. This interpreter style (a bytecode on a stack machine) probably also keeps the code relatively straightforward, at the expense of a lot of extra memory pressure shuffling data around at runtime.

                                                                          Python has always been written for ease of programming itself & ease of programming for people writing it. Performance has never been a priority.

                                                                          1. 1

                                                                            Most of what you are describing as slow parts of the interpreter are just implementation choices that could be changed. Yes, the types constantly need to be checked because they can change. That’s a property of JavaScript, too. Jython doesn’t have to “constantly tweak reference counts” because it runs on the JVM where tracing GC is used instead.

                                                                            Yes, there are parts of Python’s semantics that make it challenging to make performant. My overarching point, in this whole thread, is that this fact might limit the over all expected performance, but it doesn’t mean that Python can’t be sped up significantly despite it.

                                                                            1. 1

                                                                              Sure, if you put the same level of engineering as went into V8 into a Python interpreter, you could probably get something as fast, at the expense of some loss of compatibility.

                                                                              That’s a big ask though & also doesn’t seem to be a direction the existing Python devs want to go in.

                                                                              1. 1

                                                                                at the expense of some loss of compatibility.

                                                                                What compatibility? C extensions? Sure. Python is a language first, and a community second. If your goal is “fast Python” the hurdle is “Python’s obvious implementation is interpreted due to its semantics.” If you want community compatibility for all that’s on PyPi, then it’s going to be a much greater undertaking.

                                                                                My point continues to be that Python could be made much faster, and that the dynamic nature of it does not completely discount all performance improvement opportunities like other folks in this thread keep suggesting.

                                                                                1. 1

                                                                                  No one is saying it’s impossible, just that it’s as hard or harder than making JS fast & that took Google level expenditure over many years.

                                                                  2. 2

                                                                    Eeeeh. I’m not sure about this generalization? What are you arguing that Python gets wrong but other languages get it right?

                                                                    I’m saying (and again this is based on what I’ve been told) that if Python’s bytecode is just “let’s take that AST that we just parsed and dump that out to disk just so we don’t have to parse it again” instead of actually being designed for performance, then it’s doing much worse than every other non-toy dynamic language, because the compilation process is hardly making any meaningful transformation at all.

                                                                    Even without JIT, a bytecode compiler that does meaningful compilation in the bytecode will absolutely beat the pants off Python.

                                                                    It’s easy to prove that interpreted languages are slower that their compiled counter parts

                                                                    That’s the thing; there really aren’t any interpreted languages in widespread use, other than Python (which is compiled, but in a way that’s so trivial that it might as well be interpreted.) Every other dynamic language in widespread use either compiles to bytecode, or compiles to native code, or both. I believe the last holdout was R, which added a bytecode compiler in 2011, but of course this depends on how you define “widespread use”.

                                                                    I guess it’s possible that there are some other bytecode formats used by other languages that make the same mistake as Python and just serve as a serialization format for their AST; if that’s what you mean then I’m interested to hear more. I don’t know of any such languages myself.

                                                                    1. 3

                                                                      I’m saying (and again this is based on what I’ve been told) that if Python’s bytecode is just “let’s take that AST that we just parsed and dump that out to disk just so we don’t have to parse it again” instead of actually being designed for performance

                                                                      CPython compiles the AST to a stack based bytecode interpreter, which you might recall is similar to the JVM. The instructions are here.It does a number of peephole optimizations as well. It’s certainly the case that there’s always room for more compiler optimizations, and probably also room for super instructions (combinations of common instructions to reduce dispatch overhead). Python is probably no less sophisticated than Ruby, Lua, Perl in the optimizations it does in it’s bytecode interpreter. I’m pretty sure they all use switch statement, for instance, rather than a more sophisticated threading model. Jython, an implementation equivalent to Python 2.7, targets the JVM, and JVM bytecode, fwiw, if we’re looking for arguments on sophistication of the interpreters.

                                                                      then it’s doing much worse than every other non-toy dynamic language, because the compilation process is hardly making any meaningful transformation at all.

                                                                      I’m not sure which dynamic, interpreted, languages are doing loads of optimizations when compiling to their internal bytecode for execution. Optimization takes time, which increases execution latency. I briefly looked at Ruby’s compiler (which I think is now based on YARV) and they seem to do some optimizations, but it’s not obviously clear to me what passes (if any) they make, so most likely peephole, and “let’s avoid the obvious stupid thing” in different cases. Looking at the lua code generator, it does some peephole optimizations as fixups, and tries to be smart in specific cases to avoid unnecessary things. It does not appear to be doing sophisticated control flow analysis and doing multiple passes to generate the tightest possible bytecode.

                                                                      I’ve not dug into Perl, or Raku, but both of those languages also have bytecode interpreters internally, and I’m sure are doing some optimizations, but not significant compilation passes.

                                                                      And, of course the reason for this is latency.

                                                                      That’s the thing; there really aren’t any interpreted languages in widespread use, other than Python (which is compiled, but in a way that’s so trivial that it might as well be interpreted.)

                                                                      The 4/5 languages mentioned above have nearly the same execution design and are in widespread use. I suppose I should look at PHP, which I’m pretty sure is also now a similar design of “compile to bytecode with some small number of optimizations.” Lua is a “register machine.” I know Python and Perl use stack machines, and I’m not entirely sure about YARV. Point is, I’m not sure you’re making a coherent argument?

                                                                      1. 2

                                                                        I’m not sure you’re making a coherent argument?

                                                                        Actually I think it sounds more like the argument I’m making might have been based on something incorrect I read somewhere, and that the Python bytecode process is doing more than that?

                                                                        1. 2

                                                                          Perhaps!

                                                                      2. 1

                                                                        Even without JIT, a bytecode compiler that does meaningful compilation in the bytecode will absolutely beat the pants off Python.

                                                                        See my comment here

                                                                    2. 6

                                                                      This was one of the hardest lessons to learn when writing a derivative of Python: bytecode is not necessarily fast, just packed. I’m glad that I learned it, though.

                                                                      1. 1

                                                                        it’s compiled, but in a way that doesn’t hardly benefit from the compilation at all, as far as I can tell?

                                                                        To expand/confirm what follows: the stronger statement here is that not only it “doesn’t” benefit from compilation, it also “can’t” benefit from compilation, because executing bytecode isn’t the bottleneck. The bottleneck is the C runtime to implement very dynamic semantics of Python:

                                                                        https://blog.kevmod.com/2016/07/02/why-is-python-slow/

                                                                      2. 4

                                                                        a) Well, Blow is a game developer, so of course performance is crucial to him. He’d also dismiss most GC’d languages because of unpredictable pause times (and I know it’s less true with some modern GCs). He’d probably dismiss any non-natively-compiled language.

                                                                        b) Does this imply that the typing has to be dynamic? You can change types when you program, it’s fine.

                                                                        c) Sure, it’s a self-fulfilling prophecy. Python is so slow that no one picks it up for performance-critical software.

                                                                        Really if you restrict the argument to gamedev, I think Blow is right. In general, if you include web development, sysadmin, etc. then there are good reasons to use

                                                                        Now about the dynamic part: dynamism is a big part of why python is slow. To make it fast, you need some form of JIT, because you have to speculate about values and undo your guesses if you’re wrong. Even the bytecode, like /u/technomancy says, is slow. There’s hashtable lookups everywhere. Python’s design is just… not very amenable to performance, and that’s partly because it’s incredibly dynamic and relies on so many table lookups semantically.

                                                                        1. 2

                                                                          a) Well, Blow is a game developer, so of course performance is crucial to him. He’d also dismiss most GC’d languages because of unpredictable pause times (and I know it’s less true with some modern GCs). He’d probably dismiss any non-natively-compiled language.

                                                                          Right! If performance is your goal, look elsewhere.

                                                                          b) Does this imply that the typing has to be dynamic? You can change types when you program, it’s fine.

                                                                          No. I did not mean to imply dynamic requirements mean dynamically typed language. There are certainly lots of refactoring arguments on both sides of the coin. :)

                                                                          c) Sure, it’s a self-fulfilling prophecy. Python is so slow that no one picks it up for performance-critical software.

                                                                          ding, ding, ding!!! It’d be hard, it might require lots more memory, but you could make Python loads faster than it is today.

                                                                          Really if you restrict the argument to gamedev, I think Blow is right. In general, if you include web development, sysadmin, etc. then there are good reasons to use’

                                                                          Agreed.

                                                                          Now about the dynamic part: dynamism is a big part of why python is slow. To make it fast, you need some form of JIT, because you have to speculate about values and undo your guesses if you’re wrong. Even the bytecode, like /u/technomancy says, is slow. There’s hashtable lookups everywhere. Python’s design is just… not very amenable to performance, and that’s partly because it’s incredibly dynamic and relies on so many table lookups semantically.

                                                                          There are lots of dynamic languages that are “fast.” Common Lisp, Julia, Chez Scheme are all pretty fast. Dynamic doesn’t mean “slow.” I agree that Python’s semantics and level of dynamism are not obviously amenable to performance, but the bulk of Python code in existence today isn’t taking advantage of all this dynamism. And even if you did, it’s still not impossible to improve upon.

                                                                          1. 1

                                                                            There are lots of dynamic languages that are “fast.” Common Lisp, Julia, Chez Scheme are all pretty fast. Dynamic doesn’t mean “slow.” I agree that Python’s semantics and level of dynamism are not obviously amenable to performance, but the bulk of Python code in existence today isn’t taking advantage of all this dynamism. And even if you did, it’s still not impossible to improve upon.

                                                                            Yeah I think we basically agree. “Dynamic” is also a spectrum: for example, scheme has lexical scoping, and you can potentially pass variables in registers, perform optimization passes, compile to native code, etc. Julia is clearly designed to be JIT-compilable as well (types are runtime but are used to resolve overloading, if I understand correctly; you thus get monomorphic implementations of functions, heavily optimized, and which implementation is used is decided by runtime types).

                                                                            At the same time, Python allows you to change and observe pretty much anything at any time, so you can’t do anything behind the user’s back. My impression is that people have tried a lot to accelerate python (since unladden swallow, etc.) but it always breaks some of the semantics or some of the C API, all of which is part of the language in practice. Maybe it’s even that “python” the language could be optimized reasonably well, but “cpython” is basically impossible to optimize, and it’s what people mean when they talk about “python”.

                                                                        2. 3

                                                                          Also, Python isn’t slow because it’s dynamic. Python is slow because it’s interpreted.

                                                                          You have it backwards, strangely enough. It’s interpreted because it’s dynamic.

                                                                          Python is so dynamic that even if you try to compile it ahead of time, all you do is generate a pile of machine code that is also slow because it’s still doing all of the dynamic dispatch and hash table lookups that make Python slow. Many people have tried to make ahead-of-time compilers for dynamic languages and they rarely get anywhere near the performance of compiled static languages.

                                                                          Because you can’t compile Python’s insane level of dynamism to efficient native code anyway, you may as well just do a bytecode interpreter which is way simpler, more hackable, and makes it easier to do runtime introspection, debugging, etc.

                                                                          1. 4

                                                                            You have it backwards, strangely enough. It’s interpreted because it’s dynamic.

                                                                            I don’t think it’s strange at all, nor do I think it’s backwards. The easiest way to build a highly dynamic language, is certainly to interpret it. We agree here. But that’s not the only way to do so. You know that Knuth quote about pre-mature optimization? 97% of the time it doesn’t matter, but that 3%, oh boy, make the 3% more efficient.

                                                                            But, in a language like Python, you always pay for dynamism that you’re using “3% of the time.” That doesn’t make much sense at all. Consider the common uses of Python today:

                                                                            1. Straight up numerical computing, via IPython notebooks, etc. Half of this is calling out to Python C extensions. It’s also generally interactive, and not likely to utilize dynamic features outside of a few basic things. Here’s the thing: the slow parts (the large numerical operations) are slow enough that paying extra for dynamism isn’t a big deal. Interactive computing can deal with some latency.
                                                                            2. Scripts for system utilities, package managers, text processing, etc: These are typically written in Python to avoid commiting to static types, but often aren’t introspecting / disassembling classes, or doing anything crazy. You’re parsing text, generating new text, calling system utilities, executing syscalls, etc.You’re likely not using a REPL, and you’re almost certainly not doing things that require insane amounts of introspectability of the interpreter, or the bytecode.
                                                                            3. Web Servers: In the web server case, or any of the other uses of “run this Python code on a server somewhere and interact with it via a Layer 7 network Protocol” you’re not exposing the dynamic capabilities of the interpreter – that’d be a security nightmare, or a performance hit.
                                                                            4. Everything else: You’re almost certainly not doing crazy magic things like introspecting the runtime to hack to rewrite bytecode in odd ways, or any other magic thing you might consider.

                                                                            So what are we paying for? “97% of the time” we’re writing code that could be sped up by compiling it directly to its CPython API equivalent in C. We know this because C extensions, PyPy, and previous experiments like Cython, and the really old Psyco actually significantly increase performance. For the 3% of the time where this doesn’t work, we can afford to take the hit of throwing out the compiled stuff, and interpreting it.

                                                                            (And, as I’ve said many times in this thread: Yes, this is hard work that might take years)

                                                                            Because you can’t compile Python’s insane level of dynamism to efficient native code anyway, you may as well just do a bytecode interpreter which is way simpler, more hackable, and makes it easier to do runtime introspection, debugging, etc.

                                                                            I guess it depends on your goals. There’s obviously a lot of time and money invested in Python (specifically), and only some of that time seems to be devoted to raw performance improvements. It doesn’t seem, for instance, that the Python community has fully embraced PyPy (it’s not been mainlined, for instance), so maybe there’s a general consensus that Performance doesn’t matter more than maintenance, hackability, understandability. Not sure those are the same principles I’d work towards, but then again, I don’t really program in Python anymore. But, there’s clearly members of the community that feel otherwise, e.g. Pyston.

                                                                            1. 2

                                                                              Mypyc claims about 5x speed up for aot compilation. That seems pretty respectable.

                                                                          2. 3

                                                                            It’s a strange comment. Why should we care if the ratio is 1:1? Most of the time performance is not an issue.

                                                                            1. 1

                                                                              The trade off seems to pay dividends in the top ranks of Advent of code, at least. I appreciate how static typing and advanced algebraic times slow me down and force me to think of correctness first and foremost while on the job, but in a throw away program at the beginning of the day in December, I feel I progress faster using a dynamic language.

                                                                            1. 21

                                                                              Oh is it time to hype dsls again? That makes sense as we’re starting to all get a little embarrassed about the levels of hype for functional programming.

                                                                              I guess next we’ll be hyping up memory safe object oriented programming.

                                                                              1. 16

                                                                                I’m just sitting here with my Java books waiting for the pendulum to swing back…

                                                                                1. 9

                                                                                  I’m going to go long on eiffel books.

                                                                                  1. 6

                                                                                    I think a language heavily inspired by Eiffel, while fixing all of its (many, many) dumb mistakes, could go really far.

                                                                                    1. 2

                                                                                      I’ve just started learning Eiffel and like what ive seen so far, just curious what do you consider its mistakes?

                                                                                      1. 8
                                                                                        1. CAT-calling
                                                                                        2. Bertrand Meyer’s absolute refusal to use any standard terminology for anything in Eiffel. He calls nulls “voids”, lambdas “agents”, modules “clusters”, etc.
                                                                                        3. Also his refusal to adopt any PL innovations past 1995, like all the contortions you have to do to get “void safety” (null safety) instead of just adding some dang sum types.
                                                                                      2. 1

                                                                                        Racket!

                                                                                  2. 14

                                                                                    I, personally, very much doubt full on OOP will ever come back in the same way it did in the 90s and early 2000s. FP is overhyped by some, but “newer” languages I’ve seen incorporate ideas from FP and explicitly exclude core ideas of OOP (Go, Zig, Rust, etc.).

                                                                                    1. 5

                                                                                      I mean, all of those languages have a way to do dynamic dispatch (interfaces in Go, trait objects in Rust, vtables in Zig as of 0.10).

                                                                                      1. 13

                                                                                        And? They also all support first-class functions from FP but nobody calls them FP languages. Inheritance is the biggest thing missing, and for good reason.

                                                                                        1. 12

                                                                                          This, basically. Single dynamic dispatch is one of the few things from Java-style OO worth keeping. Looking at other classic-OO concepts: inheritance is better off missing most of the time (some will disagree), classes as encapsulation are worse than structs and modules, methods don’t need to be attached to classes or defined all in one batch, everything is not an object inheriting from a root object… did I miss anything?

                                                                                          Subtyping separate from inheritance is a useful concept, but from what I’ve seen the world seldom breaks down into such neat categories to make subtyping simple enough to use – unsigned integers are the easiest example. Plus, as far as I can tell it makes most current type system math explode. So, needs more theoretical work before it wiggles back into the mainstream.

                                                                                          1. 8

                                                                                            I’ve been thinking a lot about when inheritance is actually a good idea, and I think it comes down to two conditions:

                                                                                            1. The codebase will instantiate both Parent and Child objects
                                                                                            2. Anything that accepts a Parent will have indistinguishable behavior when passed a Child object (LSP).

                                                                                            IE a good use of Inheritance is to subclass EventReader with ProfiledEventReader.

                                                                                            1. 10

                                                                                              Take a cookie from a jar for using both LSP and LSP in a single discussion!

                                                                                              1. 4

                                                                                                Inheritance can be very useful when it’s decoupled from method dispatch.

                                                                                                Emacs mode definitions are a great example. Nary a class nor a method in sight, but the fact that markdown-mode inherits from text-mode etc is fantastically useful!

                                                                                                On the other hand, I think it’s fair to say that this is so different from OOP’s definition of inheritance that using the same word for it is just asking for confusion. (I disagree but it’s a reasonable argument.)

                                                                                                1. 2

                                                                                                  Inheritance works wonderfully in object systems with multiple dispatch, although I’m not qualified to pinpoint what is it that makes them click together.

                                                                                                  1. 1

                                                                                                    I’ve lately come across a case where inheritance is a Good Idea; if you’re plotting another of your fabulous blog posts on this, I’m happy to chat :)

                                                                                                    1. 1

                                                                                                      My impression is that inheritance is extremely useful for a peculiar kind of composition, namely open recursion. For example, you write some sort of visitor-like pattern in a virtual class, then inherit it, implement the visit method or what have you, and use this to recurse between the abstract behavior of traversing some structure, and your use-case-specific code. Without recursion you have to basically reimplement a vtable by hand and it sucks.

                                                                                                      Well, that’s my only use of inheritance in OCaml. Most of the code is just functions, sum types, records, and modules.

                                                                                                      1. 1

                                                                                                        Forrest for the trees? When you want to create a framework that has default behaviour that can be changed, extended or overridden?

                                                                                                      2. 4
                                                                                                        • obj.method syntax for calling functions — a decent idea worth keeping.
                                                                                                        • bundling behavior, mutable state, and identity into one package — not worth doing unless you are literally Erlang.
                                                                                                        1. 3

                                                                                                          IMO there is a fundamental difference between Erlang OO and Java OO to the point that bringing them up in the same conversation is rarely useful. Erlang actively discourages you from having pellets of mutable state scattered around your program: sure, threads are cheap, but that state clump is still a full-blown thread you need to care for. It needs rules on supervision, it needs an API of some kind to communicate, etc, etc. Erlang is at it’s best when you only use threads when you are at a concurrency boundary, and otherwise treat it as purely functional. Java, in contrast, encourages you to make all sorts of objects with mutable state all over the place in your program. I’d wager that MOST non-trivial methods in Java contain the “new” keyword. This results in a program with “marbled” state, which is difficult to reason about, debug, or apply any kind of static analysis to.

                                                                                                        2. 2

                                                                                                          In all honesty, you sound quite apologetic to what could be arguably considered objectively bad design.

                                                                                                          Attaching methods to types essentially boils down to scattering data (state) all over the code and writing non pure functions. Why honestly cannot understand how anyone would think this is a good idea. Other than being influenced by trends or cults or group thinking.

                                                                                                          Almost the same could be said about inheritance. Why would fiting a data model in a unique universal tree be a good idea? Supposedly to implicitly import functionality from parent classes without repeating yourself. Quite a silly way to save a line of code. Specially considering the languages that do it are rather verbose.

                                                                                                          1. 5

                                                                                                            Why honestly cannot understand how anyone would think this is a good idea. Other than being influenced by trends or cults or group thinking.

                                                                                                            Here’s a pro tip that has served me well over many years. Whenever I see millions of otherwise reasonable people doing a thing that is obviously a terribly stupid idea, it is always a lack of understanding on my part about what’s going on. Either I am blind to all of the pros of what they are doing and only see the cons, or what they’re doing is bad at one level but good at a different level in a way that outbalances it, or they are operating under constraints that I don’t see or pretend can be ignored, or something else along those lines.

                                                                                                            Billions of lines of successful shipped software have been written in object-oriented languages. Literally trillions of dollars of economic value have been generated by this software. Millions of software developers have spent decades of their careers doing this. The though that they are all under some sort of collective masochistic delusion simply does pass Hanlon’s Razor.

                                                                                                            1. 1

                                                                                                              To be honest, the more I study OOP (or rather, the hodgepodge of features and mechanisms that are claimed by various groups to be OOP), the less room I see for a genuine advantage.

                                                                                                              Except one: instantiation.

                                                                                                              Say you have a piece of state, composed of a number of things (say a couple integers, a boolean and a string), that represent some coherent whole (say the state of a lexer). The one weird trick is that instead of letting those be global variables, you put them in a struct. And now you can have several lexers running at the same time, isn’t that amazing?

                                                                                                              Don’t laugh, before OOP was popular very prominent people thought it was a good idea to have global state in Lex, Yacc, or error handling (errno). So here’s my current guess: the success we attribute to OOP doesn’t really come from any of its overly hyped features. It comes from a couple very mundane, yet very good programming practices it adopted along the way. People attributed to the hyped stuff (such as inheritance) a success they have earned mostly by avoiding global variables.

                                                                                                              Abstract data types are amazing, and used everywhere for decades, including good old C. The rest of OOP though? Contextual at best.

                                                                                                              1. 1

                                                                                                                It has been the opposite for me.

                                                                                                                • typecast everything to and from object in early versions of java
                                                                                                                • EJBs 2
                                                                                                                • Bower package manager. Its creator wrote on stack overflow that he was confused when he created the project and that it was essentially useless.
                                                                                                                • Ruby gems security incident
                                                                                                                • Left pad fiasco
                                                                                                                • Apache web server htaccess configs

                                                                                                                I could go on with more esoteric examples to an ever growing list.

                                                                                                                All these had criticism screaming long before they happened: why?

                                                                                                              2. 3

                                                                                                                Many decisions are only clearly good or bad in retrospect.

                                                                                                            2. 6

                                                                                                              Inheritance is the biggest thing missing, and for good reason.

                                                                                                              That reason being “inheritance was the very first mechanism for subtyping, ADTs, and code-reuse, and people using it got ideas for better mechanisms from it.” ;)

                                                                                                              1. 1

                                                                                                                Exactly!

                                                                                                              2. 3

                                                                                                                The first versions of Simula and Smalltalk didn’t have inheritance either. Self and other prototypal object-oriented languages don’t use traditional inheritance either. We still call all of them object-oriented.

                                                                                                                Honestly, it’s well beyond time that we retire all programming language paradigm terms. Modern languages simply aren’t organized into paradigms they way older simpler languages were.

                                                                                                                It’s like we’re looking at a Honda Accord and arguing over whether it’s a penny farthing or a carriage. The taxonomy no longer makes sense.

                                                                                                            3. 1

                                                                                                              Ah yes and that’s why it’s ripe to have a come back. :)

                                                                                                              Seriously though I expect that the next incarnation will be “oop without inheritance” or something. Probably combined with some large corporation “inventing” gc-less memory management.

                                                                                                              1. 2

                                                                                                                The good parts of OOP never really left. We already have that exact language: Rust. It has formal interfaces (Traits), encapsulation, polymorphism, and gc-less memory management.

                                                                                                                1. 10

                                                                                                                  The main thing about OOP that needs to die is the idea that OOP is a coherent concept worth discussing on its own. Talk about the individual concepts as independent things! It’s much more productive.

                                                                                                                  1. 1

                                                                                                                    Talk about the individual concepts as independent things!

                                                                                                                    IMO OOP these days really means inheritance and an object lifecycle. All the other concepts aren’t really unique to OOP.

                                                                                                                    1. 3

                                                                                                                      I think “OOP” generally means “features of object-oriented languages that I don’t like” to a lot of people. The people using those languages don’t generally get into paradigm arguments.

                                                                                                                      (Personally, I consider inheritance to be common in OOP languages but not a particularly interesting or salient part of them. Many early OOP languages didn’t have inheritance and prototypal ones have an entirely different code reuse model.)

                                                                                                                      1. 1

                                                                                                                        For some people “OOP” means “features of languages I do like”. For instance I’ve seen people include templates/generics/parametric polymorphism and unnamed functions as core parts of OOP… having learned CamlLight (OCaml without the “O”) in college, I confessed I was quite astonished.

                                                                                                                      2. 2

                                                                                                                        You say that but it means different things to different people. I don’t disagree that your definition would be a good one if you could get people to agree on it, but I can’t assume that when other people say “OOP” that’s what they’re talking about.

                                                                                                                2. 1

                                                                                                                  I think it will come back, rediscovered as something new by a new generation disillusioned with whatever has been the cool solves-everything paradigm of the previous half decade. Perhaps this time as originally envisaged with a “Scandinavian school” modeling approach.

                                                                                                                  Of course it never left as the first choice for one genre of software… the creation of frameworks featuring default behavior that can be overridden, extended or changed.

                                                                                                                  Those languages you mention (Go, Zig, Rust) are primarily languages solving problems in the computer and data sciences, computing infrastructure and technical capability spaces. Something is going to be needed to replace or update all those complex aging ignored line-of-business systems.

                                                                                                                3. 11

                                                                                                                  There isn’t really any need to “hype” DSLs because they’re already widely used in all domains of programming:

                                                                                                                  • front end: HTML / CSS / JavaScript, and most JS web frameworks introduce a new DSL (multiple JSX-like languages, Svelte, etc.)
                                                                                                                  • back end: a bajillion SQL variants, a bazillion query languages like Redis
                                                                                                                  • builds: generating Ninja, generating Make (CMake, Meson, etc.)
                                                                                                                    • there at least 10 CI platforms with their own YAML DSLs, with vars, interpolation, control flow, etc.
                                                                                                                  • In games: little scripting languages for every popular game
                                                                                                                  • Graphics: scene description languages, shader languages
                                                                                                                  • Compilers: LLVM has its own TableGen language, languages for describing compiler optimizations and architecture (in the implementation of Go, a famously “not DSL” language), languages for describing VMs (Ruby)
                                                                                                                  • Machine Learning: PyTorch, TensorFlow, etc. (these are their own languages, on top of Python)
                                                                                                                  • Distributed computing: at least 10 MapReduce-derived frameworks/languages; there are internal DSLs in Scala for example, as well as external ones
                                                                                                                  • Mathematics and CS: Coq, Lem, etc.

                                                                                                                  All of these categories can be fractally expanded, e.g. I didn’t mention the dozens of languages here: https://github.com/oilshell/oil/wiki/Survey-of-Config-Languages – many of which are commonly used and featured on this site

                                                                                                                  If you think you don’t use DSLs, then you’re probably just working on a small part of a system, and ignoring the parts you’re not working on.

                                                                                                                  ALL real systems use tons of DSLs. I think the real issue is to mitigate the downsides

                                                                                                                  1. 1

                                                                                                                    Oh yes but at the same time if you haven’t seen the hype for DSLs then you haven’t spent long enough in the industry to go through that part of the hype cycle. DSLs are what they are and it looks like we might be entering a hype cycle where people want to make them out to be much more.

                                                                                                                    1. 3

                                                                                                                      I don’t agree, I’ve been in the industry for 20+ years, there are plenty of things more hyped than DSLs (cloud, machine learning, etc.)

                                                                                                                      DSLs are accepted standard practice, and widely used, but often poorly understood

                                                                                                                      I’m not getting much light from your comments on the subject – you’ve made 2 claims of hype with no examples

                                                                                                                      1. 2

                                                                                                                        Here’s an example of recent hype https://www.codemag.com/Article/0607051/Introducing-Domain-Specific-Languages

                                                                                                                        Here’s some hype from the year 2000 https://www.researchgate.net/publication/276951339_Domain-Specific_Languages

                                                                                                                        Arguably the hype for 4GLs was the prior iteration of that specific hype.

                                                                                                                        I’m not arguing that DSLs are bad - I’m saying that they’re one of the things on the roster of perfectly good things that periodically get trumpeted as the next big thing that will revolutionize computing. These hype cycles are characterized by attempts to make lots of DSLs when there isn’t a strong need for it or any real payoff to making a language rather than a library.

                                                                                                                  2. 4

                                                                                                                    I know it might sound a bit controversial, but the way I see it we need to reach a new level of abstraction in order for large-scale software development to be sustainable. Some people might say AI is the way forward, or some other new programming technique. Either way I don’t think we’ll get there by incrementally improving on the paradigms we have—in order to reach the next level we’ll have to drop some baggage on the way up.

                                                                                                                    1. 4

                                                                                                                      I mean, humans aren’t getting better at groking abstraction, so I don’t know that “new levels of abstraction” are the way forward. Personally, I suspect it means more rigor about the software development process–if you’re building a tall tower, maybe the base shouldn’t be built with a “move fast and break things” mentality.

                                                                                                                      1. 3

                                                                                                                        Groking abstractions isn’t the problem, at the end of the day abstractions are just making decisions for the users of an abstraction. Over-abstraction is the root of many maintainability woes IMO, the more a programmer knows what’s actually going on underneath the better, but only to the degree that it’s relevant.

                                                                                                                      2. 3

                                                                                                                        I’ve heard it before. DSLs have their place, and some people love them while others hate them. This is one of a rotating cast of concepts that you’ll eventually see rehyped in 10 years.

                                                                                                                    1. 25

                                                                                                                      Markdown will never get beyond developers

                                                                                                                      I think they underestimate how many young people use discord and markdown for formatting their text.

                                                                                                                      there’s no modern semantic elements such as main, article, section, nav, header, footer, figure, picture

                                                                                                                      Well yes, but also, I don’t need that in reddit or discord. And the places that do need it, are better off doing said post processing. Also I’m very glad that I can actually use it instead of a badly behaving WYSIWYG editor (looking at you reddit trashfire that ate my text too many times). Same goes for github/gitea/gogs/gitlab entries.

                                                                                                                      It’s a modern BBCode for “forums”, which is also better in being readable without rendering.

                                                                                                                      1. 15

                                                                                                                        BBCode is an excellent comparison, because I remember the web being riven with unclosed BBCode tags and bad formatting.

                                                                                                                        What Markdown does not do is probably more important than what it does.

                                                                                                                        1. 5

                                                                                                                          What Markdown does not do is probably more important than what it does.

                                                                                                                          This. Markdown’s strength is that it doesn’t try to be all things. There’s a megaton of content in the space that sits between scientific paper / book scale organisation, and, well, let’s go for tweets as an example at the other end of the spectrum. What Markdown does really well is cater for that content. If anyone is going to argue that Markdown isn’t good enough, I’ll ask them to show me how many tools they have in their toolbox, and if there’s more than one, their argument is bunk.

                                                                                                                          Also: WYSIWYG is tremendously overrated and I’d argue that in some circles it has hindered content creation rather than supported it.

                                                                                                                          Finally: I can’t help but smile to see the Markdown formatting available just winking at me below this textarea input box in Lobsters.

                                                                                                                        2. 3

                                                                                                                          It’s just a shame Org syntax didn’t beat out Markdown syntax. Org supports such simplicity and much complexity, in a readable plaintext format.

                                                                                                                          1. 17

                                                                                                                            It might be a shame, but it’s a predictable consequence of the fact that org-mode doesn’t have an independent implementation that most people can use. Markdown had a simple perl program that you can call from a makefile or any sort of script to render your text to html. These days there are loads of libraries to choose from, to render on the client, server, GUI, etc. dynamically.

                                                                                                                            1. 5

                                                                                                                              I strongly agree. And that is still a little disappointing, because org-mode was so well suited for that kind of thing. org-babel had so much potential for literate programming. I wish that had broken differently.

                                                                                                                            2. 6

                                                                                                                              While I appreciate Org mode’s outlining features, Org mode’s syntax has some limitations that make me glad it didn’t catch on.

                                                                                                                              • Code is clunky to mark up.
                                                                                                                                • Block delimiters are #+BEGIN_SRC javascript and #+END_SRC instead of Markdown’s ```javascript and ```. That syntax uses three different punctuation characters – it’s hard to remember and type.
                                                                                                                                  • I bet that most authors type that syntax with the three keystrokes <s Tab (using the s structure template), but that only works in Emacs.
                                                                                                                                • The inline code delimiters ~#example~ are harder to visually distinguish from code than Markdown’s inline code delimiters `#example`.
                                                                                                                              • It is difficult to escape special characters. (I was going to write that Org mode did not support escaping, but I just discovered the manual page Escape Character.)
                                                                                                                                • Org mode’s escape character is the zero-width space ‘​’, which is harder to type, read, and edit than Markdown’s backslash ‘\’.
                                                                                                                                • I can’t tell from the manual whether an escaping zero-width space will still appear in exported HTML. If so, then there is formatted text that is impossible to write in Org mode. In comparison, Markdown’s escaping backslashes are never in the output, and even if you think you’ve found something you can’t write, Markdown can fall back to inline HTML.
                                                                                                                                • It’s inconsistent whether you have to type the zero-width space before or after a special character to cancel it out. In Markdown, the backslash always goes before the special character.
                                                                                                                              1. 3

                                                                                                                                There’s a lot of reasons why Org-mode didn’t end up winning out because of complexity.

                                                                                                                                • I could explain this to my elderly grandmother and be fairly confident that she could roughly get it.
                                                                                                                                • For Orgmode its a rite of passage to discover that there is spreadsheet support built in after using it for 6 months.

                                                                                                                                Orgmode is wonderful and terrible in a lot of the same ways Emacs is.

                                                                                                                            1. 26

                                                                                                                              I’m very tired of this “angry person derides everything the industry does and points at very small-scale examples as a better way” thing people often do when giving talks. We should be using notebooks, interactive or visual programming? I’d like to see demonstrations for millions-of-lines codebases. All software should contain a repl to enable live-debugging? I recently tried to download firefox’s debug symbols from debian’s repository, it’s over one gigabyte. If you’re going to make me download more than a gigabyte each time I have to update your software, while almost never needing to debug it, I’m not going to use your software.

                                                                                                                              Pointing at things and saying they’re bad without discussing the constraints that went into building the thing and how they could have been solved differently just makes you sound like an asshole. If your goal is to give a quick overview without being able to go into details, then drop the “this other thing is shit” part of your talk.

                                                                                                                              1. 7

                                                                                                                                Thank you for this comment. I got the same feeling when watching the talk. The content is somewhat interesting but the tone is arrogant and insulting to every person who has different practices or opinions.

                                                                                                                                In particular, the examples are heavily biased towards things that are easy to visualize (either pictures, or small snippets of code that do simple things). It’s a bit like Bret Victor’s talks, where everything just happens to be mostly image oriented. I’d like to see how one visualizes compiler internals, SAT solvers code, etc. because it’s the kind of things I’m interested in, not this fluffy “look at little squares drawn on a canvas” thing.

                                                                                                                                And all the interactive stuff doesn’t come for free. How much code do you need to write to produce the visualizations themselves? How do you debug that code?

                                                                                                                                I also wanted to give a quick try to Glamorous Toolkit, mentioned there, because it looks very interesting. It tooks ages to install, and on my laptop it ran at a horrendous (single digit) FPS count. It was almost laughable. I find it revealing that nowhere in the talk does this person mention performance or memory consumption at all (e.g. in the cell example). The dig at C++ is also uncalled for, imho: it’s a language where people get things done, often semi interactively (in game development, for example; see Dear Imgui for how game devs add interactivity to their code), in an environment where performance and memory consumption matter.

                                                                                                                                One last rant: the tweet “stop the world typecheckers”. No. LSP and type checking inside your editor is interactive too, and I don’t believe that a runtime environment is strictly better or strictly more interactive, since you only interact with a single concrete instance of the program’s state. Typechecking gives you some interactivity on errors that might happen in at least one instance.

                                                                                                                                1. 6

                                                                                                                                  the examples are heavily biased towards things that are easy to visualize

                                                                                                                                  And the attitude toward program representation is heavily biased toward people who can see and visualize in the first place. Yes, that’s most people. I’m afraid my comment on the orange site had a self-righteous tone to go with that of the talk. It’s just frustrating sometimes when people so completely ignore the fact that not all of us have the same abilities, and what’s constraining to some is liberating and equalizing for others. I had the same frustration with one of Bret Victor’s talks.

                                                                                                                                  1. 4

                                                                                                                                    I also wanted to give a quick try to Glamorous Toolkit, mentioned there, because it looks very interesting. It tooks ages to install, and on my laptop it ran at a horrendous (single digit) FPS count. It was almost laughable.

                                                                                                                                    Glamorous Toolkit is in beta. We should avoid judging it too harshly just because we disagree with the person who recommended it. That would be creating a cycle of dismissal and judgement.

                                                                                                                                    1. 3

                                                                                                                                      Feenk has been aggressively advertising gtoolkit on Twitter since at least 2019. They seem to think it’s production ready.

                                                                                                                                      1. 1

                                                                                                                                        Okay, so long as it’s a reaction to the author’s claims and not the claims of a third party. Judge people by their own actions and not how you feel about “their group”.

                                                                                                                                      2. 2

                                                                                                                                        You’re right. I mean the FPS was almost laughable. I don’t know how hard is it going to be for GToolkit to overcome this level of performance degradation though. I would guess that a GUI toolkit written entirely in a dynamic language might have trouble being as performant as something more, hum, “dead” as the OP would say.

                                                                                                                                        1. 3

                                                                                                                                          It’s all good! The talk was pretty dismissive and it’s hard not to take a swing at the pillars of their argument. Just want to make sure that we acknowledge that those pillars are other people who might not have as hard of takes.

                                                                                                                                          1. 2

                                                                                                                                            I appreciate that, thank you :-). It’s easy to be dismissive on the internet.

                                                                                                                                          2. 3

                                                                                                                                            I would guess that a GUI toolkit written entirely in a dynamic language might have trouble being as performant as something more, hum, “dead” as the OP would say.

                                                                                                                                            How about Smalltalk on the original PARC machines (Alto, Dorado, etc.)? If they could do it back then, presumably we can implement a reasonably fast GUI toolkit in a dynamic language on today’s much more powerful machines.

                                                                                                                                        2. 2

                                                                                                                                          One last rant: the tweet “stop the world typecheckers”.

                                                                                                                                          This topic opens up the purpose of development environments. One could seriously argue that there can and should be a difference in quality during exploring code, experimenting, and checking code into the group. The goal of development environment is to speed up the developer while capturing every developer intentions along the way. This often means type checking can be ignored until after the first run.

                                                                                                                                        3. 5

                                                                                                                                          These sort of talks are about showing alternative futures, not about presenting a complete solution. The talk should be seen to have a different aim. Not every solution has to scale. I teach high schoolers and I’d kill for a better way for my students to inspect their code. Being able to inspect state, see docs for every language construct/keyword in line, see visualizations for built-in collections. That would be tremendously useful. Maybe we can keep expanding the horizon of programs that can be worked with in this way, as hardware gets more powerful or as algorithms/languages get better.

                                                                                                                                          Or even think - even a non-scalable toolset that helps you write tricky string-validation functions or regexes or SQL queries - these would be tremendously useful. We already have great tools for inspecting regexes, ofc.

                                                                                                                                          I also liked the rapid-fire list of projects to look into. Plenty of jumping-off points for exploring this space.

                                                                                                                                          The tone and presentation style may not be great, totally agree. But that’s a presentation layer thing.

                                                                                                                                          1. 5

                                                                                                                                            In doesn’t have to be that bad. Common Lisp ticks many points: there are millions-of-lines codebases (or have been) (thinking about Google’s ITA Software and SISCOG, both are at least above 1 million lines). You deliver binaries smaller than your image, and commercial implementations (LispWorks, Allegro(?)) can strip out dead code.

                                                                                                                                            1. 2

                                                                                                                                              In doesn’t have to be that bad.

                                                                                                                                              I’m fairly certain that in some cases it does, otherwise you’re claiming that every programmer in every niche is misguided for doing things the way they’re doing it instead of your way.

                                                                                                                                              Common Lisp ticks many points: there are millions-of-lines codebases (or have been) (thinking about Google’s ITA Software and SISCOG

                                                                                                                                              Millions-of-lines codebases written in Common Lisp existing does not imply that the people developing these codebases use notebooks, interactive or visual programming. If they do, the orator should have demonstrated the gains they experience by using interactive programming on millions-of-lines pieces of software. This would have made his point stronger, as it would show that interactive programming provides gains across the entirety of the codebase-size spectrum.

                                                                                                                                              I’ll add that the “millions-of-lines codebases” was just one of the dimensions I had in mind. There are other niches like FPGA-programming, real-time programming, high-throughtput programming and many others that would also need to be studied before claiming that there is One True Way of programming that is better for all programming niches.

                                                                                                                                              You deliver binaries smaller than your image, and commercial implementations (LispWorks, Allegro(?)) can strip out dead code.

                                                                                                                                              This is already something all the big pieces of software do. It’s trivial to do with GCC (use -ffunction-sections, and then pass --gc-sections to ld). The reason the binaries are still huge is because compile-time monomorphisation is used very heavily. You might argue that you don’t need to perform monomoprhisation at compile-time and delay it to run-time instead, but then you’re going to lose start-up performance, which is pretty much my point: everything is a trade-off, there is no universal answer and all you can do is work within constraints.

                                                                                                                                              1. 4

                                                                                                                                                Having worked at ITA: the Common Lisp codebase was written using normal text editors, with code checked into version control. Emacs was quite common, and there was some sort of live prompt thing where you could push code updates during development for interactive testing… SLIME maybe?

                                                                                                                                                Last I saw (been many years at this point) it was deployed on SBCL.

                                                                                                                                                1. 1

                                                                                                                                                  Very interesting, thanks for sharing your experience!

                                                                                                                                                  and there was some sort of live prompt thing where you could push code updates during development for interactive testing

                                                                                                                                                  Was the instance of the program running on developer’s machines, or was it running on servers? Was this instance shared among developers? How often would this instance be restarted (if at all?)? Do you know of any video that would demonstrate this workflow, if not on ITA’s codebase, on something similarly large?

                                                                                                                                                  I’ve never done much REPL development (aside from debugging JavaScript in the developer’s console) but have always been fascinated by the idea, I’d love to learn more :)

                                                                                                                                                  1. 2

                                                                                                                                                    This was just a local instance, for debugging/live-testing mostly, is my impression. Production deployment was all the standard “compile from source code, deploy to N machines”.

                                                                                                                                                    It’s possible they used REPL for semi-live-development, but it all ended up in normal source code files in version control.

                                                                                                                                                    If you want a REPL-y dev experience, you can try out Jupyter notebooks, but keep in mind that people recommend restarting the notebook occasionally (or often!) so state doesn’t get messed up. Basically no one that I know of does pure REPL development for anything but one-off scripts, and for good reason.

                                                                                                                                                2. 2

                                                                                                                                                  I didn’t hear claims and I am not making any about One True Way. 2c.

                                                                                                                                                  1. 2

                                                                                                                                                    I didn’t hear claims about One True Way. 2c.

                                                                                                                                                    I think the video’s orator does claim there is One True Way when he claims that compiling ahead of time is bad and that you should really go with repl-programming instead.

                                                                                                                                                  2. 1

                                                                                                                                                    Millions-of-lines codebases written in Common Lisp existing does not imply that the people developing these codebases use notebooks, interactive or visual programming.

                                                                                                                                                    So, I’m kind of a CL newbie, but my understanding is that CL development is generally done semi-interactively. You write your code in a file, which will be under version control, and so on. But instead of stop-the-world, restart-with-changes, you’re normally working in a live image. As you update a function in your file, you run a command in your editor (usually Emacs) to evaluate the updated function, which replaces the existing function definition, and the same for CLOS class definitions - existing objects are updated to the new class definition. You can examine any aspect of the image’s internal state while it’s running.

                                                                                                                                                    The difference in CL images and Smalltalk images, as I understand it, is that in CL, your code in your editor can be out of sync with what’s loaded in the image (either because you haven’t evaluated the changes yet, or you’ve reverted to an older version and not evaluated that), whereas in Smalltalk, it can’t; the image is the only source of truth. Because I’m a n00b, I sometimes have to stop my Lisp process and restart it, loading my code from source, in order to be sure the image is in a state I understand.

                                                                                                                                                    1. 2

                                                                                                                                                      Sounds 100% right to me

                                                                                                                                                      Because I’m a n00b, I sometimes have to stop my Lisp process and restart it, loading my code from source, in order to be sure the image is in a state I understand.

                                                                                                                                                      People with lot of experience do that too (but not me ofc) , that’s why there’s a command named restart-inferior-lisp

                                                                                                                                                3. 1

                                                                                                                                                  True, and there are plenty of examples that could have been used instead. Fetch the Wayback Machine, Sherman!

                                                                                                                                                  Sabre C, renamed CodeCenter, was a popular developer tools for about $4K/seat. It included:

                                                                                                                                                  • Live debugging. You would be in the middle of loop and want to change the logic. When you typed, the interpreter started, unloaded the currently running object code into the interpreter, let you make the change, and continue stepping though the code.
                                                                                                                                                  • Set up displays (arrows and values) of values, all pointers in user space pointing to them, and walk through pointer chains by pointing and clicking.
                                                                                                                                                  • Jump to locations where the last pointer to memory was lost and to where it was allocated.
                                                                                                                                                  • Do all the usual ‘jump to declaration’ and ‘auto-reformat’ that is common now.

                                                                                                                                                  It was used on MLOC systems all the time.

                                                                                                                                                  1. 1

                                                                                                                                                    The only reference to Saber-C I could find is here and references a last release date of 1999. I did find a video of something named Saber-Plus but it doesn’t show the live-editing capabilities you’re mentionning. Perhaps live-editing of a running program was only practical when building a compiler whose performance is considered “adequate” was still doable for a single small company? I’d imagine building an interpreter for C++’s most recent revisions, able to deal with the intricacies of the object files produces by GCC, Clang or MSVC, that conserves the performances and behavior of the compiler being emulated might be out of reach for lone actors today.

                                                                                                                                                    1. 1

                                                                                                                                                      Sorry, I didn’t put a date on it. It was very popular in 1990. The company imploded around 1995, partly from competitive pressures from Sun and HP, partly from Sun and HP changing object formats quickly to cause quality issues. Between COFF and the formal specification of ELF, changes occurred which killed the ability to swap in the interpreter, sometime in 1993 IIRC. Added to that, C++ was evolving rapidly from a C-front preprocesser to its own language, and its spec changed constantly.

                                                                                                                                                      Oddly, it should not be out of reach. Charging $4K/seat for development tools is out of reach.

                                                                                                                                                1. 5

                                                                                                                                                  I rather enjoyed this. Although it’s worth noting that (from what I recall) he’s in the Games industry, so many of these bits (eg: latency) might be less important for other fields.

                                                                                                                                                  1. 8

                                                                                                                                                    I believe he mentions this in another talk at cppconf years ago…“People like you are the reason that it takes 30 seconds to open Word”.

                                                                                                                                                    (You in this case being the fellow he was addressing, not like you you.)

                                                                                                                                                    People ignore the learnings of the game industry at their peril.

                                                                                                                                                    1. 11

                                                                                                                                                      The flip side of this is a comment entitled “People like you are the reason we never ship”.

                                                                                                                                                      Knowing when and what to optimize, and what not to bother with, is also incredibly important. Even in games, they don’t optimize everything down to the absolute fastest screaming bare-metal performance, because they know which parts of the final product need that and which parts don’t. Trying to maximally optimize everything all the time is a recipe for wasting huge amounts of resources and never being able to actually ship the product.

                                                                                                                                                      1. 6

                                                                                                                                                        This is … specifically exactly what the OP is talking about. Please read the article.

                                                                                                                                                        1. 6

                                                                                                                                                          The top comment mentioned that some things might be less important outside game dev. Then someone replied with the “people like you are the reason it takes 30 seconds to open Word” and “people ignore the learnings of the game industry at their peril”. That’s not a constructive reply, and I was pointing out why it was not a constructive reply.

                                                                                                                                                          Please read the comment chain you’re replying in.

                                                                                                                                                          1. 6

                                                                                                                                                            It’s a harsh reply, but it is constructive. Perhaps not in isolation, but this comment thread gave enough context:

                                                                                                                                                            • @friendlysock mentioned learning from the games industry. The game industry does ship.
                                                                                                                                                            • Back in 2014, Mike Acton followed up with “We worry about our resources, we have to get shit out, we have to build this stuff on time.” (Click on the link given by @vamolessa)

                                                                                                                                                            People like these aren’t the reason why we never ship.


                                                                                                                                                            Now as a matter of fact, some very popular programs do take forever to boot. Photoshop was timed at about 7 seconds by Jonathan Blow around 2017. About as long as it took 20 years prior on the much slower computers of the 20th century. Even though Photoshop does the same thing in both cases (display the image it was asked to open). You could argue that 2017-Photoshop has so many more features than 1997-Photoshop, but none of those feature matter when you’re just displaying the image.

                                                                                                                                                            Oh and Photoshop’s menu: click on it, wait a full second. Click on it again… wait again. We could have forgiven them if it was fast the second time around (suggesting a cache mechanism or similar), but no, displaying a menu with less than 10 items is just that slow. And again, not justify by any expanded feature set. At that point we didn’t even get to use any of Photoshop’s features, beyond just displaying the image and opening up a menu.

                                                                                                                                                            Long story short, Photoshop got much slower than it used to be in the span of 20 years, with no good reason. That is the hallmark of the people who did not even think about performance. Photoshop being a GUI program, there are performance constraints to worry about:

                                                                                                                                                            • Startup times that exceed 200ms are noticeable. Above 1s the user is starting to wait.
                                                                                                                                                            • Opening up a menu feels sluggish when it exceeds 100ms.
                                                                                                                                                            • Any animation (even a drag animation) needs at least 30FPS to feel smooth enough. 60FPS if we can. Even 120FPS makes a difference in monitors that support it.

                                                                                                                                                            Those performance goals may be easy to attain, but they do represent limits any boring GUI program dev should be mindful of.

                                                                                                                                                            1. 5

                                                                                                                                                              It may be harsh but it’s not constructive. In fact, it borders on being content-free. It’s effectively the same sort of grousing about “kids these days” that we see in so many other areas – back in my day, we cared about a job well done performance! We respected our elders cache coherency! And so on.

                                                                                                                                                              The reality is that every generation of programmers is, for lack of a better term, shit on by their predecessors for slowing down all the software and shits on their successors for slowing down all the software.

                                                                                                                                                              Modern software is slow to start up? Well, the “show a screenshot of your last state to hide that you’re not ready for user interaction yet” trick is popular nowadays, and is even built in to some operating systems now, but it was originally invented back in the good old days when people allegedly cared about performance! Why didn’t they just write faster software so they wouldn’t need those tricks? And if they didn’t and got away with it, where’s the justification for suddenly declaring it unacceptable now?

                                                                                                                                                              And on the topic of games specifically, the industry is infamous for having trouble shipping (though not because of ruthless focus on software performance). Games are known for everything from massive “crunch” time burning out developers to needing multi-gigabyte patches downloaded on their release day just to be somewhat playable; this is not the sign of a disciplined industry in which quality and rigor are infused at every level. So if I’m going to get any “learnings”, it’s not going to be from game dev as a positive model to emulate.

                                                                                                                                                              So when we get right down to it, dismissive one-liners about performance are great for getting drive-by upvotes and terrible for understanding the complex, grimy reality of both the world and actual software. Which is why I replied as I initially did, pointing out that “performance” is a more complicated thing than the one-liner pretends it is.

                                                                                                                                                              1. 3

                                                                                                                                                                In isolation, I agree that “get off my lawn” is not constructive. But you make another point that I feel must be addressed:

                                                                                                                                                                The reality is that every generation of programmers is, for lack of a better term, shit on by their predecessors for slowing down all the software and shits on their successors for slowing down all the software.

                                                                                                                                                                You’re right, they absolutely do. As do I. Thing is, software is slowing down. Sometimes for good reasons (prettier renderings, advanced searches…), and sometimes for no reason at all.

                                                                                                                                                                I gave the Photoshop example, but I have another: my phone slowed down over the years to near unusable levels, yet I installed nothing since I first set it up, and I didn’t used or noticed any new functionality or eye candy… all while making sure the disk has at least 1.5GB of spare memory. My phone would be more valuable today if instead of pushing their crappy updates they stopped at fixing the security vulnerabilities.


                                                                                                                                                                “performance” is a more complicated thing than the one-liner pretends it is.

                                                                                                                                                                I’m not sure the one liner pretends such a thing. Let’s look back at what Mike Acton was actually responding to:

                                                                                                                                                                As I was listening to your talk, I realise that you are working in a different kind of environment than I work in. You are working in a very time constrained, hardware constrained environment.
                                                                                                                                                                [elided rant]
                                                                                                                                                                Our company’s primary constraint are engineering resources. We don’t worry so much about the time, because it’s all user interface stuff. We worry about how long it takes a programmer to develop a piece of code in a given length of time, so we don’t care about that stuff. If we can write a map in one sentence and get the value out, we don’t care really how long it takes, so long —

                                                                                                                                                                Then Mike Acton interrupted with

                                                                                                                                                                Okay, great, you don’t care how long it takes. Great. But people who don’t care how long it takes is also the reason why I have to wait 30 seconds for Word to boot for instance.

                                                                                                                                                                Most people who will say to you that performance is not that important in such and such settings, to avoid thinking of performance. They don’t measure, plan anything, or even eyeball anything. Not how much data they’re supposed to process, not what their time constraints are, not what the capabilities of the hardware they’ll be shipping in are.

                                                                                                                                                                Who is oversimplifying performance here? The guy who says it’s important, or the guy who acts at work as if computers had infinite speed? The fact is, performance always matter to some extent:

                                                                                                                                                                • There always is an upper bound to acceptable latency.
                                                                                                                                                                • There always is a lower bound to acceptable throughput.
                                                                                                                                                                • There always will be consequences for exceeding those bounds.

                                                                                                                                                                The variables are how hard it is to respect those acceptable bounds, and the consequences of exceeding them. Sometimes being fast enough is very easy, and even if we’re not it’s no big deal. Sometimes the bounds are very tight and missing them kill people. Just pretending performance isn’t important to your niche doesn’t help you pinpoint where you are in that spectrum. GUI performance for instance is more important and harder to meet than most people give it credit for. Especially for popular programs, where waiting 5 seconds for something to boot means that a million users would lose a collective amount of 35 work weeks per boot.

                                                                                                                                                                1. 7

                                                                                                                                                                  “But there is some slow software today” is not really a counterargument here. Sure, there’s software that’s slow today. You know what? There was software that was slow twenty years ago when I started out as a programmer. There was software that was slow before that. “Software is slow” is not some sort of unique new unprecedented problem.

                                                                                                                                                                  Meanwhile: I bet Mike Acton doesn’t work exclusively in hand-rolled assembly. I bet he probably uses languages that are higher-level than that. I bet he probably uses tools that automate things in ways that aren’t the best possible way he could have come up with manually. And in so doing he’s trading off some performance for some programmer convenience. Can I then retort to Mike Acton that people like him are the reason some app he didn’t even work on is slow? Because when we reflect on this, even the context of the quote becomes fundamentally dishonest – we all know that he accepts some kind of performance-versus-developer-convenience tradeoffs somewhere. Maybe not the same ones accepted by the person he was trying to dunk on, but I guarantee there are some. But haggling over which tradeoffs are acceptable doesn’t let him claim the moral high ground, so he has to dunk on the idea of the tradeoff itself.

                                                                                                                                                                  So again: “People like you are the reason that it takes 30 seconds to open Word” is not a useful statement. It’s intended solely as a conversation-ending bludgeon to make the speaker look good and the interlocutor look wrong. There’s no nuance in it. There’s no acknowledgment of complexity or tradeoffs or real-world use cases. Ironically, in that sense it almost certainly violates some of his own “expectations” for “professionals”. And as I’ve explained, it’s inherently dishonest!

                                                                                                                                                                  In other words: there is nothing constructive about it. it was not good when first said, and it was still not good when repeated in this thread. It will not become good if you keep repeating it and trying to justify it on grounds that slow software exists, because slow software has always existed and there never was a magical age of Real Programmers® who Cared About Performance™.

                                                                                                                                                                  1. 3

                                                                                                                                                                    It’s intended solely as a conversation-ending bludgeon to make the speaker look good and the interlocutor look wrong. There’s no nuance in it.

                                                                                                                                                                    Correct.

                                                                                                                                                                    In other words: there is nothing constructive about it. it was not good when first said,

                                                                                                                                                                    It was useful the first time it was said.

                                                                                                                                                                    At that time, Mike Acton just spent over an hour explaining why performance matters, why it is not a niche concern, and how to achieve it effectively. And then someone comes in explaining that the speaker comes from a very special niche, whose lessons don’t apply elsewhere, implying that almost everyone could safely ignore the whole keynote. All that packaged in a long winded statement that wasn’t even a question.

                                                                                                                                                                    We might feel sorry for the poor fellow, but he was undermining the core point of the keynote with weak arguments. Mike Acton was right to end it right then and there.


                                                                                                                                                                    Funny thing is, when I first stumbled upon that (highly recommended by the way) keynote, I was working on a little GUI program that replayed radar data (planes around an airport kind of radar). We had a potentially big file (up to a Gigabyte) that contained the data, and the software is supposed to help us navigate it. Unfortunately loading the whole thing was slow, so instead we were displaying it at a given point, then we re-read and re-parse and re-display everything starting to that point.

                                                                                                                                                                    Except it wasn’t that slow, actually. The file format was simple (I wrote the parser myself), computers of the time already had enough RAM to retain the whole file, and some pre-processing would have tremendously sped up getting from one point to another, to the point where the user could probably grab a time slider and navigate at will, and see the position of each plane update at 60 FPS. Like video games, only easier (we didn’t have that much data to display).

                                                                                                                                                                    The reason we didn’t do that? We never measured anything that mattered. What few measurements we did involved Qt refreshing the whole screen in real time, on the main thread, slowing everything down. They concluded loading the file, even without Qt’s display, would be too slow. I tried to raise the issue, but their mind was set. They would accept sluggish navigation because they believed, without having actually measured it, that loading was too slow.

                                                                                                                                                                    So I watched that keynote while I was having performance problems at work on a GUI program. The keynote gives me clues and solutions to where I should look, what I should measure. Gave me some hope that, considering how fast computers were at the time (2015), I should indeed expect more performance than my colleagues seem to hope for. And then there’s this dude telling the speaker that his keynote doesn’t apply to my work?

                                                                                                                                                                    Let’s just say I don’t remember having much empathy for the poor dude.

                                                                                                                                                                    1. 5

                                                                                                                                                                      We might feel sorry for the poor fellow, but he was undermining the core point of the keynote with weak arguments. Mike Acton was right to end it right then and there.

                                                                                                                                                                      I dunno, all these things that are supposed to make me like and respect Mike Acton are having the opposite effect. For his sake I hope it’s just that you’re doing a terrible job of making the case for him.

                                                                                                                                                                      Your own quote says the person was pointing out that game dev is a field that’s more time- and hardware-constrained than other areas of programming. Which is true! The tradeoffs that make sense for other fields of programming aren’t guaranteed to make sense in game dev. But the requirements that game devs operate under – or, let’s be brutally honest, the requirements game devs like to publicly claim they operate under – aren’t guaranteed to make sense for other fields of programming.

                                                                                                                                                                      Continuing the theme of brutal honesty: this is something I see way too much of and have, at this stage in my career, zero interest in putting up with, namely: people who generalize from their case to all cases. So the requirements they have are the requirements everyone must have. The background and skills they need are the background and skills everybody needs. Tradeoffs that are unacceptable for them are unacceptable for everyone. Things they have to know and worry about are things everybody has to know and worry about.

                                                                                                                                                                      Except… no. No, actually it’s the case that different fields of programming are different. A tradeoff that doesn’t make sense for him may well make a lot of sense for me. A skill or technique that’s indispensable to him may be utterly useless to me. And my entire beef with him, and people like him, is the inability to recognize and accept that. If he showed up to a technical session with one of my teams demanding to know things like the exact memory layout of all our data, he wouldn’t be laughed out of the room (since my teams don’t roll that way), but he might be privately pulled aside to have some things explained to him.

                                                                                                                                                                      And that is why this allegedly amazing dunk about performance, coupled with his “expectations” for “professionals”, don’t impress me at all. They just make him seem full of himself and ignorant of much of anything beyond his own particular niche.

                                                                                                                                                                      1. 4

                                                                                                                                                                        I dunno, all these things that are supposed to make me like and respect Mike Acton

                                                                                                                                                                        I was defending this particular statement more than Mike Acton himself. I could try and defend his keynote as a whole, but… just watch it.

                                                                                                                                                                        people who generalize from their case to all cases.

                                                                                                                                                                        I see why you’d see that pattern here. Still, performance is not a niche concern. Most of us care about performance at one point or another, and I saw this throughout my career:

                                                                                                                                                                        • First big job, I’m working on some GIS software written in C++. The thing is slow, the code has an inheritance hierarchy 9 level deep, and some of the processing I wrote myself had noticeable delays (it was simple image processing.
                                                                                                                                                                        • Second job was the radar thing I mentioned in the grandparent comment.
                                                                                                                                                                        • Fourth job was a real time testing environment. Constraints weren’t tight, but I still needed to not to pessimize everything.
                                                                                                                                                                        • The performance of the crypto library I wrote is an important selling point.
                                                                                                                                                                        • Sixth job was an ADAS system for a car. Embedded and all, though the embedded computer was still pretty powerful. Still, map matching (from GPS coordinates to a point on a given road) was a bottleneck.
                                                                                                                                                                        • Seventh job was a Qt GUI program displaying some data from a monitoring box. I quickly noticed that some ways of doing stuff would render the program sluggish if I didn’t pay attention, and I hardly had any data.
                                                                                                                                                                        • Overall, half of my gigs compiled slowly enough that it was annoying.

                                                                                                                                                                        Now a front-end web dev might not care that much about the CPU cache (that with the browser being such tall tower of abstractions), but they certainly would care about response time or network latency, or bandwidth. They need to (and do!) measure that stuff to get a decently performing website out of the door. I would guess most of the details of Mike Acton’s talk wouldn’t apply to them, but the philosophy of it (pay attention to the hardware you’re working with) apply even when your bottleneck is CISCO router you don’t even control.

                                                                                                                                                                        his “expectations” for “professionals”, don’t impress me at all.

                                                                                                                                                                        That list is very long, and some of the items are definitely more important than others depending on one’s working environment. Still, there aren’t that many criteria in there I would be comfortable not meeting.

                                                                                                                                                                        1. 1

                                                                                                                                                                          Have you watched his talks, out of curiosity?

                                                                                                                                                                          What sort of software do you write? Django/Python, right?

                                                                                                                                                                          1. 2

                                                                                                                                                                            If you’re going to do “until you’ve watched all his talks and read all his material you can’t criticize it”, I’m going to require the same standard of you: that you read everything I’ve published/watch every talk I’ve given over the course of my career before disagreeing with me.

                                                                                                                                                                            Meanwhile, I’ve explained at length why I think it was a bad thing to say here and why I think the the people defending it here are wrong. If you have nothing more to contribute to that, I’ll move on.

                                                                                                                                                                            1. 1

                                                                                                                                                                              I don’t know if you actually understand and can articulate his position and claims (and judging by your posts, I suspect you haven’t), which changes how to discuss this with you.

                                                                                                                                                                              (You don’t have to agree with them, by any stretch! I just would find it helpful if I know that you have familiarity with them.)

                                                                                                                                                                              I also have no idea if you have any relevant experience with programming in this style–if you’ve done work where performance as perceived by the end user has never been a problem or worked in languages where you simply can’t do a lot of optimization or benchmarking, that’s a different discussion to have. I’m trying to avoid talking past each other.

                                                                                                                                                                              If you want to boil this down to “mean bully programmers chest-thumping about MAH REAL PROGRAMMING AND MUH PERFORMANCE and that’s not something that matters in civilized society, the goons”, there’s not much we can talk about.

                                                                                                                                                                              1. 1

                                                                                                                                                                                I’ve explained my position at length. If you want to boil it down to whatever this line in your comment is:

                                                                                                                                                                                mean bully programmers chest-thumping about MAH REAL PROGRAMMING AND MUH PERFORMANCE and that’s not something that matters in civilized society, the goons

                                                                                                                                                                                then I agree there’s not much we can talk about.

                                                                                                                                                                    2. 2

                                                                                                                                                                      Who’s oversimplifying? I’d say the guy who cares more about performance than the paying customers do. The guy who says he’d still prioritize it in a business development position.

                                                                                                                                                                      Maybe Word takes 30 seconds to open: the customers don’t seem to care. Sorry that bothers him.

                                                                                                                                                                      The reason a game developer might care a lot about performance is, unsurprisingly, because that’s something their paying customers care a lot about.

                                                                                                                                                                      It’s still a pretty good list.

                                                                                                                                                                      1. 2

                                                                                                                                                                        Who’s oversimplifying? I’d say the guy who cares more about performance than the paying customers do.

                                                                                                                                                                        That guy is certainly not Mike Acton. The guy ships games that must sell at some point.

                                                                                                                                                                        Maybe Word takes 30 seconds to open: the customers don’t seem to care.

                                                                                                                                                                        That’s part of it. A bigger part is that customers, even more so than programmers, have no clue what performance they ought to expect. The biggest part though, I think, is just that customers often have no choice. To little competition, if at all depending on your niche.

                                                                                                                                                                        Except for games. Lots of games out there, lots of competition. Sluggish games go out of business more often than adequately performing ones. Also, there are lots of examples of fast games, so gamers actually know what they can demand.

                                                                                                                                                                    3. 1

                                                                                                                                                                      And if they didn’t and got away with it, where’s the justification for suddenly declaring it unacceptable now?

                                                                                                                                                                      Maybe the justification is the fact that computers are faster now, so all of the excuses that software used to have for being slow (at most tasks, at least) are now gone. This commenter certainly thinks so.

                                                                                                                                                                      To be honest, the grousing about the performance of software these days has taken its toll on my morale, and possibly productivity, this year. The main product that I’ve developed this year is based on Electron. Need I say more? We all know how much Electron is hated by message board commenters. And yes, the startup time of my specific Electron app is noticeably sluggish. I reluctantly chose it because I knew that using it would increase my chances of shipping in a timely manner. But then, the feeling that I was inflicting another piece of crappy software on the world may have hurt my productivity, especially before the public beta when we started getting responses from real users. As far as I can recall, just one user complained about the product making their older computer hot (and even that user went on to be happy with the product in general), and nobody specifically griped about it using Electron. And yet, maybe the message board commenters are right; maybe my users just don’t know what they should expect, and I really am inflicting more crap software on people. Maybe the world’s right to be free of crap software trumps my need to ship a product (and by extension, to make a living as a developer of such a product). The one thing that reassures me that I’ve done the right thing is that this software solves a problem that, as far as I and my cofounder know, no other software has solved. So I can reassure myself that it’s not just about making money.

                                                                                                                                                                      1. 3

                                                                                                                                                                        Maybe the world’s right to be free of crap software trumps my need to ship a product (and by extension, to make a living as a developer of such a product).

                                                                                                                                                                        On the other hand, maybe your client’s need to have software is more important than your need to give them fast software. If you don’t ship in a timely manner, then your users don’t have any software that solves their needs.

                                                                                                                                                                  2. 4

                                                                                                                                                                    “people ignore the learnings of the game industry at their peril”

                                                                                                                                                                    The reason I mentioned this is that the games industry has learned a lot that the rest of tech should probably be paying attention to:

                                                                                                                                                                    1. Games are written to be interacted with, for multiple hours, by people who are solely focused on the game. That means that any UX issues are going to be picked up on, any crashes are going to be experienced, any issues will be felt by the consumer. This software doesn’t respond easily to “just refresh the page”.
                                                                                                                                                                    2. Games have to run on a range of genuinely different platforms and operating systems. “We just need a polyfill” don’t cut it when you’re targeting x86, Xenon, and Cell.
                                                                                                                                                                    3. Games have hard scheduling deadlines because of marketing (e.g., “ship by Christmas or don’t come back here”).
                                                                                                                                                                    4. Games have done a lot of exploration around how reusable code can be, with engines like Unreal and Unity on the extreme end. They’ve been doing this for decades (going back to, say, old id licensing).
                                                                                                                                                                    5. Games in a lot of ways pioneered distributed and remote development, going back to both mod teams and then indie teams with contractors.
                                                                                                                                                                    6. Game code in a lot of ways contains both the most pessimistic workloads for computers as well as complicated near-ideal numerical workloads. Programming and tuning in this regime is a whole thing.
                                                                                                                                                                    7. AAA games (and smaller studios as well) have a really clear demarcation of programmers/content folks/tooling folks that can teach us a lot about how to manage large multidisciplinary projects and empower teams.
                                                                                                                                                                    8. Some of the first widely-deployed VMs were out of the games industry: Quake, Scumm, Z-machine.
                                                                                                                                                                    9. Some things like React’s rendering model were basically already old-hat by near a decade in games.

                                                                                                                                                                    I didn’t mean that as a throwaway comment: there is a wealth of untapped knowledge and experience from that industry and its veterans, and I honestly cringe at the failure to learn from its suffering we see elsewhere in tech.

                                                                                                                                                                    I’m sure there’s stuff the games industry can learn as well–say, source control outside of the world of Perforce–and more generally people should just look around and see what other wisdom exists outside of their bubble.

                                                                                                                                                                    1. 6

                                                                                                                                                                      Games also usually ship and then are never meaningfully updated again. That makes them unlike most other software produced in industry. Maintenance matters.

                                                                                                                                                                      1. 1

                                                                                                                                                                        I believe this is flatly incorrect. You’ll see updates tail off usually after a few years, but there is very much a culture of patching.

                                                                                                                                                                        Look for changelogs for Halo, Counter-Strike, Minecraft, Natural Selection (going back like a decade), Tremulous…even porn games.

                                                                                                                                                                        This was perhaps true back with cartridge games, but patches and updates have been a semi-regular thing since the mid 00s.

                                                                                                                                                                        1. 2

                                                                                                                                                                          I knew someone was gonna do this, call out those superminority of games that established longevity, and try to claim it as a refutation of my point. I knew it.

                                                                                                                                                                          Among all games produced, what percentage do you think are updated on any kind of regular cadence? 1%? 10%? 50%? Among those games, how do you think their release cadence compares to typical software in industry? Just as fast? Half as fast? Twice as fast?

                                                                                                                                                                          I don’t have data on the precise answers to those questions, but they’re in the neighborhood of 1% and 100-1000x less often, respectively. I can’t remember the last org I’ve worked for which didn’t deploy new versions multiple times per day. The domains are fundamentally incomparable.

                                                                                                                                                                          1. 1

                                                                                                                                                                            We also only play 0.1% of all games or something like that. There are so many games out there that just fall flat with nobody playing them. Or so few people. So we need a way to count:

                                                                                                                                                                            • Do we give the same weight to each game?
                                                                                                                                                                            • Do we count by how much effort it took to write?
                                                                                                                                                                            • Do we count by number of players?

                                                                                                                                                                            My point is, there is a huge difference between “games” and “relevant games” (for various definitions of “relevant”). And there’s a good chance we should only care about relevant games. Meaning games we actually play. There’ll be a bigger proportion of long standing games in that restricted space. Or we could go by studio, and only pick those studios who survived enough years, or did a certain number of game before going out of business. The effect of this selection bias should be similar.

                                                                                                                                                                            The whole point of this sub-thread is to learn the lessons of the video games industry. Of course this is not about learning the lessons of crappy (or unlucky) studios that went out of business before they got to release their first game. You want to learn from the cream of the top, the successful ones. Whatever they are doing is more likely to work.

                                                                                                                                                                            I can’t remember the last org I’ve worked for which didn’t deploy new versions multiple times per day.

                                                                                                                                                                            I can’t remember the last org I’ve worked for which deployed new versions faster than twice a month. I don’t work in games, but I did touch various domains. Just not the web.

                                                                                                                                                                            1. 3

                                                                                                                                                                              My read on “the lessons of the video game industry” hyped by the guy in the OP and reinforced by posters in this thread is that they can broadly be described as performance optimizations.

                                                                                                                                                                              I think you can divide the set of all possible performance optimizations into two broad categories, let’s call them “not being stupid” and “being smart” maybe.

                                                                                                                                                                              Not being stupid means if you need to get 1000 records from an API, you make 1 batch request, and not 1000 individual requests. Or maybe using assembly optimized functions in your language’s standard library to find substrings in a string, rather than walking each character manually. Every program benefits from not being stupid. It brings substantial benefits, and generally has few to no costs, in terms of coherence or maintenance or whatever. No matter how often your code churns, or how often you deploy, not being stupid is probably going to pay off.

                                                                                                                                                                              I don’t think “the lessons of the video game industry” refer to this class of performance optimization. Every time I read about cool performance stuff in video games, it’s stuff like the inverse square root function in Quake. This is a “being smart” optimization. This stuff can be super powerful, and can bring huge benefits. But! It also almost always carries enormous costs, too. It is definitionally harder to understand, and therefore maintain, versus the “not smart” alternative. Those costs are subtle, long-lived, difficult to measure, and tend to create pathological downstream consequences which are practically impossible to unwind. Multiply those costs over an entire code base, a team of engineers, the release cadence required to satisfy business needs, over however long the software is expected to remain in service to deliver value, and the ROI gets negative really quickly.

                                                                                                                                                                              These “being smart” optimizations are in direct tension with many other critical properties of software engineering in the large. And long-lived, highly-played, frequently-patched games suffer from that tension, quite visibly! I play PUBG, which is actually quite ancient and not particularly performant, but nevertheless gets patches every month or so. That’s glacial compared to most industries. And, even at that glacial pace, when they release a patch — which is usually 10–50GB in size (!) — they literally shut down the entire infrastructure for the game, every server, in the entire world, for 8–12 hours. You can’t play the game at all during an upgrade. That’s the price you pay if you want to try to straddle the boundary we’re discussing.

                                                                                                                                                                              Anyway. Just some thoughts.

                                                                                                                                                                              1. 1

                                                                                                                                                                                I think you can divide the set of all possible performance optimizations into two broad categories, let’s call them “not being stupid” and “being smart” maybe.

                                                                                                                                                                                That’s a good first order approximation. Casey Muratory did a similar classification using different names (not being stupid is called “Non Pessimization”, being smart is called “Optimisation”). He has a third category called “fake optimisation” but we can ignore that here.

                                                                                                                                                                                I don’t think “the lessons of the video game industry” refer to this class of performance optimization.

                                                                                                                                                                                It really does.

                                                                                                                                                                                If you took some time to watch Mike Acton’s talk, you would have noted that he’s also about not being stupid. One of the most important things he mentions is the memory hierarchy, and its enormous effect on performance. Among other things he notes that having a boolean in a structure is generally pretty stupid, because you often end up loading an entire cache line just to get one single bit of information. Ideally you’d want to “not be stupid” about memory loads, and make sure that when you load a cache line, it’s packed full of useful information your comparatively blazing fast CPU can process right away.

                                                                                                                                                                                Many people would put the “pay attention to your memory layout” to the being clever category, but personally I would disagree with that assessment. Not knowing the constraints of your environment is not an excuse for engineers of any kind, and the hardware we use is a pretty important set of constraints.

                                                                                                                                                                                Guys like Casey Muratori and Mike Acton actually fairly rarely get to be smart. They just do the “not being stupid” part until it’s not fast enough. When that happen they measure stuff, find the cause, optimise it… the standard “measure before you optimise”.

                                                                                                                                                                                Every time I read about cool performance stuff in video games, it’s stuff like the inverse square root function in Quake.

                                                                                                                                                                                That’s an obvious selection bias, where the most impressive stuff gets pushed to the front, until you get the impression that it’s all there is. John Carmack doesn’t do that kind of stuff very often. First because he tend to optimise at the system level, but also because he also chose the simple stuff over the ultra-efficient stuff when it was good enough. (Source: Jonathan Blow recalling some criticism he voiced over level loading or something, don’t remember which video. He noted that his criticism was wrong, for the exact reasons you cited: the complexity of the smart approach has its own costs.)


                                                                                                                                                                                Those PUBG updates are a bummer. However, recall what @ friendlysock said here:

                                                                                                                                                                                My point was there is a lot to learn from the games industry. That doesn’t mean blindly aping practices or tech, that doesn’t mean they’ve discovered the One True Way of doing anything–in fact, a lot of their practices like crunch are abhorrent.

                                                                                                                                                                                But…the industry is a pressure cooker for software engineering and has created a ton of data, both examples and counterexamples, about what seems to work and what doesn’t. There is also for many studios a culture of public post-mortems, sharing what they’ve learned about during their development.

                                                                                                                                                                                Of course it’s not all rainbows and unicorns.

                                                                                                                                                                                1. 3

                                                                                                                                                                                  I don’t think “the lessons of the video game industry” refer to this class of performance optimization.

                                                                                                                                                                                  It really does.

                                                                                                                                                                                  When you define some category set (software industries) and then isolate one element of that set (the video game industry) the point is to highlight the unique properties of that element. But “not being stupid” optimizations are — nearly by definition — equally useful in all software domains, and you can find them, in equal measure, everywhere. The unique thing about games in the context of performance is precisely the clever stuff, the stuff that’s architecture-specific, the stuff that exploits all of the exploitable stuff, and deals with the impact of the costs.

                                                                                                                                                                                  At least, this was my perspective when I responded. YMMV.

                                                                                                                                                                                  One of the most important things he mentions is the memory hierarchy, and its enormous effect on performance. Among other things he notes that having a boolean in a structure is generally pretty stupid, because you often end up loading an entire cache line just to get one single bit of information. Ideally you’d want to “not be stupid” about memory loads, and make sure that when you load a cache line, it’s packed full of useful information your comparatively blazing fast CPU can process right away.

                                                                                                                                                                                  There is a performance threshold beyond which the efficiency of cache lines is meaningful. And I agree that there is a large class of software for which this is an important detail to be aware of. But that class of software is still a tiny minority of all software produced, and it would be a disaster to incept the notion that a performance pathology, under certain architectures, in certain narrow circumstances, is sufficient motivation to avoid using a primitive type in a primitive language construct in general.

                                                                                                                                                                                  If I’m hired to work on some line-of-business software written as a web service with a p99 latency SLO of 10ms, then, in general, the things that impact performance are several layers of abstraction above cache line performance.

                                                                                                                                                                                  If I see some code that models a truly boolean bit of state as a literal bit, packed into a uint8 with 5 other bits of unrelated state (or whatever)? Absent a benchmark or some sort of evidence that this optimization is important, that’s actively a problem. If I ever touch that code I’m refactoring it as a first step.

                                                                                                                                                                                  Many people would put the “pay attention to your memory layout” to the being clever category, but personally I would disagree with that assessment. Not knowing the constraints of your environment is not an excuse for engineers of any kind, and the hardware we use is a pretty important set of constraints.

                                                                                                                                                                                  Memory layout, allocations, bytes on the wire, CPU cycles spent encoding and decoding data types, binary size, infinitely many other metrics that impact performance — all of them carry benefits and incur costs, and all of them are, or should be, managed based on an engineering calculus that weighs both costs and benefits appropriately. There are plenty of circumstances where programmers should think about and optimize memory layout of their types, where the benefits outweigh the costs. There are also plenty of circumstances where the costs outweigh the benefits.

                                                                                                                                                                                  (Tons of code, maybe even most code, is written without any knowledge at all of the hardware on which it will run!)

                                                                                                                                                                                  1. 1

                                                                                                                                                                                    But “not being stupid” optimizations are — nearly by definition — equally useful in all software domains, and you can find them, in equal measure, everywhere.

                                                                                                                                                                                    Equally useful everywhere, yes. That’s why Mike Acton’s talk applies pretty much everywhere. I’m less sure you can find them in equal measure everywhere though. And I know for a fact that the lack of stupidity is not evenly distributed. Some people think Mike Acton’s talk doesn’t apply to them, and Photoshop still take too long to boot for instance.

                                                                                                                                                                                    There is a performance threshold beyond which the efficiency of cache lines is meaningful.

                                                                                                                                                                                    Note that this performance threshold is much lower than the performance threshold beyond which compiler optimisations are relevant. Another point of Acton’s talk was that 90% of the performance of your program comes from good memory access patterns. The compiler can only optimise the remaining 10%. Making this relevant for a relatively large class of software.

                                                                                                                                                                                    Heck I have a recent example from my own experience: see this code ? Nicely formatted with Pygments so I could apply some CSS pretty colours. That thing is so slow that it takes more than a second per page of code. Very annoying, I have to turn it off when doing quick iterations. Sure I could have set up an incremental build to begin with, but come on: the thing is little more than a parser, what could possibly justify it being so slow? But that illustrates my point: even this little too, used on relatively little code, has performance requirements.

                                                                                                                                                                                    Absent a benchmark or some sort of evidence that this optimization is important, that’s actively a problem.

                                                                                                                                                                                    Good thing Acton insisted that you should measure first.

                                                                                                                                                                                    Memory layout, allocations, bytes on the wire, CPU cycles spent encoding and decoding data types, binary size, infinitely many other metrics that impact performance — all of them carry benefits and incur costs, and all of them are, or should be, managed based on an engineering calculus that weighs both costs and benefits appropriately.

                                                                                                                                                                                    Yes. Acton pretty much said as much.

                                                                                                                                                                                    (Tons of code, maybe even most code, is written without any knowledge at all of the hardware on which it will run!)

                                                                                                                                                                                    The range of hardware we actually target remains finite. Ignoring that range doesn’t help much.

                                                                                                                                                                      2. 4

                                                                                                                                                                        Also, games are mostly bought by the same person who uses them. Entreprise software is often inflicted on users by higher-up deciders who don’t care about the actual experience of using it.

                                                                                                                                                                        1. 2

                                                                                                                                                                          There is a danger in point 1. Games are meant to be fun, not productive. An optimal game UI for efficiency would have a single button to start the game, it would then show the end of game sequence: congratulations, you have completed the game. Games are all about taking a long path to achieve something, productive UIs are the opposite of this.

                                                                                                                                                                          Given how many games use WINE for porting, I’m not sure I buy the second argument.

                                                                                                                                                                          Duke Nukem Forever? There are a lot of high-profile examples of games with multi-year delays. The E3 reporting every year has a load of articles about games that miss their deadlines.

                                                                                                                                                                          Unity and Unreal are much younger than a lot of the codebases that I’ve worked on in non-game contexts. Games did not invent reusable libraries and were some of the later adopters. The Unreal engine exists at all because it stared as a completely new implementation of an FPS engine rather than licensing an existing one.

                                                                                                                                                                          Distributed communities for open source development have been the norm since the era when a large team for game development was 6 people. Game companies have largely followed industry trends here, not led them.

                                                                                                                                                                          Visual Basic and various Pascal compilers were shipping interpreted and JIT’d VM code from the ‘80s. Most of the game VMs learned from these, not the other way around.

                                                                                                                                                                          1. 3

                                                                                                                                                                            Given how many games use WINE for porting, I’m not sure I buy the second argument.

                                                                                                                                                                            Today, most consoles are some variant of x64. This was emphatically not the case 10 years ago, and definitely not the case 20. MIPS, PowerPC, x86, Cell (which is PPC with helper friends), all kinds of weird shit. Even inside x86, the 386/486SX/486DX/Pentium issues–setting aside all the crazy EGA/CGA/VGA weirdness–it was a total jungle. Even today, the ARM/x64 split for mobile game ports vs desktop games is still a thing.

                                                                                                                                                                            Duke Nukem Forever?

                                                                                                                                                                            One of the outliers that proves the general rule that for most games a schedule slip is a Big Deal.

                                                                                                                                                                            The Unreal engine exists at all because it stared as a completely new implementation of an FPS engine rather than licensing an existing one.

                                                                                                                                                                            At the time the two competitors were Carmack’s Quake engine and Silverman’s Build engine. Many people did license those engines (and make great games!). Many people rolled their own (for example, Bungie and Looking Glass/Irrational). We learned a whole lot from the experiences of both camps.

                                                                                                                                                                            Also, Unreal dates back to 1995. There are plenty of old codebases, but that’s one that’s still showing it’s age. Quake code of a similar vintage is still kicking around.

                                                                                                                                                                            Game companies have largely followed industry trends here, not led them.

                                                                                                                                                                            Game companies have scaled all the way up to AAA studios, looking a lot like big software houses, to garage-band teams sharing a dorm room, to indie teams coordinating lots of developers on multiple continents both for mods and for paid work due to budget. The point is, there’s a whole lot of information out there about what worked and didn’t work.

                                                                                                                                                                            Most of the game VMs learned from these, not the other way around.

                                                                                                                                                                            I didn’t claim otherwise. My point is that for widely-deployed VMs, you can learn a lot from what worked and didn’t work over in games. Id engines, for example, switched from no VM (Doom/Wolf3d) to VM (Quake) to no VM (Quake 2) to hybrid (Quake 3). There is interesting knowledge to be had by seeing why that choice was made.


                                                                                                                                                                            I don’t know what everybody’s deal out here is, but this whole thread has just been remarkably disappointing to read. My point was there is a lot to learn from the games industry. That doesn’t mean blindly aping practices or tech, that doesn’t mean they’ve discovered the One True Way of doing anything–in fact, a lot of their practices like crunch are abhorrent.

                                                                                                                                                                            But…the industry is a pressure cooker for software engineering and has created a ton of data, both examples and counterexamples, about what seems to work and what doesn’t. There is also for many studios a culture of public post-mortems, sharing what they’ve learned about during their development.

                                                                                                                                                                            As per my original post, we ignore that experience at our peril.

                                                                                                                                                                            1. 2

                                                                                                                                                                              I don’t know what everybody’s deal out here is, but this whole thread has just been remarkably disappointing to read. My point was there is a lot to learn from the games industry. That doesn’t mean blindly aping practices or tech, that doesn’t mean they’ve discovered the One True Way of doing anything–in fact, a lot of their practices like crunch are abhorrent.

                                                                                                                                                                              I suspect that a lot of the negativity comes from the fact that you are making a load of claims about the games industry as if those things were unique to the games industry. Every single thing that you’ve listed is something that I’ve seen, at scale, in non-games software companies, often including things that they’ve been doing since the late ’80s, and including companies that range from one person and a few of his friends up to trillion-dollar enterprises. Your post comes across as saying that everyone should learn from you while simultaneously indicating that you are unwilling to learn from anyone else.

                                                                                                                                                                              1. 1

                                                                                                                                                                                Part of the cause for my disappointment is exactly due to your observation: at no point did I say we don’t have lots to learn from other parts of software–I just said that gamedev has a lot to offer and gave examples I figured would substantiate that claim for the skeptical.

                                                                                                                                                                                1. 2

                                                                                                                                                                                  The examples that you give are all things that the rest of the rest of the industry has done well for decades. By framing them as things that the game developer ecosystem can teach everyone else, you are implicitly framing your argument as if they were things he rest of the industry needs to learn. Worse, dome of your examples are things where the game industry was one of the later adopters. This comes across as dismissive of the experiences and skills of others.

                                                                                                                                                                            2. 1

                                                                                                                                                                              Games are all about taking a long path to achieve something, productive UIs are the opposite of this.

                                                                                                                                                                              Obvious counter-example: Factorio. It’s a logistics optimisation game where you construct an ever growing factory, and there is a lot of stuff to do and automate. A lot of thought has been put into making the user interface as productive as possible (within the constraints of the game world, which I admit are artificial), and from my personal experience it made a big difference.

                                                                                                                                                                      3. 1

                                                                                                                                                                        “People like you are the reason we never ship”.

                                                                                                                                                                        As pointed out elsewhere, the games industry has a great many failings but not shipping is very seldom one of them.

                                                                                                                                                                      4. 4

                                                                                                                                                                        That would be the great “Data-Oriented Design and C++” talk at CppCon14 https://youtu.be/rX0ItVEVjHc?t=4681 (it’s at the right time ;) )

                                                                                                                                                                        1. 1

                                                                                                                                                                          That is a great talk. One slide gave me pause, though; he says they don’t use the following parts of C++:

                                                                                                                                                                          • Exceptions
                                                                                                                                                                          • Templates
                                                                                                                                                                          • iostream
                                                                                                                                                                          • Multiple inheritance
                                                                                                                                                                          • Operator overloading
                                                                                                                                                                          • RTTI
                                                                                                                                                                          • STL

                                                                                                                                                                          At that point… why not just use C?

                                                                                                                                                                          1. 3

                                                                                                                                                                            He actually goes on to reply this very question that someone in the audience asks. Basically he would personally prefer C99, but the team uses C++ because of convenience. (then he would even point out that msvc supports c++ better than c)

                                                                                                                                                                            1. 1

                                                                                                                                                                              Oh, I stopped the video before the end of the audience questions as I wasn’t really finding them very enlightening… I should have stuck it out to the end!

                                                                                                                                                                            2. 1

                                                                                                                                                                              At that point… why not just use C?

                                                                                                                                                                              For a great many years, games’ industry C++ for many shops could be charitably described as “C with classes”.

                                                                                                                                                                              This makes a great deal of sense when you consider both the quality of vendor and Microsoft compilers at the time, as well as how poorly the common STL implementations of the era matched the requirements of game development.

                                                                                                                                                                              1. 2

                                                                                                                                                                                I’d agree with everything on that list except templates. The first thing I ever saw that convinced me I should look at templates seriously was the code for a game engine, which had a load of clean abstractions for vector and matrix arithmetic and (after inlining) compiled down to an incredibly efficient set of MMX operations (which should give you an idea of how long ago this was). A single tweak to a template parameter let them build MMX and non-MMX versions of the hot parts of the engine and select the version to use at load time via cpuid. The only way of getting that mix of performance and usability without templates is to have something else generate the C++ code from a higher level abstraction.

                                                                                                                                                                                For high-performance code today, C++ templates are my most useful tool for creating maintainable code that executes fast. Most recently, I wrote a C++ implementation of memcpy that outperforms hand-written assembly implementations on multiple architectures and yet defines the architecture-specific bits by tuning a few template parameters. Writing it with templates made it possible to do a parameter sweep of a bunch of heuristics and find the sweet spots very quickly. I could write the same code in the end without templates but it would be a lot more fragile.

                                                                                                                                                                                1. 1

                                                                                                                                                                                  I think his argument against templates was mostly about how much they slow down the build process (and possibly the increase in code size? I can’t remember now if he said that or not). In the past, I’ve used templates with explicit instantiation: at least then you are always made aware of when you’re adding another implementation to the code.

                                                                                                                                                                                  I have mixed feelings about templates but Eigen is great.

                                                                                                                                                                                  1. 1

                                                                                                                                                                                    Compile time is definitely an issue with templates, but often it’s a trade of compile time against run time and, in general, people run code more times than they build it[1] and so a 100% increase in compile times in exchange for a 1% speedup is a big win (for release builds, at least). The code size issue is definitely important because this can make it easy to write code that does well in microbenchmarks and very badly in macrobenchmarks. In general, my recommendation for templates is to always add explicit always- or never-inline attributes on every method in a templated class and on every templated function so that you think about where the cost is paid. If your method is not either small, or exposes opportunities to make it small after inlining, then templates are probably the wrong tool for the job and you should use dynamic dispatch.

                                                                                                                                                                                    The last point is much easier in Rust than C++, where you can switch between dynamic dispatch and compile-time reification with a single attribute.

                                                                                                                                                                                    [1] Okay, that’s not true for a bunch of my code, but it is for anything in production.

                                                                                                                                                                                2. 1

                                                                                                                                                                                  I can completely understand why they would eschew all those parts of C++, I just couldn’t understand why they would use C++ at all if they’re left with “C with classes” and - as he explained in that talk - he’s not a fan of OO-style classes.

                                                                                                                                                                                  However, it turns out (thanks @vamolessa) that Mike Acton would actually prefer to be using C, so the world makes sense again. Phew.

                                                                                                                                                                        1. 3

                                                                                                                                                                          My takeaway is that more than I realized the word “type” seems to be synonym for “magic” in a lot of heads.

                                                                                                                                                                          My expected takeaway was something about sheaf theory but okay seems like that kind of article will eventually arrive and this is more about telling us that fact.

                                                                                                                                                                          1. 2

                                                                                                                                                                            Valim’s post seems pretty straightforward and sensible to me.

                                                                                                                                                                            I think you have some context in your mind that is not in mine… because your response makes no sense to me, probably because I’m missing that context.

                                                                                                                                                                            Could you elaborate?

                                                                                                                                                                            1. 1

                                                                                                                                                                              The post is fine, just one “about” higher than I anticipated (so about the plan to make something “about” the title).

                                                                                                                                                                              My guess when I saw set-theoretic types for a language where every function runs in its own thread was that it would be about sheafs (~set valued functors) but sorry I can’t really elaborate because I never had a chance to go that far (although I would like to), my academic life ended broke on the street and now I am an ignored madman hoping against hope that people can be persuaded to change their priorities without a bloody catastrophe.

                                                                                                                                                                              Here’s some justification for my guess (i.e. the context you may be missing): https://www.researchgate.net/publication/222422736_Sheaves_Objects_and_Distributed_Systems

                                                                                                                                                                              1. 1

                                                                                                                                                                                I don’t think every function does run in it’s own thread.. (since it’s a pure language I guess in principle it could).

                                                                                                                                                                                Academia is a strange distorted universe.

                                                                                                                                                                                people can be persuaded to change their priorities without a bloody catastrophe.

                                                                                                                                                                                Oh dear, some mighty catastrophes are clearly incoming… the recent record on “people changing their priorities” is not good.

                                                                                                                                                                                Sigh. I always argue with myself for a pragmatic multiparadigm close to metal language like D, or mathematically correct like Haskell… but feel if I wanted to go that way I’d want Agda, or comfy old Ruby, or cleaner Elixir….

                                                                                                                                                                                But I think Valim and Matz are on the right track, much about types can be deduced from the code itself.

                                                                                                                                                                                1. 1

                                                                                                                                                                                  I think all languages have something to offer which is why I slowly got dragged into thinking about the tooling side and interop.

                                                                                                                                                                                  W.r.t. what we want from the runtime then this guy gets it: https://youtu.be/8Ab3ArE8W3s

                                                                                                                                                                                  1. 2

                                                                                                                                                                                    I got around to watching that…

                                                                                                                                                                                    Hah! The guy is pretty much the same era as me, I (mentally) put my hand up for all his “do you know what / have you used …” questions…

                                                                                                                                                                                    And yes, it frustrates the hell out of me that we have gone backwards on so many fronts.

                                                                                                                                                                                    Alas, I believe the ludicrous situation we’re in is a symptom of how the economy is designed.

                                                                                                                                                                                    eg. I used an excellent smalltalk system in the 1990’s that had a lot of what he is walking about…. it (and a bunch of other excellent products) were bought out…. and left to whither and die.

                                                                                                                                                                                    Parable of the Software Industry: The software industry is like when a miner, through incredible luck, found diamonds lying loose on the ground. He picked them up with just his hands and became the richest miner in the area, hiring hundreds of men to dig for diamonds with just their bare hands. So all the other miners threw away their picks and shovels and used their bare hands… Because that’s what the most successful people were doing.

                                                                                                                                                                                    1. 1

                                                                                                                                                                                      I think the place to do static analysis is the version control system… and as soon as you do risk assessments then you have premise for “block rewards” (or you know whatever, I’m saying you can have finality with X% risk)..

                                                                                                                                                                            2. 2

                                                                                                                                                                              I would enjoy more detail but I think they are really in a preliminar phase I guess.

                                                                                                                                                                              From set theoretic types I think they mean intersection and union types.

                                                                                                                                                                              I have an hypothesis about how an effect system would be like.

                                                                                                                                                                              I think the dynamic calls (apply/3) can be a challenge, because the type of the message must match a type of the actor, so we would need type PIDs. Since there are process registries, they would need to carry some information about type.

                                                                                                                                                                              1. 1

                                                                                                                                                                                “static types” is kind of short for predictable, well behaved static analysis that work on the whole language (or most of it, when you bolt it in after the fact). I don’t really think it’s “magic” :). I think the talk makes a good job of listing reasons the Elixir team wants them, namely tooling (like typescript), documentation, to allow designing type-first for those who like it, and as proofs in the restricted language afforded by whatever type system they eventually pick.

                                                                                                                                                                                I’m not following why you jump to sheaves from “set theoretic types” though. Clearly the simplest interpretation of that, is that types are seen as sets of Elixir values, and there would be intersection and union operators on types to handle the fact that Elixir allows to define multiple overloads of a function, and restrictions on a given clause. I don’t know much about sheaves but they seem to be a category theoretic abstraction of set-like things, which doesn’t bring much to the table when talking about a concrete use of sets.

                                                                                                                                                                                1. 1

                                                                                                                                                                                  It wasn’t just because they said “set-theory” it was because it is elixir and they talked about set-theory, but anyway sure, sorry for being rude.

                                                                                                                                                                              1. 5

                                                                                                                                                                                Disclaimer:

                                                                                                                                                                                • I like many things about StandardML.
                                                                                                                                                                                • I think Poly/ML is the best SML compiler.
                                                                                                                                                                                • The formalization of SML is incredibly cool, whatever one may think about its effect on its further evolution.
                                                                                                                                                                                • Everyone who worked on MLs in their early days and helped them become what they are now has my deepest respect and admiration.

                                                                                                                                                                                However, that article states many things about OCaml that are either wrong or not as simple, and @yawaramin only scratched the surface in his comment. I’ll omit the points he covered and focus on those he didn’t, mainly in the “not as simple” category.

                                                                                                                                                                                As I recall, Caml launched with basic modules that worked with UNIX Makefiles to generate .o files, which was regarded as a big win. OCaml modules today apparently resemble Standard ML’s, if this guy is correct.

                                                                                                                                                                                Objective Caml (which is now OCaml) has supported functors since at least 1998. I wanted to say “from its inception in 1996”, but I don’t have a source handy to prove it.

                                                                                                                                                                                Regarding Caml Light, its goal was to bring ML to affordable and widely available machines, which in 1990 meant absurdly low-powered ones. True separate compilation was indeed a big win at the time. Hell, it still is a big win for developer experience and one of the reasons I like OCaml so much — it’s not just fast by itself, but also never rebuilds anything that doesn’t need rebuilding.

                                                                                                                                                                                ZINC report, the document on Caml Light’s design and implementation, also mentions design issues with ML module syntax design at the time:

                                                                                                                                                                                Second, StandardML modules are not particularly convenient to use. The problem is that to use some values or types defined in another module… the programmer has to insert their signature … at the beginning of the current module.

                                                                                                                                                                                I’m not familiar with that part of ML history so I’m not sure what it looked like at the time, but I assume that SML modules and OCaml modules got easy to use around the same time in the mid ’90s.

                                                                                                                                                                                Weirdest of all, you could actually omit the brackets in (x,y,z), which may be the actual reason why list elements must be separated by semicolons.

                                                                                                                                                                                One advantage of semicolons as list item separators is that a list of tuples (which is a common way to write an immutable associative array) can be written without any parens: let kvs = ["foo", 1; "bar", 2]. Not requiring parens around tuples also makes multiple bindings look cleaner: let x, y = 1, 2.

                                                                                                                                                                                Astonishingly, implementors actually used it, achieving compatibility to such an extent that Isabelle could be compiled with either Standard ML of New Jersey or Poly/ML

                                                                                                                                                                                A fun thing is that Poly/ML requires a function named main in standalone programs while MLton and SML/NJ evaluate all expressions top-down, so any program one wants to be buildable with both needs to be aware of that. The definition of SML says nothing about the program entry point.

                                                                                                                                                                                support for multi-threading (also here); they say that OCaml is finally catching up, 15 years later, thanks to having 100 times as much funding

                                                                                                                                                                                OCaml has supported threads for at least 20 years by now. The big deal about OCaml 5.0 is a parallel GC that doesn’t require pauses and doesn’t worsen single-threaded program performance — the latter part, to my knowledge, is a big breakthrough in memory management for any language.

                                                                                                                                                                                Poly/ML uses the standard fare “stop-the-world” GC, as that very paper says:

                                                                                                                                                                                The garbage collector is single-threaded. When a garbage collection is required all ML threads are stopped and the collector runs to completion before releasing the ML threads.

                                                                                                                                                                                1. 1

                                                                                                                                                                                  I think the new multicore GC for OCaml has stop the world minor collections. Major collections are parallel.

                                                                                                                                                                                  In general I think what Paulson looks at OCaml through the interactive theorem prover lens, in particular through the Isabelle/HOL lens. That means that ML is the meta-language for your proofs, and that it’s loaded dynamically. The criticism of HOL light make a lot of sense then, because OCaml’s toplevel is not particularly designed for such a use case. However, the main proof assistant written in OCaml, Coq, uses OCaml as its implementation language, not as its meta-language. There, OCaml code natively compiled into a binary works very well (and I suspect it’s faster on a single thread than poly/ML).

                                                                                                                                                                                1. 4

                                                                                                                                                                                  It’s sad that well into the 21st Century, Computer Science has so regressed that people no longer see the point of distinguishing between a programming language and its implementation.

                                                                                                                                                                                  It hasn’t regressed. It has industrialized.

                                                                                                                                                                                  As somebody told me at the time, “the French did not get anything they wanted”. We know how that ended.

                                                                                                                                                                                  They ended up creating Caml which had none of the things they allegedly wanted…? 🤔

                                                                                                                                                                                  Much of the contention was over the ambiguity inherent in the following declaration:…let f x = ……With ISWIM, this defines f to be a function taking argument x. The same declaration in OCaml similarly declares the function f, unless f is a datatype constructor, when instead it declares x by pattern matching.

                                                                                                                                                                                  Well, no. In OCaml f cannot be a datatype constructor. Datatype constructors need to start with an uppercase. There’s actually no ambiguity.

                                                                                                                                                                                  it’s odd that OCaml doesn’t have a streamlined syntax for (drum roll) declaring a function using pattern-matching

                                                                                                                                                                                  Except it does…? Two of them in fact, one for non-refutable patterns: fun X -> .... And another for refutable patterns: function X -> ... | Y -> ...

                                                                                                                                                                                  It’s odd that the floating-point operators are +., -., etc., rather than being overloaded, as they are in almost every other programming language.

                                                                                                                                                                                  It also makes typechecking much easier, for each arithmetic operator you know its exact concrete type and it’s obvious from the code what arithmetic type you’re working with.

                                                                                                                                                                                  Attempts to move forward with ML were stymied by disagreements among its main proponents.

                                                                                                                                                                                  This is ironic after saying that ‘It’s sad that well into the 21st Century, Computer Science has so regressed that people no longer see the point of distinguishing between a programming language and its implementation’ and not seeing the connection.

                                                                                                                                                                                  All Caml strings were mutable, so you could change any constant, even T into F. It’s now fixed in OCaml, but who could overlook such a point when your key objective is soundness?

                                                                                                                                                                                  Mutability of strings has nothing to do with soundness. All mainstream languages have mutable bytestrings, I doubt people would go around claiming this makes them unsound.

                                                                                                                                                                                  With no way to save an image, you had to rebuild HOL Light from sources at every launch

                                                                                                                                                                                  This doesn’t make sense after saying that Caml emitted .o files and worked with Unix Makefiles just a few paragraphs earlier. It sounds like a HOL thing specifically and not a Caml thing.

                                                                                                                                                                                  But the full analysis library would not finish loading until after lunch. This is a world-beating system?

                                                                                                                                                                                  The bitterness starts to creep in here 😂

                                                                                                                                                                                  1. 3

                                                                                                                                                                                    It also makes typechecking much easier, for each arithmetic operator you know its exact concrete type and it’s obvious from the code what arithmetic type you’re working with.

                                                                                                                                                                                    There are quite a few things in OCaml I wish worked differently or seem like compromises to me, like the uppercase requirement for data constructors and inability to apply data constructors as functions. But separate operators for floats and integers seem like a feature that other languages should borrow. After all, machine floats and integers are fundamentally different and conversion can lead to results that are, at the very least, surprising.

                                                                                                                                                                                    If a language treats infix operators like functions and allows re-binding them, then any code that mostly works with floats can easily do let (+) = (+.) and avoid the verbosity without compromising type safety.

                                                                                                                                                                                    1. 1

                                                                                                                                                                                      Yup, in fact I even deliberately left out local opens in OCaml so as not to confuse people too much!

                                                                                                                                                                                    2. 3

                                                                                                                                                                                      This doesn’t make sense after saying that Caml emitted .o files and worked with Unix Makefiles just a few paragraphs earlier. It sounds like a HOL thing specifically and not a Caml thing.

                                                                                                                                                                                      SML is an image-based system in many implementations. OCaml is not.

                                                                                                                                                                                      1. 1

                                                                                                                                                                                        From the article:

                                                                                                                                                                                        you had to rebuild HOL Light from sources at every launch

                                                                                                                                                                                        OCaml and Caml before it are ‘image-based’ in the sense that they compile down to some ‘machine image’ code e.g. bytecode or native executable code. And they are designed to support incremental compilation in almost exactly the same way as C. So it doesn’t make sense to me that HOL Light required recompilation at every launch. Maybe if some linked modules changed, those would need to be recompiled and relinked, sure. Maybe some core parts changed, which would trigger ripple effect recompilations. But on every launch?

                                                                                                                                                                                        1. 4

                                                                                                                                                                                          are ‘image-based’ in the sense that they compile down to some ‘machine image’ code e.g. bytecode or native executable code.

                                                                                                                                                                                          ‘Image-based’ typically refers to something like Smalltalk or many Common Lisp environments where the code gets loaded in to create a runtime state of the system, then that runtime state is persisted and loaded again later as a means of running the program rather than emitting a binary in a format meant to be invoked by anything but the original runtime.

                                                                                                                                                                                          1. 3

                                                                                                                                                                                            That is not what an image-based system means. In HOL Light you are essentially in OCaml’s toplevel repl. You can’t save the image of the interactive repl, which is a problem for HOL Light, not only because loading it takes minutes, but you can’t save the state of your progress either. This is a non-issue in SML (and Lisp, and Smalltalk…).

                                                                                                                                                                                            HOL Light users use some Linux debugging hacks to be able to save and restore the OCaml toplevel.

                                                                                                                                                                                        2. 2

                                                                                                                                                                                          It hasn’t regressed. It has industrialized.

                                                                                                                                                                                          This is implying that industrialization and formalization are somehow at odds, which they aren’t.

                                                                                                                                                                                          Well, no. In OCaml f cannot be a datatype constructor. Datatype constructors need to start with an uppercase. There’s actually no ambiguity.

                                                                                                                                                                                          They’re clearly using f as a placeholder here. The point was that OCaml allowing pattern matching in a let binding was a difference from Standard ML.

                                                                                                                                                                                          It also makes typechecking much easier, for each arithmetic operator you know its exact concrete type and it’s obvious from the code what arithmetic type you’re working with.

                                                                                                                                                                                          This doesn’t seem to be an issue in any other language, i.e. type classes in Haskell.

                                                                                                                                                                                          1. 3

                                                                                                                                                                                            Haskell doesn’t have the equivalent of OCaml functors. As far as I know, the combination of functors and typeclasses (and opaque types, which OCaml has) is a research problem and no language currently supports both.

                                                                                                                                                                                            1. 1

                                                                                                                                                                                              I believe you are right. The advantage of functors is that you can instantiate them multiple times for the same types. Haskell’s polymorphism is based on type classes, which means you can only have a single instance for a given type. It’s very convenient for common cases, but leaks through in some edge cases in funny ways, like the fact that there are Sum and Product types that decorate numbers so that you can make them Foldable.

                                                                                                                                                                                            2. 2

                                                                                                                                                                                              They’re clearly using f as a placeholder here.

                                                                                                                                                                                              It’s far from clear, because if we look at the actual syntax, it’s pretty unambiguous because of the casing. The OP’s use of ‘meta’ f when the case is actually important, is misleading at best.

                                                                                                                                                                                              This doesn’t seem to be an issue in any other language, i.e. type classes in Haskell.

                                                                                                                                                                                              It’s actually a rather big issue in these languages. Haskell will give you indecipherable errors like ‘No instance for Num a found…’ and Elm and others have weird pseudonumber types that they use to get around the issues. OCaml keeps it simple and gives you a clear type error–you want an int or a float? Pick one.

                                                                                                                                                                                            3. 1

                                                                                                                                                                                              Can you expand on what you mean by “industrialized”?

                                                                                                                                                                                              1. 2

                                                                                                                                                                                                In industry (as opposed to academia), many tools/languages are driven by industrial needs and not by research agendas. Usually the industrial backers are not going to sit back and wait several years for their language to be formalized.

                                                                                                                                                                                                1. 2

                                                                                                                                                                                                  But every new OCaml feature is formalized before being added to the language. All parts of OCaml are formalized, it just lacks a formal specification.

                                                                                                                                                                                                  1. 1

                                                                                                                                                                                                    I think my meaning was clear, I’m talking about the specification of the language as opposed to its implementation. That’s pretty clearly the meaning from the OP too.

                                                                                                                                                                                                    1. 1

                                                                                                                                                                                                      But the argument put forward was that due to commercial pressures there is no time do to formalization, and the opposite happens in OCaml. No only there is time, but the formalization happens before implementation. So the reason for OCaml lacking a formal specification cannot be this.

                                                                                                                                                                                                      1. 1

                                                                                                                                                                                                        Yes, my language was slightly imprecise but to probably 95% of programmers there’s not that much difference.

                                                                                                                                                                                            1. 2

                                                                                                                                                                                              For those who use OCaml, what is your application domain?

                                                                                                                                                                                              1. 5

                                                                                                                                                                                                I wouldn’t call myself one that uses OCaml, but I used Lambda Soup recently to do some markup parsing/rewriting and it was a lot nicer than doing the same in Python and Beautiful Soup. Would recommend.

                                                                                                                                                                                                1. 4

                                                                                                                                                                                                  It’s a general-purpose language so anywhere really. You can use it in the same domains as Python, Haskell, Go, Java, JavaScript, PHP, and many such languages.

                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                    I know, I have used OCaml and other languages from the ML family extensively.

                                                                                                                                                                                                    However, libraries pretty much dictate what you use a language for, in practice.

                                                                                                                                                                                                    1. 3

                                                                                                                                                                                                      There is a respectable number of libraries covering most domains. Although it’s hard to tell what different people consider essential. One person can’t use it unless it has Google Cloud bindings. Another can’t use it without Qt bindings. These are all in various states of completion because they’re not officially maintained by anyone, they’re just side projects.

                                                                                                                                                                                                  2. 4

                                                                                                                                                                                                    so far i’ve mostly been using it as a hobby language for word game related programs (crosswords, anagram generator), but it’s a pleasant general purpose language. only thing i’ve really missed is a good ORM.

                                                                                                                                                                                                    1. 2

                                                                                                                                                                                                      Formal verification, logic, automated reasoning. Admittedly a lot of that was during my time in academia but it shines at it.

                                                                                                                                                                                                    1. 4

                                                                                                                                                                                                      Will the addition of multicore and algebraic effects make OCaml gain more traction?

                                                                                                                                                                                                      Where does it stand compared to F#, Haskell, Scala or Rust right now?

                                                                                                                                                                                                      1. 9

                                                                                                                                                                                                        Will the addition of multicore and algebraic effects make OCaml gain more traction?

                                                                                                                                                                                                        It’s difficult to make predictions, especially about the future, but in my opinion, I very much doubt these things would have any effect on traction. In my experience people will not use OCaml because they don’t know, or don’t want to know functional programming, or simply don’t want it for various personal or organizational reasons. It’s almost never about the lack of any particular feature. In fact, people who reject OCaml outright very rarely have any idea what feature OCaml has or lacks.

                                                                                                                                                                                                        Where does it stand compared to F#, Haskell, Scala or Rust right now?

                                                                                                                                                                                                        I am not sure comparing any of those languages makes sense.

                                                                                                                                                                                                        F# is .NET tech, yes you can run it on Linux or whatever, but realistically speaking, if you are not into Windows/.NET you would never use this. Also F# has the fewest features of the bunch, which is not necessarily a bad thing, but you mentioned fancy features like algebraic effects whereas F# is late 80s level of sophistication. F# doesn’t even have ML-style modules. And btw, this is not a me taking at jab at F#, 80s level of sophistication is great when the industry is still stuck in the 60s. Of course, F# is great if you are stuck in the .NET ecosystem.

                                                                                                                                                                                                        Haskell is a non-strict language. OCaml makes it much easier to reason about performance, and to get good performance. Haskell has many features OCaml will never have (too many to list here), but now they have linear functions, and are discussing ways of adding dependent types. Very cool stuff, but unrelated to the reason people use OCaml.

                                                                                                                                                                                                        Scala is in the Java ecosystem. Just like with F#, Scala is a non-option if you are not already in the Java ecosystem. And if you are, all the other languages you mention will not be an option, so it’s all moot.

                                                                                                                                                                                                        Rust is not a functional language and has a very different applicability domain compared to any of the languages mentioned so far which makes any comparison difficult and pointless. It’s the only one specifically targeting low-level stuff. Yes, I am very aware of MirageOS, it’s a very niche thing, and has far less applicability than Rust in general. E.g. people use rust to run code on microcontrollers today. Feature wise, its type system is the most limited compared to all languages discussed above. I very much doubt Rust will ever get algebraic effects, because algebraic effects are hard without a GC. And I have no idea if OCaml will ever get linear types, or if they even want them. Linear types are difficult to mix with effects.

                                                                                                                                                                                                        Algebraic effects and linear types are very different things, but both can be employed to tame IO-effects, and both OCaml (with effects) and Rust (with linear types) do exactly that. So since they solved “the IO problem”, I very much doubt they would be interested in adding more features, especially Rust.

                                                                                                                                                                                                        For the record, I am working on a language that has both effects and linear types.

                                                                                                                                                                                                        1. 2

                                                                                                                                                                                                          For the record, I am working on a language that has both effects and linear types.

                                                                                                                                                                                                          Sounds interesting - have you published anything about it so far?

                                                                                                                                                                                                          1. 3

                                                                                                                                                                                                            Not yet.

                                                                                                                                                                                                        2. 7

                                                                                                                                                                                                          One of the killers of D’s momentum over a decade ago was the conflict between two (three?) possible standard libraries.

                                                                                                                                                                                                          While a lot of other problems have been solved, Ocaml still suffers from this one. Until the whole mess with multicore/lwt/async sorts itself out (which will also impact any standard library), and the community completely settles on a single standard library, I do not think there is that much hope for increased traction. Also: documentation that isn’t just type signatures, which (especially) many third-party libraries suffer from.

                                                                                                                                                                                                          For better or worse, Rust is a better option for the time being, mostly due to better tooling and a vastly larger library ecosystem. I’ve been down this road several times, and ecosystem size almost always wins out, unless you’re tinkering.

                                                                                                                                                                                                          1. 7

                                                                                                                                                                                                            Until the whole mess with multicore/lwt/async sorts itself out (which will also impact any standard library), and the community completely settles on a single standard library …

                                                                                                                                                                                                            Effects mean you don’t need monadic libraries like lwt and async now, and most “concurrent” code can just be written as plain OCaml, so things will likely converge. Porting code is straight-forward and can be done incrementally. See e.g. the migration instructions at lwt_eio (which allows running Lwt code under the (effects-based) Eio event loop).

                                                                                                                                                                                                            You can even run both Lwt and Async code together, along with new effects based Eio code, in a single domain. See e.g. https://github.com/talex5/async-eio-lwt-chimera!

                                                                                                                                                                                                            1. 1

                                                                                                                                                                                                              For context, u/talex5 is the main dev of Eio. I think a lot of us (OCaml users) are looking forward to a world without monadic IO libraries, and Eio is exactly that, along with clean primitives for byte streams, structured concurrency, etc. If there’s a time where OCaml can gain more interest, I think OCaml 5.0 (or 5.1) is exactly that time.

                                                                                                                                                                                                        1. 7

                                                                                                                                                                                                          A decade ago I’ve spent a lot of time learning Haskell and trying to apply that knowledge solving practical problems. Maybe my brain was already wired in a different direction, but I’ve failed to finish every non-trivial project I was starting. During this time I was showing the same amount of zeal and evanghelism in praising Haskell whenever I got the chance, the rust community is giving us nowadays.

                                                                                                                                                                                                          Then I’ve tried Scala, and things were better, but I was cheating writing Scala the Java way.

                                                                                                                                                                                                          To this day I believe that the masses don’t need Functional programming in its fullest, but maybe a few functional features here and there that can make the code more compact. Or maybe I am failed and bitter functional developer :).

                                                                                                                                                                                                          1. 3

                                                                                                                                                                                                            Both Scala and Haskell are very complex languages. If you want to learn statically typed functional programming, Elm can be a cool first experience. If you prefer dynamic, Elixir is good too.

                                                                                                                                                                                                            1. 3

                                                                                                                                                                                                              Then you’re in luck, because OCaml is not purely functional :-). It offers you a blend of imperative and functional that allows you to pick the best flavor for a particular task (a bit like Java-style Scala, I’d say). You can have actual pure values where it simplifies your life (caches, messages in concurrent programming, etc.) but you also don’t need to embed in a monad to perform IO.

                                                                                                                                                                                                              1. 2

                                                                                                                                                                                                                Ditto. I learned a lot of cool and important concepts from Haskell, and I try to apply those ideas in other languages, but overall I spent too much time fighting with the language to make it worthwhile for me to use Haskell itself.

                                                                                                                                                                                                                1. 4

                                                                                                                                                                                                                  Haskell, Smalltalk, and Erlang are all languages that I think everyone should learn, but very few people should use. They all take a useful idea to its extreme and learning to think in the ways that they encourage gives you a useful set of tools that can be applied almost anywhere.

                                                                                                                                                                                                              1. 2

                                                                                                                                                                                                                I wish there was a version of sqlite that was as strict with types as postgres is.

                                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                                    Neat, this mode seems to be a recent addition, but it is currently a bit restrictive though. I like the improved ANY behavior, but there is no json, coordinate or date type for it to constrain. Yet this is a big improvement! Thanks

                                                                                                                                                                                                                    1. 4

                                                                                                                                                                                                                      You can enforce the syntax of column values (ensure they are valid JSON for example) using check constraints in a CREATE TABLE.

                                                                                                                                                                                                                      1. 3

                                                                                                                                                                                                                        Here’s how to do that:

                                                                                                                                                                                                                        sqlite> create table test (id integer primary key, tags text, check (json(tags) is not null));
                                                                                                                                                                                                                        sqlite> sqlite> insert into test (tags ('["one", "two"]');
                                                                                                                                                                                                                        sqlite> insert into test (tags) values ('["one", "two"');
                                                                                                                                                                                                                        Error: stepping, malformed JSON (1)
                                                                                                                                                                                                                        
                                                                                                                                                                                                                  2. 3

                                                                                                                                                                                                                    Yeah. Every time I see this stuff I think it’s cool, but I really don’t want to give up the features of postgres in terms of types and triggers.