Threads for brendan

  1. 5

    fathom is another project to watch in this space.

    1. 2

      Thanks for the mention, I’m working on Fathom right now!

      Responding to some of the criticisms of Katai:

      parsing a kaitai schema is significantly more involved than parsing an openapi schema. one contributing factor is that kaitai features an embedded expression language, allowing almost any attribute to be bound to expressions involving other fields, rather than simple constants. this makes is hard for an ecosystem to thrive because of a high technical barrier for entry.

      The tricky thing about real-world binary formats is just how interwoven they are with programming logic: often they start off implemented as some random bit of C code and then we have to go back and pick up the pieces in order to attempt to describe them declaratively and safely (and ideally in a way can be checked using a proof assistant in the future). It’s a massive pain.

      the question of serialization remains; however, the technical challenges seem largely connected to the use of a feature in kaitai called “instances” rather than “sequences”. this feature allows defining a field at a fixed offset, rather than defining it in context to the field before or after. this feature is obviously very useful to reverse engineers, who may understand only a certain chunk of a binary blob and need to annotate its fields before moving on, but it wouldn’t be a desireable feature in a schema that serves as the source definition for a binary format.

      This is really hard to avoid if you want to deal with real world formats. Formats like OpenType are sprawling, with offsets and pointers to other parts of the binary data, sometimes with mutual dependencies between offset data.

      Ultimately it sounds like the author really wants a restricted language for restricted a subset of new binary formats. If you have the luxury of writing your own binary format from scratch, then metaformats like Protocol Buffers or Cap’n Proto could offer a more restricted approach that might make this kind of thing easier. I’m not sure how they fare for binary files though, as I’m pretty sure they are more designed to describe streaming data.

      While I’m here, I’d also like to call out some additional tools that I think don’t get enough airtime:

      • binary-data: DSL for parsing binary data for Dylan
      • Everparse: parser tools that extract an ‘RFC’ format to verified F* code and ultimately C, used as part of Project Everest (formally verified TLS stack by Inria and Microsoft) (paper)
      • Narcissus: a DSL embedded in Coq for formally verified descriptions of binary data formats (paper)

      More interesting tools and languages can be found at dloss/binary-parsing.

    1. 2

      Makes me appreciate the conceptual simplicity of dependent types.

      1. 2

        Way harder to typecheck in practice, though.

        1. 3

          Not necessarily. Check out Andrej Bauer’s “Spartan Type Theory” for a simple OCaml implementation. The link to the lecture recording no longer seems to work, but there are some slides too.

          1. 2

            Not too much harder? What complexity are you thinking of?

        1. 11

          Writing these book is fundamentally incompatible with Haskell’s approach of constantly making breaking changes to everything. Books take time to write. On the time scale of 1-2 years. The reality of the Haskell world is that the contents of the resulting book would be so woefully out of date as to be useless.

          I say this sitting on most of a book on modern ML in Haskell.

          It takes a lot of time to explain something coherently, make examples, describe the logic behind how a system is designed, etc. How can you possibly do that if everything constantly changes? I can either write materials to explain things in Haskell, where everything will be out of date within a year, or I can explain things in Python, where years later I don’t need to revise the materials.

          Take ‘Haskell in Depth’ published in 2021. People literally cannot run the code months later: https://github.com/bravit/hid-examples/issues/10 That’s absurd. Writing niche books isn’t particularly lucrative anyway, having to constantly rewrite the same book is borderline madness.

          The best example of how impossible it is to write such materials is Haskell itself. There is no document explaining the Haskell type system and how to perform inference in it. At best, there are some outdated papers going back many years that each sort of describe pieces of the system.

          1. 7

            I’ve been writing Haskell for 15+ years and not yet encourtered a serious breaking change.

            1. 5

              “Serious breaking changes” aren’t the biggest issue at all. Small constant breaking changes to the compiler, to language, to core libraries, and to the entire ecosystem in general are the issue. Seriously, that book is barely out and the code doesn’t work anymore.

              Every change that breaks the API of a package, a commandline flag, etc. you need to track all of those down in hundreds pages of material. In hundreds of examples. Constantly. It’s hard enough to keep up with changes to code that compiles. If I did this for a book on ML it would be my full-time job.

              1. 6

                Seriously, that book is barely out and the code doesn’t work anymore.

                Do you mean doesn’t work on the latest GHC with bleeding edge packages from Cabal? Maybe I’m insulated from this a bit by sticking to the version of GHC in Debian stable and avoiding new versions of packages when I don’t need them.

                1. 2

                  It’s the opposite problem. The published code doesn’t build using the versions it states.

                  The repo’s stack.yaml uses resolver: lts-17.4 and the linked issue’s solution is to upgrade to lts-18.17. and upgrade dependency versions.

                  1. 1

                    I’m not very familiar with how stack works. Does this mean that stack allowed breaking changes inside an existing release, or what exactly is the actual issue?

              2. 0

                this is false

              3. 6

                People literally cannot run the code months later

                This seems unlikely, as both stack and cabal are fully reproducible[1]. One wouldn’t expect a correct build description to stop working because new versions of packages and the compiler are released. Perhaps there’s just a bug in the book?

                [1] Ok, not Nix level reproducible!

                1. 4

                  There is no document explaining the Haskell type system and how to perform inference in it.

                  The Haskell Report and the GHC User’s Guide should fully describe the behaviour of the type system. If you mean there’s no single document explaining the implementation of the type system then you may be right, but is there such a document for similar compilers such as OCaml, Scala, F#? Perhaps Rust has one because it is much newer, but I’m not sure.

                  1. 3

                    The Haskell Report describes a 20 year old version of the language. The GHC user guide is a vague overview, it’s completely impossible to implement anything from the details provided in the user guide.

                    It’s not a matter of describing the current implementation. There is no document that describes the type system as it exists today at all. As in, there is no mathematical description of what Haskell is and how you are supposed to perform type inference. The best that exists is a 10+ year old paper (the OutsideIn paper) but that’s incredibly outdated, you couldn’t type check common Haskell code used today with it, and it’s not even clear it ever corresponded to any version Haskell precisely.

                    There are good reasons why this is. It takes a lot of time to write such documents. But if the language developers can’t keep up with the language themselves, it’s hard to imagine that others will be able to do so.

                    For what it’s worth, OCaml is very clearly described mathematically in plenty of places, even in textbooks. My understanding about the situation in Scala is not just that the type system is described accurately, it’s actually been machine checked. I’m least familiar with the situation in F#, but it’s in the OCaml family. There probably aren’t any surprises there.

                    1. 3

                      The Haskell Report describes a 20 year old version of the language.

                      12 year old version, let’s not over age ourselves :)

                      There has been no new version of Haskell since 2010. Some consider that an issue, and it may well be, but until there is a new one the fact that it is old does not make it wrong.

                      1. 1

                        It’s more like 24 years actually.

                        Haskell 2010 is not Haskell as it was in 2010. It’s a minor update to Haskell 98 because already no one could keep up with the changes.

                        It’s all in the first few paragraphs of the report where they describe this problem. Then they say this is an incremental and conservative update that only adds a few minor fixes and features to 98.

                        1. 1

                          Haskell 2010 defines what Haskell is it’s not descriptive of some mystical “Haskell” that may exist somewhere else that it incompletely describes. It’s a definition.

                          Standards being conservative is good. Can you imagine if every crazy language extension in GHC became part of Haskell? Some of them are even in competition or contradictory!

                          1. 2

                            You haven’t read the report. You should. It literally starts by saying it doesn’t define what Haskell is in 2010.

                            1. 1

                              I linked the Haskell 2010 Report in a sibling thread. I don’t see where it says it doesn’t define what Haskell is in 2010. Could you please point it out?

                              1. 5

                                I linked the Haskell 2010 Report in a sibling thread. I don’t see where it says it doesn’t define what Haskell is in 2010. Could you please point it out?

                                page xiv says the language has grown so much and the effort to document it is now so high, that this is going to be a small incremental update with more to come. More did not come, the documentation burden was so high everyone gave up. And the update was indeed very very small, covering only 3 major changes: FFI, pattern guards, and hierarchical module names. I pasted the contents below.

                                For reference, GHC 7 came out in 2010.

                                Even at the time in 2009 people were wondering what is up, because bread and butter parts of the language weren’t going to be included in Haskell 2020. For example, no GADTs, no associated types, no rank n types, etc. Here is someone in 2009 asking about this and getting the response that, no, this doesn’t reflect the language, but it’s the best anyone can do because keeping up with the language is so hard: https://mail.haskell.org/pipermail/haskell-prime/2009-July/002817.html The main barrier to entry is if anyone can describe that extension faithfully, but no one could.

                                Sadly, most of the archives of the haskell-prime effort seem to have been lost.

                                In any case. This is not a criticism of the Haskell2010 authors. They did their best. But, it’s important that the community realizes that the sorry state of the documentation, the lack of high quality materials like books, and the lack of in-depth ecosystems for areas like ML, are all a consequence of this decision to keep making breaking changes to the language, core libraries, and ecosystem as a whole.

                                At the 2005 Haskell Workshop, the consensus was that so many extensions to the official language were widely used (and supported by multiple implementations), that it was worthwhile to define another iteration of the language standard, essentially to codify (and legitimise) the status quo.

                                The Haskell Prime effort was thus conceived as a relatively conservative extension of Haskell 98, taking on board new features only where they were well understood and widely agreed upon. It too was intended to be a “stable” language, yet reflecting the considerable progress in research on language design in recent years.

                                After several years exploring the design space, it was decided that a single monolithic revision of the language was too large a task, and the best way to make progress was to evolve the language in small incremental steps, each revision integrating only a small number of well-understood extensions and changes. Haskell 2010 is the first revision to be created in this way, and new revisions are expected once per year.

                                1. 3

                                  I see. So, reflecting on what you wrote, the lack of published Haskell standard doesn’t seem to be the problem you are experiencing just the symptom. After all, Python doesn’t have a standard but you state that it would be a fine target for writing reference materials.

                                  I can think of only one change to the language that has caused me frustration: simplified subsumption. There have been a few frustrating library-level changes too.

                                  Could you elaborate on which changes in the Haskell ecosystem have led to concrete problems for you? Firstly, I may be able to offer advice or a fix. Secondly, I may actually be able to tackle the root cause and make future experience better for you and others.

                      2. 2

                        I am highly sceptical that there is a mathematical description of OCaml or Scala that matches how the language is used in production today. I could easily be wrong, because it’s not my area of expertise, but I know for sure that those languages move, albeit slower than Haskell, and I doubt that any published material can keep up with how those compilers develop in practice. Someone was telling me recently he is adding algebraic effects to the OCaml compiler! It seems unlikely that is in the textbooks that you mentioned.

                        I would be grateful for any material you can link me to that elaborates on the difference between Haskell and the other languages in this regard.

                        That said, this is getting somewhat removed from your original comment, because if you stick to the well-trodden parts of the type system, Haskell2010 certainly, and what is becoming GHC2021 most likely, then none of the breakage to the book you are writing will be to do with the type system per se.

                        1. 1

                          For what it’s worth, OCaml is very clearly described mathematically in plenty of places, even in textbooks. My understanding about the situation in Scala is not just that the type system is described accurately, it’s actually been machine checked. I’m least familiar with the situation in F#, but it’s in the OCaml family. There probably aren’t any surprises there.

                          Are you confusing OCaml with Standard ML?

                          Scala 3’s core language (the DOT calculus) has been specified and verified in Coq, but not the surface language as far as I’m aware. I’m also not sure if the implementation is based on elaboration into the core language (this is generally the approach that dependently typed languages use to improve confidence in the implementation).

                    1. 3

                      There are very limited options for saying “I’ll do this part later”… You application has to compile.

                      I think there are many good things in Elm, but I do find this a bit frustrating, given that there are tools available to address this for statically typed languages, like typed holes and deferred type errors, that can reduce the amount of friction during development while not compromising on correctness.

                      1. 3

                        PureScript breaking out to FFI was an easy way to hack around this for now and clean up or upstream later. The best “hacks” in my experience involved things along the lines of using globalThis.Intl, instantiating an Intl formatter in the FFI file with it’s options and then just consuming IntlObj.format as a pure, synchronous function to get quick date/number formats.

                        1. 2

                          Usually when I write Elm, I start with the types and let the type errors guide me to what I want to achieve. So I guess the workflow is types => base functions => actual usage in update/view. I have a similar workflow in Derw, but it’s actually possible to work around that currently by calling some global function since they aren’t type checked. So you could have

                          isNumber: string -> boolean
                          isNumber str = 
                            globalThis.doNothing
                          

                          and replace globalThis.doNothing once you’ve figured out what goes there.

                          1. 1

                            Does Elm not have the equivalent of Haskell’s undefined?

                            1. 1

                              It has Debug.todo "some message" which you can use similarly. The difference is you can’t use any Debug functions in prod builds. I use them all the time when I’m blocking out my code, and then remove then 1 at a time till I get what I need.

                        1. 6

                          I will stay out of the mater in discussion. Inline formating vs optional appended formating. What I really enjoyed and will point out, is how properly you layed out you idea. What motivate you to write this, and all the clear reasoning for why you built this the way you did. All pros and cons clearly analysied.

                          I wish software development would have more of this culture. So many useless projects, so much unnecessary overhead that could be avoided if people had a better culture of thinking before opening their IDE. Keep up.

                          1. 2

                            Historically I haven’t been so good at thinking things through and getting ideas into well structured symbols. I’ve been making a sustained effort to improve. So thank you for the kind words. Maybe it means I’ve made at least a little progress.

                            1. 2

                              This could be a post of its own.

                              1. 2

                                This is why I’m grateful for a lot of academic work (especially in the field I’m interested in: type systems and programming languages). While there are many issues with the academic publishing system, I never cease to be grateful for the level of quality of work you can find there. It’s always extremely helpful to see a comparison to prior work, and a specification sketched out, along with motivating examples etc. I remember reading this post thinking ‘this reminds me of standoff markup’ and then being pleasantly surprised to see that mentioned under a “Related work” section. Props to the OP for doing this!

                              1. 15

                                You lost me at “the great work from homebrew”

                                Ignoring UNIX security best practices of the last. I dunno, 30, 40 years, and then actively preventing people from using the tool in any fashion that might be more secure, and refusing to acknowledge any such concerns is hardly “great work”.

                                I could go on about their abysmal dependency resolution logic, but really if the security shit show wasn’t enough to convince you, the other failings won’t either.

                                But also - suggesting Apple ship a first party container management tool because “other solutions use a VM”, suggests that either you think a lot of people want macOS containers (I’m pretty sure they don’t) or that you don’t understand what a container is/how it works.

                                The “WSL is great because now I don’t need a VM” is either ridiculously good sarcasm, or yet again, evidence that you don’t know how something works. (For those unaware, WSL2 is just a VM. Yes it’s prettied up to make it more seamless, but it’s a VM.).

                                1. 23

                                  I don’t know what’s SO wrong about Homebrew that every time it’s mentioned someone has to come and say that it sucks.

                                  For the use case of a personal computer, Homebrew is great. The packages are simple, it’s possible and easy to install packages locally (I install mine in ~/.Homebrew) and all my dependencies are always up to date. What would a « proper » package manager do better than Homebrew that I care about? Be specific please because I have no idea what you’re talking about in terms of security « shit show » or « abysmal » dependency resolution.

                                  1. 12
                                    • A proper package manager wouldn’t allow unauthenticated installs into a global (from a $PATH perspective) location.
                                    • A proper package manager wouldn’t actively prevent the user from removing the “WTF DILIGAF” permissions Homebrew sets and requiring authenticated installs.
                                    • A proper package manager that has some form of “install binaries from source” would support and actively encourage building as an untrusted user, and requiring authentication to install.
                                    • A proper package manager would resolve dynamic dependencies at install time not at build time.
                                    • A proper open source community wouldn’t close down any conversation that dares to criticise their shit.
                                    1. 11

                                      Literally none of those things have ever had any impact on me after what, like a decade of using Homebrew? I’m sorry if you’ve run into problems in the past, but it’s never a good idea to project your experience onto an entire community of people. That way lies frustration.

                                      1. 5

                                        Who knew that people would have different experiences using software.

                                        it’s never a good idea to project your experience onto an entire community of people

                                        You should take your own advice. The things I stated are objective facts. I didn’t comment on how they will affect you as an individual, I stated what the core underlying issue is.

                                        1. 6

                                          You summarized your opinion on “proper” package managers and presented it as an authoritative standpoint. I don’t see objectiveness anywhere.

                                      2. 3

                                        I don’t really understand the fuss about point 1. The vast majority of developer machines are single user systems. If an attacker manages to get into the user account it barely matters if they can or cannot install packages since they can already read your bank passwords, SSH keys and so on. Mandatory relevant xkcd.

                                        Surely, having the package manager require root to install packages would be useful in many scenarios but most users of Homebrew rightfully don’t care.

                                      3. 8

                                        As an occasional Python developer, I dislike that Homebrew breaks old versions of Python, including old virtualenvs, when a new version comes out. I get that the system is designed to always get you the latest version of stuff and have it all work together, but in the case of Python, Node, Ruby, etc. it should really be designed that it gets you the latest point releases, but leaves the 3.X versions to be installed side by side, since too much breaks from 3.6 to 3.7 or whatever.

                                        1. 8

                                          In my opinion for languages that can break between minor releases you should use a version manager (python seems to have pyenv). That’s what I do with node: I use Homebrew to install nvm and I use nvm to manage my node versions. For Go in comparison I just use the latest version from Homebrew because I know their goal is retro compatibility.

                                          1. 5

                                            Yeah, I eventually switched to Pyenv, but like, why? Homebrew is a package manager. Pyenv is a package manager… just for Python. Why can’t homebrew just do this for me instead of requiring me to use another tool?

                                            1. 1

                                              Or you could use asdf for managing python and node.

                                            2. 7

                                              FWIW I treat Homebrew’s Python as a dependency for other apps installed via Homebrew. I avoid using it for my own projects. I can’t speak on behalf of Homebrew officially, but that’s generally how Homebrew treats the compilers and runtimes. That is, you can use what Homebrew installs if you’re willing to accept that Homebrew is a rolling package manager that strives always to be up-to-date with the latest releases.

                                              If you’re building software that needs to support a version of Python that is not Homebrew’s favored version, you’re best off using pyenv with brew install pyenv or a similar tool. Getting my teams at work off of brewed Python and onto pyenv-managed Python was a short work that’s saved a good bit of troubleshooting time.

                                              1. 2

                                                This is how I have started treating Homebrew as well, but I wish it were different and suitable for use as pyenv replacement.

                                                1. 2

                                                  asdf is another decent option too.

                                                2. 5

                                                  I’m a Python developer, and I use virtual environments, and I use Homebrew, and I understand how this could theoretically happen… yet I’ve literally never experienced it.

                                                  it should really be designed that it gets you the latest point releases, but leaves the 3.X versions to be installed side by side, since too much breaks from 3.6 to 3.7 or whatever.

                                                  Yep, that’s what it does. Install python@3.7 and you’ve got Python 3.7.x forever.

                                                  1. 1

                                                    Maybe I’m just holding it wrong. :-/

                                                  2. 3

                                                    I found this article helpful that was floating around a few months ago: https://justinmayer.com/posts/homebrew-python-is-not-for-you/

                                                    I use macports btw where I have python 3.8, 3.9 and 3.10 installed side by side and it works reasonably well.

                                                    For node I gave up (only need it for small things) and I use nvm now.

                                                  3. 8

                                                    Homebrew is decent, but Nix for Darwin is usually available. There are in-depth comparisons between them, but in ten words or less: atomic upgrade and rollback; also, reproducibility by default.

                                                    1. 9

                                                      And Apple causes tons of grief for the Nix team every macOS release. It would be nice if they stopped doing that.

                                                      1. 2

                                                        I stopped using Nix on macOS after it is required to create an separate unencrypted volume just for Nix. Fortunately, NixOS works great on VM.

                                                        1. 2

                                                          It seems to work on an encrypted volume now at least!

                                                    2. 4

                                                      I really really hate how homebrew never ask me for confirmation. If I run brew upgrade it just does it. I have zero control over it.

                                                      I come from zypper and dnf, which are both great examples of really good UX. I guess if all you know is homebrew or .dmg files, homebrew is amazing. Compared to other package managers, it might even be worse than winget….

                                                      1. 2

                                                        If I run brew upgrade it just does it

                                                        … yeah? Can we agree that this is a weird criticism or is it just me?

                                                      2. 2

                                                        Overall I like it a lot and I’m very grateful brew exists. It’s smooth sailing the vast majority of the time.

                                                        The only downside I get is: upgrades are not perfectly reliable. I’ve seen it break software on upgrades, with nasty dynamic linker errors.

                                                        Aside from that it works great. IME it works very reliably if I install all the applications I want in one go from a clean slate and then don’t poke brew again.

                                                      3. 4

                                                        you think a lot of people want macOS containers (I’m pretty sure they don’t)

                                                        I would LOVE macOS containers! Right now, in order to run a build on a macOS in CI I have to accept whatever the machine I’m given has installed (and the version of the OS) and just hope that’s good enough, or I have to script a bunch of install / configuration stuff (and I still can’t change the OS version) that has to run every single time.

                                                        Basically, I’d love to be able to use macOS containers in the exact same way I use Linux containers for CI.

                                                        1. 1

                                                          Yes!!

                                                          1. Headless macos would be wonderful
                                                          2. Containers would be fantastic. Even without the docker-like incremental builds, something like FreeBSD jails or LXC containers would be very empowering for build environments, dev servers, etc
                                                          1. 1

                                                            Containers would be fantastic. Even without the docker-like incremental builds, something like FreeBSD jails or LXC containers would be very empowering for build environments, dev servers, etc

                                                            These days, Docker (well, Moby) delegates to containerd for managing both isolation environments and image management.

                                                            Docker originally used a union filesystem abstraction and tried to emulate that everywhere. Containerd provides a snapshot abstraction and tries to emulate that everywhere. This works a lot better because you can trivially implement snapshots with union mounts (each snapshot is a separate directory that you union mount on top of another one) but the converse is hard. APFS has ZFS-like snapshot support and so adding an APFS snapshotter to containerd is ‘just work’ - it doesn’t require anything else.

                                                            If the OS provides a filesystem with snapshotting and a isolation mechanism then it’s relatively easy to add a containerd snapshotter and shim to use them (at least, in comparison with writing a container management system from scratch).

                                                            Even without a shared-kernel virtualisation system, you could probably use xhyve[1] to run macOS VMs for each container. As far as I recall, the macOS EULA allows you to run as many macOS VMs on Apple hardware as you want.

                                                            [1] xhyve is a port of FreeBSD’s bhyve to run on top of the XNU hypervisor framework, which is used by the Mac version of Docker to run Linux VMs.

                                                        2. 2

                                                          Ignoring which particular bits of Unix security practices is problematic? There are functionally no Macs in use today that are multi-user systems.

                                                          1. 3

                                                            All of my macs and my families macs are multi-user.

                                                            1. 2

                                                              The different services in OS are running as different users. It is in general good thing to run different services with minimal required privileges, different OS provided services run with different privileges, different Homebrew services run with different privileges, etc. So reducing the blast radius, even if there is only one human user is a pro, as there are often more users at once, just not all users are meatbags.

                                                            2. 1

                                                              I’ve been a homebrew user since my latest mac (2018) but my previous one (2011) I used macports, given you seem to have more of an understanding of what a package manager should do than I have, do you have any thoughts on macports?

                                                              1. 4

                                                                I believe MacPorts does a better job of things, but I can’t speak to it specifically, as I haven’t used it in a very long time.

                                                                1. 1

                                                                  Thanks for the response, it does seem like it’s lost its popularity and I’m not quite sure why. I went with brew simply because it seemed to be what most articles/docs I looked at were using.

                                                                  1. 3

                                                                    I went with brew simply because it seemed to be what most articles/docs I looked at were using.

                                                                    Pretty much this reason. Homebrew came out when macports was still source-only installs and had some other subtle gotchas. Since then, those have been cleared up but homebrew had already snowballed into “it’s what my friends are all using”

                                                                    I will always install MP on every Mac I use, but I’ve known I’ve been in the minority for quite awhile.

                                                                    1. 1

                                                                      Do you find the number of packages to be comparable to brew? I don’t have a good enough reason to switch but would potentially use it again when I get another mac in the future.

                                                                      1. 3

                                                                        I’ve usually been able to find something unless it’s extremely new, obscure, or has bulky dependencies like gtk/qt or specific versions of llvm/gcc. The other nice thing is that if the build is relatively standard, uses ‘configure’ or fits into an existing PortGroup, it’s usually pretty quick to whip up a local Portfile(which are TCL-based so it’s easy to copy a similar package config and modify to fit).

                                                                        Disclaimer: I don’t work on web frontends so I usually don’t deal with node or JS/TS-specific tools.

                                                                        1. 3

                                                                          On MacPorts vs Homebrew I usually blame popularity first and irrational fear of the term ‘Ports’ as in “BSD Ports System”, second. On the second cause, a lot of people just don’t seem to know that what started off as a way to have ‘configure; make; make install’ more maintainable across multiple machines has turned into a binary package creation system. I don’t anything about Homebrew so I can’t comment there.

                                                              1. 9

                                                                You can have very nice “easter eggs” and properly document them (but do they still deserve that name then?).

                                                                For instance, I like the Redis command “LOLWUT”: https://redis.io/commands/lolwut

                                                                1. 4

                                                                  Yes. Who says easter eggs should be undocumented? As long as they are documented and require a specific input that is never used in normal program operation to fire, I have nothing against them, even in “critical” projects (critical is subjective, of course).

                                                                  1. 3

                                                                    the teapot is another nice example of this.

                                                                    1. 2

                                                                      418 forever, I’m with you!

                                                                      1. 2

                                                                        I use the HTTP 418 response code in a critical project—my gopher server to inform wayward web bots they’re not talking to a web server. It’s great for that.

                                                                    2. 3

                                                                      LOLWUT wants to be a reminder that there is more in programming than just putting some code together in order to create something useful.

                                                                      I love this sentiment.

                                                                    1. 2

                                                                      I’ve enjoyed how they render Coq proofs with all the prove state extracted from Coq and available on hover.

                                                                      1. 1

                                                                        That is legitimately awesome. Does anyone have any idea what plugin / library enables that? And, if it works for other proof assistants like Isabelle? (I’m not a Coq user)

                                                                          1. 1

                                                                            Awesome, thank you. Wish it worked for Isabelle, but that’s great to have a tool to reference.

                                                                            1. 1

                                                                              Ahh, based on the README it seems like it has preliminary support for Lean 3, which is neat!

                                                                            2. 2

                                                                              You might also enjoy this presentation by the creator, and the paper that goes with it: Untangling mechanized proofs.

                                                                              1. 1

                                                                                Yes, that’s great content, thank you.

                                                                                Proof assistants are amazing tools, and some of them do offer a way to print out the full proof that they end up automating. Showing intermediate proof states is definitely nice, but some tactics still take very large leaps (like tauto used in the paper). In those cases, I want to see insight into how the step was taken in between goals as well.

                                                                          1. 3

                                                                            Still super jealous of SML’s module system! Hoping that more programming language designers learn about them, and from later extensions like MixML, 1ML, and Modular Implicits. Would be great to see them pop up again in future languages.

                                                                            1. 4

                                                                              I have mostly warm feelings towards SML, and I absolutely admire its formally verified specification. However, that specification also made SML itself impractical to extend, and it hasn’t really changed, while OCaml keeps incorporating new ideas into the language.

                                                                              I also think OCaml’s idea to determine infix operator precedence by its first character is better all-around in practice because it both prevents opening modules from messing up with precedence, and it also creates a convention that experienced users can follow without looking in third-party library docs every time.

                                                                              1. 3

                                                                                SML/NJ is probably the most adventurous mainstream implementation, for example adding support for nested functors that isn’t part of the spec. MLton has taken ideas from SML/NJ too over time outside of the spec. But Poly/ML has been pretty conservative.

                                                                                SML# is extremely practical though with an embedded SQL query language, a builtin JSON library, etc.

                                                                                It just historically wasn’t well marketed to an English audience. Actually in digging these examples up I’m amazed at how much they’ve made accessible now in English. I need to try SML# out…

                                                                            1. 1

                                                                              Maybe it is obvious (it’s been a while since I used Go), but how would it look with the upcoming generics?

                                                                              1. 2

                                                                                This is pretty out of touch, TBH. If you follow the Go dev issues, there’s a really obvious way to do sum types that falls out of how generics will work in Go 1.18. They haven’t talked about it publicly yet because they don’t want to promise it in case it turns out to be infeasible or create weird edge cases for whatever reason, but you can just look at how interface constraints work and remove an otherwise useless clause from the generics spec and tada you’ve got sum types.

                                                                                Long story short, in generic Go, if you want to write an add function for numeric types, you do something like

                                                                                type Number interface {
                                                                                  int, int8, uint8 //etc...
                                                                                }
                                                                                
                                                                                func add[T Number](a, b T) T {
                                                                                   return a + b
                                                                                }
                                                                                

                                                                                Well, you’ve got Number and it’s defined as being an int or an int8, etc. Why can’t you use it a Number as a regular variable? Well because the spec says you can’t because they haven’t worked out all the implications yet. But if you could it would be sum type. So the obvious thing to do is just to lift that restriction and let you use constraint types as regular interfaces.

                                                                                1. 3

                                                                                  Lifting the restriction of using constraints as regular values would be great, and it’s something I wish to see pursued, but they wouldn’t be sum types, they would be union types.

                                                                                  1. 1

                                                                                    Please give us union types. I can’t count how often sum types lead to more complicated code and less composability. Try representing errors or events as sum types and you’ll quickly run into components having to wrap other components‘ types and getting ugly nesting (edit: or lots of copy-paste). With union types it’s trivial to add more cases or handle one case and pass the rest on. I never understood why people want sum types.

                                                                                    1. 3

                                                                                      I agree that union types are great and are usually what people need, and it’s a shame that so few languages have them.

                                                                                      As for why are sum types more popular, sum types give rise to a richer algebra, which apart from its intrinsic quality means you can have type-inference, and they can also be somewhat combined with subtyping while retaining type erasure. This is much harder (impossible?) with union types.

                                                                                      1. 1

                                                                                        I wasn’t familiar with this distinction. Is this a good intro? https://blog.waleedkhan.name/union-vs-sum-types/

                                                                                        1. 3

                                                                                          It’s a pretty good tutorial that shows the practical differences in usage, although the mathematical treatment is somewhat fast and loose.

                                                                                          If you are interested in the rigorous treatment of union types in the context of type theory (as opposed to naive set theory, like the above article), the canonical texts are a collection of papers by Benjamin C. Pierce.

                                                                                        2. 1

                                                                                          Definitely, union types are much harder for the type system - though the recently released Scala 3 has them, too (apart from TypeScript), so I wouldn’t say it’s impossible. Maybe with structural typing you have other difficulties. I still believe it’s possible, even if one has to redefine the problem.

                                                                                          But what I meant was why “end-developers” (as opposed to compiler writers) often want sum types even if they’ve heard about union types. It’s probably more about showing with good examples how one would use union types and why they’re strictly better than sum types, i.e. without any disadvantages.

                                                                                        3. 1

                                                                                          I think the main downside of union types is ambiguity. For the example of:

                                                                                          type Either[A, B any] union {
                                                                                            a A
                                                                                            b B
                                                                                          }
                                                                                          
                                                                                          func analyse[A, B any](it: Either[A, B]) {
                                                                                          switch it.(type):
                                                                                            case A: fmt.Println("A")
                                                                                            case B: fmt.Println("B")
                                                                                          }
                                                                                          
                                                                                          func widget() {
                                                                                           val thing Either[int, int] = Either { b: 42 }
                                                                                           analyse(thing)
                                                                                          }
                                                                                          

                                                                                          What should it print? If Either[int, int] is represented as a plain union type, which relies on Go’s run type type information (not to mention boxing) to tell which case it represents, because both cases are int, there’s no way to tell which case it was originally.

                                                                                          Not to mention fun edge cases around interface types that might well overlap. Just imagine we have a plain interface for Number, and we want to have a function that returns a Result[Number, error]. Let’s also imagine we’re working with floating point numbers, and someone who loves chaos decides that because floats include a value for “Not a Number”, it should implement error.

                                                                                          There’s also plenty of practical experience writing industry software with sum types, in everything from O’Caml, Rust, and even Typescript.

                                                                                          … and you’ll quickly run into components having to wrap other components‘ types and getting ugly nesting

                                                                                          I’ve not seen that myself–do you have an example?

                                                                                          1. 3

                                                                                            Your example is not the best way to use union types. Union types can model sum types: An Either consists of Left and Right (or A and B in your case), so define those two, define typedef Either = Left | Right and put your ints wrapped into those two - exactly like you would with sum types.

                                                                                            The point of union types is not to leave off all tags/constructors (Left and Right). The point is to not have nominal typing over the union (the name Either), but structural typing (what it stands for: Left | Right) and be allowed to arbitrarily add more cases like Either | int (equivalent to Left | Right | int).

                                                                                            So, the Either just becomes an alias for Left | Right. In contrast, with sum types the Either becomes a island that can’t be combined into Either | int. Instead you have to double-wrap Either[Either[X, Y], int] or convert that to Triple[X, Y, int] and if you later want to remove the Y you again convert that to Either[X, int]. Horrible.

                                                                                            I see this happen in practice e.g. under Android where people often model the ViewModel’s states and events as sum types (sealed classes) and then when trying to subclass and add an extra event type or extra state you’re in a copy-paste or wrapping mess. This is why I’ve turned to interfaces for modeling events because then you can at least use multiple inheritance to have set-like composition. It’s not quite as good as union types, but so much better than sum types and provides the same safety guarantees (exhaustive checks) because you’re forced to implement the whole interface (each event being one method) and can’t forget anything.

                                                                                            1. 1

                                                                                              Ah, thanks, I see what you’re getting at – yeah, having being able to represent an open set of cases is definitely useful in a lot of cases. And if you had families of related types (eg: all events except mouse related ones, or simplifying special forms in a compiler) that would be a real pain. Have you seen O’Caml’s Polymorphic Variants? They’re used quite heavily in the GUI libraries (Tk, last time I checked) for one, and I’d definitely love to see them more widely adopted.

                                                                                              1. 2

                                                                                                Polymorphic variants still require tags. That’s bad.

                                                                                                Tags are optional in union types, which allows for more ergonomic APIs e.g. when your function want to accept string | List<string>. This way you can pass around values without unpacking/repacking in different tags to translate between different worlds just because you were forced to use tags even where it doesn’t matter.

                                                                                                With tags you have a function taking Either[string, int] and a different function taking Either[int, string] and they’re just incompatible without translation. With union types you have just int | string and string | int and there’s no translation needed. Not even when forwarding the value to a function taking string | int | bool. It’s still a compatible type.

                                                                                                With enforced tags you’re forced to over-categorize and over-specify the problem which breaks composability. Left and Right are just meaningless tags with nonsense names. How do you decide if your int goes into Left or Right? That’s just random. It’s much more meaningful to work with Error | int or Buy | Sell or string | List<string> or whatever meaningful names and sets of tagged and tagless types are best for describing the problem space. The whole Either = Left | Right type is just an ugly hack caused by a limitation of sum types (enforced tags).

                                                                                        4. 1

                                                                                          I hadn’t seen people distinguish “sum” vs. “union” before instead of just treating them as synonyms, but it’s a useful distinction. I think there’s a better than even chance that Go will adopt union types. I think there is virtually no chance that Go will adopt sum types given the current leadership. (I dunno, maybe if Google fires the Plan 9 guys, but not as long as they lead the project.)

                                                                                          1. 2

                                                                                            You’re not alone! It’s a common thing we have to clarify on the PL design discord I frequent. Sum types are better though of as ‘tagged unions’ and are pretty useful in their own right. But ‘unions’ are handy too.

                                                                                    1. 4

                                                                                      I wonder if there is any relationship with this approach of something like smallcheck? They have a paper, “SmallCheck and Lazy SmallCheck: automatic exhaustive testing for small values” which might be of interest? I’m not very familiar with it though, so I could be off-the mark on this connection.

                                                                                      1. 2

                                                                                        I came here to talk about SmallCheck, which was created to do exhaustive checking of inputs. Yes! This is the same thing. There are lots of fun publications in this area, though Rudy Matela wins the trophy with his PhD thesis and many libraries.

                                                                                        Rudy wrote leancheck for intelligent enumerative testing, fitspec to find missing properties or overspecified properties, speculate to discover properties, and extrapolate for finding generalized counter-examples to a property.

                                                                                        But wait, there’s more! If you get as excited about property based testing as I do, you will enjoy this PhD thesis.

                                                                                        1. 1

                                                                                          Anyone know if there’s an equivalent library in Rust?

                                                                                          1. 1

                                                                                            Let me google that for myself: https://github.com/blt/smallcheck

                                                                                        2. 1

                                                                                          It’s related, but not in a fundamental way. Both the post and the paper are about exhaustively enumerating small cases. SmallCheck takes a type-driven approach, while Gen focuses on values (you don’t generate lists and then filter them to be permutations, you just directly generate permutations).

                                                                                          The other paper linked in the discussion, Capturing the Future by Replaying the Past, captures the fundamentals. Gen is essentially what is described in “WARM-UP: REPLAY-BASED NONDETERMINISM”.

                                                                                        1. 11

                                                                                          Note: the OCaml Foundation funded John Whitington to write a new “getting started” page/tutorial for ocaml.org, it’s online at Up and Running with OCaml, and I think it’s doing a good job at getting this sort of information to newcomers. (Of course it will need maintenance as practices evolve.)

                                                                                          1. 3

                                                                                            It’s a bit confusing the way it uses two different conventions for commands:

                                                                                            # environment setup
                                                                                            opam init
                                                                                            eval `opam env`
                                                                                            
                                                                                            $ which ocaml
                                                                                            /Users/frank/.opam/4.11.1/bin/ocaml
                                                                                            

                                                                                            Especially give the long-standing convention that lines starting with # are to be run as root.

                                                                                            1. 1

                                                                                              I’m not sure that is as long-standing (or maybe better described as “as well-known”) as you think. I’ve not seen that in a while and then only rarely.

                                                                                              You are right about the inconsistent part though

                                                                                            2. 1

                                                                                              Nice! Seems to miss out the opam switch link thing though? This is something that I’d probably need as a beginner… I’ve found switches really frustrating to deal with in the past (having to manually set them when switching projects), and only recently learned that this is what I should have been using instead!

                                                                                            1. 6

                                                                                              I recently watched a talk by Bryan Cantrill called Rust, Wright’s Law, and the Future of Low-Latency Systems, here… https://www.youtube.com/watch?v=cuvp-e4ztC0

                                                                                              and was impressed about the implications of writing memory safe kernel, in my amateur perspective, it sounded like an OS written in Rust would be orders of magnitude safer than even the safest OS written in C today. ( Safe in terms of potential exploitability of the kernel and kernel drivers )

                                                                                              Does anyone with more knowledge than I care to chime in with their thoughts on an OS purely written in Rust vs C, as well as the ambition of this OS called Nuta/Kerla?

                                                                                              1. 8

                                                                                                There’s also SeL4, which is proven-safe.

                                                                                                1. 9

                                                                                                  seL4 is seriously impressive! You kind of have to be careful when saying stuff like ‘proven safe’ (more info on the proof’s assumption here and in the FAQs, but it’s definitely impressive, and goes beyond just verifying that the implementation matches memory safety properties.

                                                                                                  1. 3

                                                                                                    Well sure, there are things not covered in the proof, but stuff that is, to the best of what we know, it’s actually correct.

                                                                                              1. 18

                                                                                                Pattern matching has been available in functional programming languages for decades now, it was introduced in the 70s. (Logic programming languages expose even more expressive forms, at higher runtime cost.) It obviously improves readability of code manipulating symbolic expressions/trees, and there is a lot of code like this. I find it surprising that in the 2020s there are still people wondering whether “the feature provides enough value to justify its complexity”.

                                                                                                (The fact that Python did without for so long was rather a sign of closed-mindedness of its designer subgroup. The same applies, in my opinion, to languages (including Python, Go, etc.) that still don’t have proper support for disjoint union types / variants / sums / sealed case classes.)

                                                                                                1. 45

                                                                                                  Pretty much every feature that has ever been added to every language ever is useful in some way. You can leave a comment like this on almost any feature that a language may not want to implement for one reason or the other.

                                                                                                  1. 14

                                                                                                    I think it makes more sense in statically typed languages, especially functional ones. That said, languages make different choices. For me, Python has always been about simplicity and readability, and as I’ve tried to show in the article, at least in Python, structural pattern matching is only useful in a relatively few cases. But it’s also a question of taste: I really value the simplicity of the Go language (and C before it), and don’t mind a little bit of verbosity if it makes things clearer and simpler. I did some Scala for a while, and I can see how people like the “power” of it, but the learning curve of its type system was very steep, and there were so many different ways to do things (not to mention the compiler was very slow, partly because of the very complex type system).

                                                                                                    1. 22

                                                                                                      For the record, pattern-matching was developed mostly in dynamically-typed languages before being adopted in statically-typed languages, and it works just as well in a dynamically-typed world. (In the ML-family world, sum types and pattern-matching were introduced by Hope, an experimental dynamically-typed language; in the logic world, they are basic constructs of Prolog, which is also dynamically-typed – although some more-typed dialects exist.)

                                                                                                      as I’ve tried to show in the article, at least in Python, structural pattern matching is only useful in a relatively few cases

                                                                                                      Out of the 4 cases you describe in the tutorial, I believe your description of two of them is overly advantageous to if..elif:

                                                                                                      • In the match event.get() case, the example you show is a variation of the original example (the longer of the three such examples in the tutorial), and the change you made makes it easier to write an equivalent if..elif version, because you integrated a case (from another version) that ignores all other Click() events. Without this case (as in the original tutorial example), rewriting with if..elif is harder, you need to duplicate the failure case.
                                                                                                      • In the eval_expr example, you consider the two versions as readable, but the pattern-version is much easier to maintain. Consider, for example, supporting operations with 4 or 5 parameters, or adding an extra parameter to an existing operator; it’s an easy change with the pattern-matching version, and requires boilerplate-y, non-local transformations with if..elif. These may be uncommon needs for standard mathematical operations, but they are very common when working with other domain-specific languages.
                                                                                                      1. 1

                                                                                                        the change you made makes it easier to write an equivalent if..elif version

                                                                                                        Sorry if it appeared that way – that was certainly not my intention. I’m not quite sure what you mean, though. The first/original event example in the tutorial handles all click events with no filtering using the same code path, so it’s even simpler to convert. I added the Button.LEFT filtering from a subsequent example to give it a bit more interest so it wasn’t quite so simple. I might be missing something, though.

                                                                                                        In the eval_expr example, you consider the two versions as readable, but the pattern-version is much easier to maintain. Consider, for example, supporting operations with 4 or 5 parameters, or adding an extra parameter to an existing operator;

                                                                                                        I think those examples are very hypothetical – as you indicate, binary and unary operators aren’t suddenly going to support 4 or 5 parameters. A new operation might, but that’s okay. The only line that’s slightly repetitive is the “attribute unpacking”: w, x, y, z = expr.w, expr.x, expr.y, expr.z.

                                                                                                        These may be uncommon needs for standard mathematical operations, but they are very common when working with other domain-specific languages.

                                                                                                        You’re right, and that’s part of my point. Python isn’t used for implementing compilers or interpreters all that often. That’s where I’m coming from when I ask, “does the feature provide enough value to justify the complexity?” If 90% of Python developers will only rarely use this complex feature, does it make sense to add it to the language?

                                                                                                        1. 3

                                                                                                          that was certainly not my intention.

                                                                                                          To be clear, I’m not suggesting that the change was intentional or sneaky, I’m just pointing out that the translation would be more subtle.

                                                                                                          The first/original event example does not ignore “all other Click events” (there is no Click() case), and therefore an accurate if..elif translation would have to do things differently if there is no position field or if it’s not a pair, namely it would have to fall back to the ValueError case.

                                                                                                          You’re right, and that’s part of my point. Python isn’t used for implementing compilers or interpreters all that often.

                                                                                                          You don’t need to implement a compiler for C or Java, or anything people recognize as a programming language (or HTML or CSS, etc.), to be dealing with a domain-specific languages. Many problem domains contain pieces of data that are effectively expressions in some DSL, and recognizing this can very helpful to write programs in those domains – if the language supports the right features to make this convenient. For example:

                                                                                                          • to start with the obvious, many programs start by interpreting some configuration file to influence their behavior; many programs have simple needs well-served by linear formats, but many programs (eg. cron jobs, etc.) require more elaborate configurations that are DSL-like. Even if the configuration is written in some standard format (INI, Yaml, etc.) – so parsing can be delegated to a library – the programmer will still write code to interpret or analyze the configuration data.
                                                                                                          • more gnerally, “structured data formats” are often DSL-shaped; ingesting structured data is something we do super-often in programs
                                                                                                          • programs that offer a “query” capability typically provide a small language to express those queries
                                                                                                          • events in an event loop typically form a small language
                                                                                                      2. 14

                                                                                                        I think it makes more sense in statically typed languages, especially functional ones.

                                                                                                        In addition to the earlier ones gasche mentioned (it’s important to remember this history), it’s used to pervasively in Erlang, and later Elixir. Clojure has core.match, Racket has match, as does Guile. It’s now in Ruby as well!

                                                                                                        1. 3

                                                                                                          Thanks! I didn’t know that. I have used pattern matching in statically typed language (mostly Scala), and had seen it in the likes of Haskell and OCaml, so I’d incorrectly assumed it was mainly a statically-typed language thing.

                                                                                                          1. 1

                                                                                                            It is an important feature of OCaml.

                                                                                                            1. 3

                                                                                                              I am aware - was focusing on dynamically typed languages.

                                                                                                          2. 7

                                                                                                            For me, it is the combination of algebraic data types + pattern matching + compile time exhaustiveness checking that is the real game changer. With just 1 out of 3, pattern matching in Python is much less compelling.

                                                                                                            1. 1

                                                                                                              I agree. I wonder if they plan to add exhaustiveness checking to mypy. The way the PEP is so no hold barred makes it seem like the goal was featurefulness and not an attempt to support exhaustiveness checking.

                                                                                                              1. 2

                                                                                                                I wonder if they plan to add exhaustiveness checking to mypy.

                                                                                                                I don’t think that’s possible in the general case. If I understand the PEP correctly, __match_args__ may be a @property getter method, which could read the contents of a file, or perform a network request, etc.

                                                                                                          3. 11

                                                                                                            I find it surprising that in the 2020s there are still people wondering whether “the feature provides enough value to justify its complexity”.

                                                                                                            I find it surprising that people find this surprising.

                                                                                                            Adding features like pattern matching isn’t trivial, and adding it too hastily can backfire in the long term; especially for an established language like Python. As such I would prefer a language take their time, rather than slapping things on because somebody on the internet said it was a good idea.

                                                                                                            1. 3

                                                                                                              That’s always been the Scheme philosophy:

                                                                                                              Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary.

                                                                                                              And indeed, this pays off: in the Scheme world, there’s been a match package floating around for a long time, implemented simply as a macro. No changes to the core language needed.

                                                                                                              1. 4

                                                                                                                No changes to the core language needed.

                                                                                                                I’m sure you recognize that this situation does not translate to other languages like in this case Python. Implementing it as a macro is just not feasible. And even in Scheme the usage of match macros is rather low. This can be because it is not that useful, but also might be because of the hurdle of adding dependencies is not worth the payoff. Once a feature is integrated in a language, its usage “costs” nothing, thus the value proposition when writing code can be quite different.

                                                                                                                1. 7

                                                                                                                  This is rather unrelated to the overall discussion, but as a user of the match macros in Scheme, I must say that I find the lack of integration into the base forms slightly annoying. You cannot pattern-match on a let or lambda, you have to use match-let and match-lambda, define/match (the latter only in Racket I think), etc. This makes reaching for pattern-matching feel heavier, and it may be a partial cause to their comparatively lower usage. ML-family languages generalize all binding positions to accept patterns, which is very nice to decompose records for example (or other single-case data structures). I wish Scheme dialects would embrace this generalization, but they haven’t for now – at least not Racket or Clojure.

                                                                                                                  1. 2

                                                                                                                    In the case of Clojure while it doesn’t have pattern matching built-in, it does have quite comprehensive destructuring forms (like nested matching in maps, with rather elaborate mechanisms) that works in all binding positions.

                                                                                                                    1. 2

                                                                                                                      Nice! I suppose (from your post above) that pattern-matching is somehow “integrated” in the Clojure implementation, rather than just being part of the base macro layer that all users see.

                                                                                                                      1. 2

                                                                                                                        I think the case is that Clojure core special forms support it (I suppose the implementation itself is here and called “binding-forms”, which is then used by let, fn and loop which user defined macros often end up expanding to). Thus it is somewhat under the base layer that people use.

                                                                                                                        But bear in mind this is destructuring, in a more general manner than what Python 2.x already supported, not pattern matching. It also tends to get messy with deep destructuring, but the same can be said of deep pattern matches through multiple layers of constructors.

                                                                                                            2. 8

                                                                                                              I agree about pattern matching and Python in general. It’s depressing how many features have died in python-ideas because it takes more than a few seconds for an established programmer to grok them. Function composition comes to mind.

                                                                                                              But I think Python might be too complicated for pattern matching. The mechanism they’ve settled on is pretty gnarly. I wrote a thing for pattern matching regexps to see how it’d turn out (admittedly against an early version of the PEP; I haven’t checked it against the current state) and I think the results speak for themselves.

                                                                                                              1. 6

                                                                                                                But I think Python might be too complicated for pattern matching. The mechanism they’ve settled on is pretty gnarly.

                                                                                                                I mostly agree. I generally like pattern matching and have been excited about this feature, but am still feeling out exactly when I’ll use it and how it lines up with my intuition.

                                                                                                                The part that does feel very Pythonic is that destructuring/unpacking is already pretty pervasive in Python. Not only for basic assignments, but also integrated into control flow constructs. For example, it’s idiomatic to do something like:

                                                                                                                for key, val in some_dictionary.items():
                                                                                                                    # ...
                                                                                                                

                                                                                                                Rather than:

                                                                                                                for item in some_dictionary.items():
                                                                                                                    key, val = item
                                                                                                                    # ...
                                                                                                                

                                                                                                                Or something even worse, like explicit item[0] and item[1]. So the lack of a conditional-with-destructuring, the way we already have foreach-with-destructuring, did seem like a real gap to me, making you have to write the moral equivalent of code that looks more like the 2nd case than the 1st. That hole is now filled by pattern matching. But I agree there are pitfalls around how all these features interact.

                                                                                                              2. 2
                                                                                                                for i, (k, v) in enumerate(d.items(), 1): pass
                                                                                                                

                                                                                                                looks like pattern matching to me

                                                                                                                1. 2

                                                                                                                  Go aims for simplicity of maintenance and deployment. It doesn’t “still don’t have those features”. The Go authors avoided them on purpose. If you want endless abstractions in Go, embedding Lisp is a possibilty: https://github.com/glycerine/zygomys

                                                                                                                  1. 5

                                                                                                                    Disjoint sums are a basic programming feature (it models data whose shape is “either this or that or that other thing”, which ubiquitous in the wild just like pairs/records/structs). It is not an “endless abstraction”, and it is perfectly compatible with maintenance and deployment. Go is a nice language in some respects, the runtime is excellent, the tooling is impressive, etc etc. But this is no rational excuse for the lack of some basic language features.

                                                                                                                    We are in the 2020s, there is no excuse for lacking support for sum types and/or pattern matching. Those features have been available for 30 years, their implementation is well-understood, they require no specific runtime support, and they are useful in basically all problem domains.

                                                                                                                    I’m not trying to bash a language and attract defensive reactions, but rather to discuss (with concrete examples) the fact that language designer’s mindsets can be influenced by some design cultures more than others, and as a result sometimes the design is held back by a lack of interest for things they are unfamiliar with. Not everyone is fortunate to be working with a deeply knowledgeable and curious language designer, such as Graydon Hoare; we need more such people in our language design teams. The default is for people to keep working on what they know; this sort of closed-ecosystem evolution can lead to beautiful ideas (some bits of Perl 6 for example are very nice!), but it can also hold back.

                                                                                                                    1. 3

                                                                                                                      But this is no rational excuse for the lack of some basic language features.

                                                                                                                      Yes there is. Everyone has a favorite feature, and if all of those are implemented, there would easily be feature bloat, long build times and projects with too many dependencies that depend on too many dependencies, like in C++.

                                                                                                                      In my opinion, the question is not if a language lacks a feature that someone wants or not, but if it’s usable for goals that people wish to achieve, and Go is clearly suitable for many goals.

                                                                                                                  2. 3

                                                                                                                    Ah yes, Python is famously closed-minded and hateful toward useful features. For example, they’d never adopt something like, say, list comprehensions. The language’s leaders are far too closed-minded, and dogmatically unwilling to ever consider superior ideas, to pick up something like that. Same for any sort of ability to work with lazy iterables, or do useful combinatoric work with them. That’s something that definitely will never be adopted into Python due to the closed-mindedness of its leaders. And don’t get me started on basic FP building blocks like map and folds. It’s well known that Guido hates them so much that they’re permanently forbidden from ever being in the language!

                                                                                                                    (the fact that Python is not Lisp was always unforgivable to many people; the fact that it is not Haskell has now apparently overtaken that on the list of irredeemable sins; yet somehow we Python programmers continue to get useful work done and shrug off the sneers and insults of our self-proclaimed betters much as we always have)

                                                                                                                    1. 25

                                                                                                                      It is well-documented that Guido Van Rossum planned to remove lambda from Python 3. (For the record, I agree that map and filter on lists are much less useful in presence of list comprehensions.) It is also well-documented that recursion is severely limited in Python, making many elegant definitions impractical.

                                                                                                                      Sure, Python adopted (in 2000 I believe?) list comprehensions from ABC (due to Guido working with the language in the 1980s), and a couple of library-definable iterators. I don’t think this contradicts my claim. New ideas came to the language since (generators, decorators), but it remains notable that the language seems to have resisted incorporating strong ideas from other languages. (More so than, say, Ruby, C#, Kotlin, etc.)

                                                                                                                      Meta: One aspect of your post that I find unpleasant is the tone. You speak of “sneers and insults”, but it is your post that is highly sarcastic and full of stray exagerations at this or that language community. I’m not interested in escalating in this direction.

                                                                                                                      1. 7

                                                                                                                        less useful in presence of list comprehension

                                                                                                                        I’m certainly biased, but I find Python’s list comprehension an abomination towards readability in comparison to higher-order pipelines or recursion. I’ve not personally coded Python in 8-9 years, but when I see examples, I feel like I need to put my head on upsidedown to understand it.

                                                                                                                        1. 6

                                                                                                                          It is also well-documented that recursion is severely limited in Python, making many elegant definitions impractical.

                                                                                                                          For a subjective definition of “elegant”. But this basically is just “Python is not Lisp” (or more specifically, “Python is not Scheme”). And that’s OK. Not every language has to have Scheme’s approach to programming, and Scheme’s history has shown that maybe it’s a good thing for other languages not to be Scheme, since Scheme has been badly held back by its community’s insistence that tail-recursive implementations of algorithms should be the only implementations of those algorithms.

                                                                                                                          You speak of “sneers and insults”, but it is your post that is highly sarcastic and full of stray exagerations at this or that language community.

                                                                                                                          Your original comment started from a place of assuming – and there really is no other way to read it! – that the programming patterns you care about are objectively superior to other patterns, that languages which do not adopt those patterns are inherently inferior, and that the only reason why a language would not adopt them is due to “closed-mindedness”. Nowhere in your comment is there room for the (ironically) open-minded possibility that someone else might look at patterns you personally subjectively love, evaluate them rationally, and come to a different conclusion than you did – rather, you assume that people who disagree with your stance must be doing so because of personal faults on their part.

                                                                                                                          And, well, like I said we’ve got decades of experience of people looking down their noses at Python and/or its core team + community for not becoming a copy of their preferred languages. Your comment really is just another instance of that.

                                                                                                                          1. 8

                                                                                                                            I’m not specifically pointing out the lack of tail-call optimization (TCO) in Python (which I think is unfortunate indeed; the main argument is that call stack matters, but it’s technically fully possible to preserve call stacks on the side with TC-optimizing implementations). Ignoring TCO for a minute, the main problem would be the fact that the CPython interpreter severely limits the call space (iirc it’s 1K calls by default; compare that to the 8Mb default on most Unix systems), making recursion mostly unusable in practice, except for logarithmic-space algorithms (balanced trees, etc.).

                                                                                                                            Scheme has been badly held back by its community’s insistence that tail-recursive implementations of algorithms should be the only implementations of those algorithms.

                                                                                                                            I’m not sure what you mean – that does not make any sense to me.

                                                                                                                            [you assume] that the programming patterns you care about are objectively superior to other patterns [..]

                                                                                                                            Well, I claimed

                                                                                                                            [pattern matching] obviously improves readability of code manipulating symbolic expressions/trees

                                                                                                                            and I stand by this rather modest claim, which I believe is an objective statement. In fact it is supported quite well by the blog post that this comment thread is about. (Pattern-matching combines very well with static typing, and it will be interesting to see what Python typers make of it; but its benefits are already evident in a dynamically-typed context.)

                                                                                                                            1. 4

                                                                                                                              and I stand by this rather modest claim, which I believe is an objective statement.

                                                                                                                              Nit: I don’t think you can have an objective statement of value.

                                                                                                                              1. 4

                                                                                                                                Again: your original comment admits of no other interpretation than that you do not believe anyone could rationally look at the feature you like and come to a different conclusion about it. Thus you had to resort to trying to find personal fault in anyone who did.

                                                                                                                                This does not indicate “closed-mindedness” on the part of others. They may prioritize things differently than you do. They may take different views of complexity and tradeoffs (which are the core of any new language-feature proposal) than you do. Or perhaps they simply do not like the feature as much as you do. But you were unwilling to allow for this — if someone didn’t agree with your stance it must be due to personal fault. You allowed for no other explanation.

                                                                                                                                That is a problem. And from someone who’s used to seeing that sort of attitude it will get you a dismissive “here we go again”. Which is exactly what you got.

                                                                                                                            2. 4

                                                                                                                              This is perhaps more of a feeling, but saying that Rust isn’t adopting features as quickly as Ruby seems a bit off. Static type adoption in the Python community has been quicker. async/await has been painful, but is being attempted. Stuff like generalized unpacking (and this!) is also shipping out!

                                                                                                                              Maybe it can be faster, but honestly Python probably has one of the lowest “funding amount relative to impact” of the modern languages which makes the whole project not be able to just get things done as quickly IMO.

                                                                                                                              Python is truly in a funny place, where many people loudly complain about it not adopting enough features, and many other loudly complain about it loudly adopting too many! It’s of course “different people have different opinions” but still funny to see all on the same page.

                                                                                                                              1. 3

                                                                                                                                It is well-documented that Guido Van Rossum planned to remove lambda from Python 3

                                                                                                                                Thank you for sharing that document. I think Guido was right: it’s not pythonic to map, nor to use lambdas in most cases.

                                                                                                                                Every feature is useful, but some ecosystems work better without certain features. I’m not sure where go’s generics fall on this spectrum, but I’m sure most proposed features for python move it away from it’s core competency, rather than augmenting a strong core.

                                                                                                                                1. 1

                                                                                                                                  We have previously discussed their tone problem. It comes from their political position within the Python ecosystem and they’re relatively blind to it. Just try to stay cool, I suppose?

                                                                                                                                  1. 6

                                                                                                                                    I really do recommend clicking through to that link, and seeing just what an unbelievably awful thing I said that the user above called out as “emblematic” of the “contempt” I display to Python users. Or the horrific ulterior motive I was found to have further down.

                                                                                                                                    Please, though, before clicking through, shield the eyes of children and anyone else who might be affected by seeing such content.

                                                                                                                                2. 5

                                                                                                                                  To pick one of my favorite examples, I talked to the author of PEP 498 after a presentation that they gave on f-strings, and asked why they did not add destructuring for f-strings, as well as whether they knew about customizeable template literals in ECMAScript, which trace their lineage through quasiliterals in E all the way back to quasiquotation in formal logic. The author knew of all of this history too, but told me that they were unable to convince CPython’s core developers to adopt any of the more advanced language features because they were not seen as useful.

                                                                                                                                  I think that this perspective is the one which might help you understand. Where you see one new feature in PEP 498, I see three missing subfeatures. Where you see itertools as a successful borrowing of many different ideas from many different languages, I see a failure to embrace the arrays and tacit programming of APL and K, and a lack of pattern-matching and custom operators compared to Haskell and SML.

                                                                                                                                3. 1

                                                                                                                                  I think the issue is more about pattern matching being a late addition to Python, which means there will be lots of code floating around that isn’t using match expressions. Since it’s not realistic to expect this code to be ported, the old style if … elif will continue to live on. All of this adds up to a larger language surface area, which makes tool support, learning and consistency more difficult.

                                                                                                                                  I’m not really a big fan of this “pile of features” style of language design - if you add something I’d prefer if something got taken away as well. Otherwise you’ll end up with something like Perl 5

                                                                                                                                1. 5

                                                                                                                                  Excellent and fascinating series! It’s far too seldom you get to see people seriously involved with a large programming language’s design do this sort of retrospective. It’s also somewhat enlightening how many of these are warts for compatibility with C/ObjC API design, for example this one.

                                                                                                                                  Suggest plt tag as well, maybe?

                                                                                                                                  1. 1

                                                                                                                                    It’s not really theory though. It’s more about language design decisions?

                                                                                                                                  1. 7

                                                                                                                                    This distinction is very similar to the one made in this article, except it splits the module manager into two subcategories:

                                                                                                                                    • Language package managers, e.g. go get, which manage packages for a particular language, globally.
                                                                                                                                    • Project dependency managers, e.g. cargo, which manage packages for a particular language and a particular local project.

                                                                                                                                    To be fair, many package managers play both roles by allowing you to install a package locally or globally. I tend to think that global package installation is an anti-pattern, and the use cases for it are better served by improving the UX around setting up local projects. For example, nix-shell makes it extremely easy to create an ad-hoc environment containing some set of packages, and as a result there’s rarely a need to use nix-env.

                                                                                                                                    1. 13

                                                                                                                                      I tend to think that global package installation is an anti-pattern

                                                                                                                                      From experience, I agree with this very strongly. Any “how to do X” tutorial that encourages you to run something like “sudo pio install …” or “sudo gem install …” is immediately very suspect. It’s such a pain in the hindquarters to cope with the mess that ends up accruing.

                                                                                                                                      1. 3

                                                                                                                                        Honestly I’m surprised to read that this still exists in newer languages.

                                                                                                                                        Back when I was hacking on Rubygems in 2008 or so it was very clear that this was a mistake, and tools like isolate and bundler were having to backport the project-local model onto an ecosystem which had spent over a decade building around a flawed global install model, and it was really ugly. The idea that people would repeat those same mistakes without the excuse of a legacy ecosystem is somewhat boggling.

                                                                                                                                        1. 3

                                                                                                                                          Gah, this is one thing that frustrates me so much about OPAM. Keeping things scoped to a a specific project is not the default, global installations of libraries is more prominently encouraged in the docs, and you need to figure out how to use a complicated, stateful workflow using global ‘switches’ to avoid getting into trouble.

                                                                                                                                          1. 3

                                                                                                                                            One big exception… sudo gem install bundler ;)

                                                                                                                                            (Though in prod I do actually find it easier/more comfortable to just use Bundler from APT.)

                                                                                                                                        1. 20

                                                                                                                                          Some rough thoughts: I guess I am a bit curmudgeonly about the term “crypto” getting taken over by “cryptocurrency” after several decades of use, it’s a bit galling to cede the term. Though at some point, yeah, it’s a practical reality that’s not worth confusing people over. We’ve seen cryptocurrency promotion get tagged both with crypto and merkle-trees so I’m not really sure expanding to cryptocurrency would change anything; I think all technical communities are going to continue to suffer from promotion for at least a few more years.

                                                                                                                                          Worth discussing, though, maybe someone has more pros or cons? Maybe it’s worth just trying for a year?

                                                                                                                                          1. 7

                                                                                                                                            I’ve renamed the crypto tag to cryptography. There’s strong support for it, it’s cheap, and implicitly ceding a little ground to scammers is a pretty negligible downside.

                                                                                                                                            1. 6

                                                                                                                                              I think the realistic view is that it has become ambiguous and would be regardless of what the new referent that “takes over” is. What are the cons in expanding it to “cryptography” exactly? I don’t see any, only pros.

                                                                                                                                              1. 2

                                                                                                                                                I would rather not have any bitcoin related stuff on the site if possible.

                                                                                                                                                1. 2

                                                                                                                                                  Add a cryptocurrency tag, then when used, show a warning like with banned domains and don’t submit the url. Sort of like a fishbait tag. You can add one for business news as well. Preventive moderation of sorts

                                                                                                                                                  1. 1

                                                                                                                                                    Tbh, I’m pretty appreciative of the similar stance for the ml tag.

                                                                                                                                                    On another note, I do kind of wish there was a more general programming-languages or pl-design tag, as a lot of things that get tagged plt aren’t really PL theory, more about PL design etc?

                                                                                                                                                    1. 1

                                                                                                                                                      What do you think about this plan:

                                                                                                                                                      1. Add suffix for cryptography
                                                                                                                                                      2. Add cryptocurrency
                                                                                                                                                      3. Hotness mod cryptocurrency of -0.5
                                                                                                                                                      1. 1

                                                                                                                                                        There are technical aspects of cryptocurrencies that are technical, on-topic, insightful (presumably? I don’t read them), and I don’t think it would be fair to apply a hotness downgrade to them.

                                                                                                                                                        The problem is cryptocurrency promotion/spam, which isn’t on-topic at all. Lobsters also gets a fair amount of spam on other topics like cloud wazamabobs we don’t downgrade the entire topic for it.

                                                                                                                                                        1. 3

                                                                                                                                                          If it’s described as a cryptocurrency, then it’s likely already moved into the scam category. There are interesting uses for verifiable append-only ledgers. There are interesting use cases for distributed consensus algorithms. These have nothing to do with cryptocurrencies other than the underlying technology.

                                                                                                                                                          1. 1

                                                                                                                                                            I do happen to agree that at a first approximation all cryptocurrencies are scams, but even a more charitable reading would lead them to fall under the umbrella of “business news” and thus off-topic anyway.

                                                                                                                                                          2. 1

                                                                                                                                                            I thought we use merkle-trees for the technical aspects?

                                                                                                                                                            A downgrade is just a downgrade; e.g. culture tags are downgraded. If the link is good, votes will still rise it to the top.

                                                                                                                                                        2. 1

                                                                                                                                                          I am in favor of expanding the existing tag to cryptography and keeping the implicit ban on cryptocurrency promotion by only keeping the relevant merkle-trees tag.

                                                                                                                                                        1. 5

                                                                                                                                                          Code is at https://github.com/jamii/imp and https://github.com/jamii/focus.

                                                                                                                                                          Sorry about the volume :S

                                                                                                                                                          1. 2

                                                                                                                                                            Cool work! Love interactive stuff like this!

                                                                                                                                                            Is there any relationship to the other IMP languages (eg. Edinburgh IMP, or the NSA’s IMP, or the little imperative languages found in Software Foundations and Concrete Semantics), or is this something completely different?

                                                                                                                                                            1. 7

                                                                                                                                                              Totally unrelated. The name came from reading a paper about personifying compilers to make them more friendly. I felt like programming is less of a helpful assistant kind of thing and more of a folkloric deal-with-the-devil kind of thing, where you get exactly what you asked for.

                                                                                                                                                              1. 1

                                                                                                                                                                Ahh, so it’s meant to be ‘imp’ as in the magical creature, as opposed to being short for ‘imperative language’? That’s what I was thinking and then I got a bit confused when I stuff about relational programming, haha.

                                                                                                                                                          1. 5

                                                                                                                                                            Over time you slowly build out a list of good “node papers” (mostly literature reviews) and useful terms to speed up this process, but it’s always gonna be super time consuming.

                                                                                                                                                            Although it doesn’t really replace the manual thoroughness, I’ve found Connected Papers and Citation Tree to be handy tools for augmenting this process – if only to get a quick lay of the land!

                                                                                                                                                            1. 45

                                                                                                                                                              PDFs are not mobile friendly. That’s already a strong reason for HTML.

                                                                                                                                                              I know the author claims that PDF reflow allegedly solves this issue, but I’ve yet to find a reader that supports it.

                                                                                                                                                              1. 21

                                                                                                                                                                Strongly agree. And what about accessibility? PDFs are awkward for people who need large text accommodations. And I don’t know how amenable they are to screen readers nowadays: originally it was quite difficult to recover the text from a PDF because each line of text was an independent element with no clear indication of how they connect to each other.

                                                                                                                                                                1. 2

                                                                                                                                                                  These days you can at least add accessibility annotations to PDFs that support devices like screen readers. Not sure how many readers support this stuff though.

                                                                                                                                                                2. 2

                                                                                                                                                                  KOReader does a halfway decent job, but I agree it’s not nearly good enough to obviate HTML.