1. 2

    I appreciate most of the arguments, but the counter-point around security is missing the spot. For distributions, it is far easier to apply a patch on a single package. Rebuilding or not is not really the difficulty. Now, if many applications are bundling/pinning specific versions, distributions need to patch each version. Some of these versions may be very old and the patch may be more difficult to apply. This is a lot of work. Distributions cannot just bump the dependency as it goes against the stability promise and introduce bugs and changes. Distributions have to support what they ship for around 5 years (because many users use distributions for this exact purpose) while developers usually like to support things for a few months.

    Unfortunately both sides do not want to move an inch. When packaging for Debian, I would appreciate being able to bundle dependencies instead of packaging each single dependency, but there must be some ways to guarantee we are not just multiplying the amount of work we need to provide in the future. However, this is not new. Even with C, many devs do not like distributions freezing their software for 5 years.

    1. 11

      The real “issue” from the distro perspective is that they’re now trying to package ecosystems that work completely differently than the stuff they’re used to packaging, and specifically ecosystems where the build process is tied tightly to the language’s own tooling, rather than the distro’s tooling.

      This is why people keep talking about distros being stuck on twenty-years-ago’s way of building software. Or, really, stuck on C’s way of building software. C doesn’t come with a compiler, or a build configuration tool, or a standard way to specify dependencies and make sure they’re present and available either during build or at runtime. C is more or less just a spec for what the code ought to do when it’s run. So distros, and everybody else doing development in C, have come up with their own implementations for all of that, and grown used to that way of doing things.

      More recently-developed languages, though, treat a compiler and build tool and dependencies/packaging as a basic requirement, and tightly integrate to their standard tooling. Which then means that the distro’s existing and allegedly language-agnostic tooling doesn’t work, or at least doesn’t work as well, and may not have been as language-agnostic as they hoped.

      Which is why so many of the arguments in these threads have been red herrings. It’s not that “what dependencies does this have” is some mysterious unanswerable question in Rust, it’s that the answer to the question is available in a toolchain that isn’t the one the distro wants to use. It’s not that “rebuild the stuff that had the vulnerable dependency” is some nightmare of tracking down impossible-to-know information and hoping you caught and patched everything, it’s that it’s meant to be done using a toolchain and a build approach that isn’t the one the distro wants to use.

      And there’s not really a distro-friendly thing the upstream developers can do, because each distro has its own separate preferred way of doing this stuff, so that’s basically pushing the combinatorial nightmare upstream and saying “take the information you already provide in your language’s standard toolchain, and also provide and maintain one additional copy of it for each distro, in that distro’s preferred format”. The only solution is for the distros to evolve their tooling to be able to handle these languages, because the build approach used in Rust, Go, etc. isn’t going away anytime soon, and in fact is likely to become more popular over time.

      1. 5

        The only solution is for the distros to evolve their tooling to be able to handle these languages

        The nixpkgs community has been doing this a lot. Their response to the existence of other build tools has been to write things like bundix, cabal2nix and cargo2nix. IIRC people (used to) use cabal2nix to make the whole of hackage usable in nixpkgs?

        From the outside it looks like the nix community’s culture emphasizes a strategy of enforcing policy by making automations whose outputs follow it.

        1. 4

          Or, really, stuck on C’s way of building software.

          I think it’s at least slightly more nuanced than that. Most Linux distributions, in particular, have been handling Perl modules since their earliest days. Debian/Ubuntu use them fairly extensively even in base system software. Perl has its own language ecosystem for building modules, distributing them in CPAN, etc., yet distros have generally been able to bundle Perl modules and their dependencies into their own package system. End users are of course free to use Perl’s own CPAN tooling, but if you apt-get install something on Debian that uses Perl, it doesn’t go that route, and instead pulls in various libxxx-perl packages. I don’t know enough of the details to know why Rust is proving more intractable than Perl though.

          1. 6

            I don’t know enough of the details to know why Rust is proving more intractable than Perl though

            There is a big difference between C, Perl, Python on the one side and Rust on the other.

            The former have a concept of “search path”: there’s a global namespace where all libraries live. That’s include path for C, PYTHONPATH for Python and @INC (?) for Perl. To install a library, you put it into some blessed directory on the file system, and it becomes globally available. The corollary here is that every one is using the same version of library. If you try to install two different versions, you’ll get a name conflict.

            Rust doesn’t have global search path / global namespace. “Installing rust library” is not a thing. Instead, when you build a piece of Rust software, you need to explicitly specify path for every dependency. Naturally, doing this “by hand” is hard, so the build system (Cargo) has a lot of machinery for wiring a set of interdependent crates together.

            1. 2

              there’s a global namespace where all libraries live

              Yes, this is one of the biggest differences. Python, Perl, etc. come out of the Unix-y C-based tradition of not having a concept of an “application” you run or a “project” you work on, but instead only of a library search path that’s assumed to be shared by all programs in that language, or at best per-user unique so that one user’s set of libraries doesn’t pollute everyone else’s.

              Python has trended away from this and toward isolating each application/project – that’s the point of virtual environments – but does so by just creating a per-virtualenv search path.

              More recently-developed languages like Rust have avoided ever using the shared-search-path approach in the first place, and instead isolate everything by default, with its own project-local copies of all dependencies.

              (the amount of code generation/specialization that happens at compile time in Rust for things like generics is a separate issue, but one that distros – with their ability to already handle C++ – should in theory not have trouble with)

          2. 4

            This is why people keep talking about distros being stuck on twenty-years-ago’s way of building software. Or, really, stuck on C’s way of building software. C doesn’t come with a compiler, or a build configuration tool, or a standard way to specify dependencies and make sure they’re present and available either during build or at runtime. C is more or less just a spec for what the code ought to do when it’s run. So distros, and everybody else doing development in C, have come up with their own implementations for all of that, and grown used to that way of doing things.

            More than that, I’d say they’re language-specific package managers around autotools+C.

        1. 8

          Getting LSP integrated pervasively in Emacs in a way that it reliably just works and performs well out of the box

          I don’t get how this should work, when setting up LSP servers is now the issue. If you want to use rust-analyser, you have to download rust-nightly. If you want to use a Java LSP server, you have to do class-path magic. If you want to use a Python LSP server, you’ll have to let pip download every single python library again to get the right versions. If you want to compile a Haskell LSP server, you’ll need to buy some more RAM.

          But if this would be solved, and all I have to do is run “sudo package-manager install java-lsp-server”, then it would be a considerable advancement. The best one can do, as I understand is planned for 28.1, is to get Eglot into a working and stable state, so that it’s ready for when the server implementations have matured.

          Having a wizard showing up in new Emacs installations might be a great low-hanging fruit way of making Emacs more accessible.

          I have discussed this on the mailing list, but haven’t ever had the time to implement this myself (yet). What Emacs needs is a wizard language, that can be used to teach Emacs itself, but also any package (just like ert is used for tests). Most proposals haven’t been too pretty though.

          Take a look at the screenshots of these Neovim GUIs:

          I don’t see any notable difference, tbh. What I would get is better integration into window managers (opening downloaded files in existing emacs sessions), but I have never understood what neovim actually achieves by splitting UI and backend. Sure, it’s a good design decision in principle, but considering the complexity of Emacs display engine, I’m not sure if it is worth it.

          It is very easy to either freeze Emacs or cause it to run very slowly. Multiple times a day I have to hit C-g incessantly to bring it back from being frozen. When that fails, I am sometimes able to get it back with pkill -SIGUSR2 Emacs. At least once per week I have to pkill -9 Emacs because it turned completely unresponsive.

          I used to have these problems, but in my experience this comes from fighting Emacs instead of cooperating with it. Using unnecessary packages, weird themes and wanting popups everywhere will simply slow stuff down. The main parallelism in Emacs is done by starting external processes, since Emacs is more of a Unix shell than Lisp machine.

          It would be great to have more and more of these influencing and being incorporated to the Emacs Lisp standard library and made to be very performant.

          Please not, dash and s are quite annoying and don’t always lend themselves to idiomatic Elisp (I understand they are influenced by Clojure, which has a difference heritage). seq.el and map.el are nice, but since they are based on generic functions, they are often slower.

          It would be great to move to a forge style of contribution.

          As long as it doesn’t require new accounts, and everybody is free to participate in the conversations, ok, but I’ve only seen this implemented by source-hut, and none of the GitHub clones that have spoiled so many people.

          1. 6

            Emacs is more of a Unix shell than Lisp machine

            Strongly agree with this. It’s extremely good as a “visual shell” / UI framework. It’s not usually a good UX when you try to implement too much of your world in Elisp with dozens of packages, spinners, networking, etc. You end up in C-g hell as the author describes. Over 15 years of usage I’ve actually evolved away from the “rewrite everything in Elisp” view and towards “drive scripts from Elisp and slurp results into a buffer for processing”

            Please not, dash and s … don’t always lend themselves to idiomatic Elisp

            Also agree. Writing idiomatic Elisp using the built in APIs isn’t that hard, but it’s more “old skool” and can be imperative/verbose, so I guess some people don’t like it

            On the other hand I think it’s kind of dumb to have to import a “cool” string library to install a library that does something I actually want.

            Elisp isn’t Clojure and I’m glad for that. I think a lot of the complaints are that it isn’t Clojure/CL/etc, but its not supposed to be. It’s basically a 70s era MacLisp clone.

            In general I think two (no three) things:

            1. People can write whatever they want to write, it’s free software, their itch etc
            2. I wish a lot of the people trying to make Emacs “better” (by their definition) but who are also possibly adding complexity to the code base, changing variables and APIs and default settings etc would just chill and learn more about what it is they’re trying to change before concluding “this needs feature X, let’s hack it in and now everyone has to live with it”
            3. I know #2 is something of a straw man and not 100% fair but Emacs is where I live all day every day and somebody else’s “cool new feature” is my “unnecessary risk and complexity” (and possibly lost hours troubleshooting)
            1. 5

              Take a look at the screenshots of these Neovim GUIs:

              I don’t see any notable difference, tbh. What I would get is better integration into window managers (opening downloaded files in existing emacs sessions), but I have never understood what neovim actually achieves by splitting UI and backend. Sure, it’s a good design decision in principle, but considering the complexity of Emacs display engine, I’m not sure if it is worth it.

              The exactly same thing has been said for vim, yet neovim managed to pull if off. Just for the sake of it being a great design decision, it would be worth implementing it. The case in points are so many different forks of emacs, trying to make it run on different platforms (emacs-mac, xemacs, …).

              1. 4

                I don’t get how this should work, when setting up LSP servers is now the issue. If you want to use rust-analyser, you have to download rust-nightly.

                This imo is exactly the problem which should be solved by editor’s extension. In VS Code, the user just installs the rust-analyzer extension and the binary is downloaded automatically. IIRC, coc-nvim provides a similar experience.

                It is important that LSP support in an editor allows for each language to have a dedicated, easy discoverable plugin which deals with dirty integration bits.

                1. 4

                  I’ve had pretty hit-or-miss luck with things actually working in VS Code when you do that. Just yesterday I tried using it for C# for the first time, responded “yes” to the prompt when it asked if I wanted to install the recommended C# extensions, and now I have some kind of dependency hell around .NET version mismatches (“Error: The reference assemblies for .NETFramework,Version=v4.7.1 were not found.”). Some of the language extensions install without problems, but I’ve come not to expect it.

                2. 3

                  If you want to compile a Haskell LSP server, you’ll need to buy some more RAM.

                  haskell language server provides precompiled binaries that transparently download for vsode, and lsp-mode is working on reusing this ability.

                  You can also install prebuild binaries via ghcup

                  1. 3

                    That’s basically the same problem as with rustup, that I have to install software from external sources, instead of having it all done by my (system’s) package manager.

                    1. 3

                      You can consider using Nix to declaratively create project development environments that includes build tools like LSP servers, which is exactly what I do in my open source Haskell project (run nix-shell --run "code ." and things “just work”, be it on Linux, macOS or WSL).

                  2. 2

                    If you want to use rust-analyser, you have to download rust-nightly.

                    rust-analyzer provides pre-built binaries

                    1. 2

                      I used to have these problems, but in my experience this comes from fighting Emacs instead of cooperating with it. Using unnecessary packages, weird themes and wanting popups everywhere will simply slow stuff down.

                      I don’t think it’s productive to put blame for these problems on the user. Extensibility is the main selling point of Emacs for many, people use it because they want those features. Of course it’s always possible to shot oneself in the foot. But if some of the problems are solvable on the level of platform, they should be solved there. When I pick a theme I want to pick it for its look and not think if it will slow down my editor. If an extension delays startup, becomes unresponsive or slows down every key stroke I want to be told by Emacs about what’s to blame but I also want it to be hard to achieve.

                      It shouldn’t be as easy to bring the editor to a crawl. The user doesn’t have to be an expert or to use vanilla Emacs for it to be fast. The machine I’m typing this on has 8 cores, I don’t see why crazy computations to paint parentheses in rainbow colours should slow down my typing or scrolling. I think support for asynchronous execution is one of the main reasons why Neovim succeeded.

                      1. 1

                        Hard agree.

                        1. 1

                          Extensibility is the main selling point of Emacs for many, people use it because they want those features.

                          What features? My comments are more related to UI stuff like pop-ups or constant redrawing effects – in other words non-functional aspects. They can be nice and fun, but since most of the time they are hacks, it’s nor surprising that they aren’t efficient.

                          If an extension delays startup, becomes unresponsive or slows down every key stroke I want to be told by Emacs about what’s to blame but I also want it to be hard to achieve.

                          The problem here is that there is no such thing as an extension. There is no means to distinguish between the core and (so-called) extension, because they are equal.

                          It shouldn’t be as easy to bring the editor to a crawl.

                          As an editor, yes, but again Emacs is a fully programmable environment. Evaluate

                          (defalias 'eval #'identity)
                          

                          and slowly everything will start to break. Emacs’ power necessarily includes the ability to break stuff. The barrier can be lowered, but adding more (finite) computing power doesn’t solve the “issue”.

                        2. 1

                          But if this would be solved, and all I have to do is run “sudo package-manager install java-lsp-server”, then it would be a considerable advancement.

                          I’ve only tried clangd, which I was able to just grab via my distro’s package manager and it just worked (other than that every few hours clangd gets OOM killed…)

                        1. 5

                          People seem to miss the point of OO. The point is about names. The example defines save_items, but realistically, to avoid name clashes, it would be client_save_items. Similarly, “bag of functions” leads to names like treemap_add and hashmap_add, which is redundant, because it can be inferred from the runtime type of the first argument. The point of OO is to avoid redundancies of TreeMap and treemap_add and HashMap and hashmap_add and collapse them to add.

                          1. 6

                            For purely namespacing purposes I personally prefer modules over classes in Python, in a lot of cases. Put a collection of functions into foo.py, then import foo from another file, and all those functions are nicely namespaced as foo.name. Admittedly classes vs modules depends a lot on the project and programming style.

                            1. 2

                              treemap.add is not really an improvement over treemap_add. The point is you write m.add(k, v), not m.treemap_add(k, v). Compare treemap_add(m, k, v) and treemap.add(m, k, v).

                            2. 3

                              I think I can safely speak for Alan Kay when I say that this is not the point of OO.

                              The points of OO are dynamic/late binding, encapsulation, and inheritance, IMHO and IIRC.

                              However, I agree that OO-style naming is a good thing. You can have it without objects using function overloading, which pairs well with a universal-function-call syntax as found in Nim and some other languages.

                              1. 5

                                “Inheritance” was definitely not included in Kay’s famous quote:

                                OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things.

                                I argue that inheritance is an anti-pattern, and is in large part responsible for OO hate. OO with composition only and immutable value objects can be pretty nice.

                                EDIT: This got me to search for something more specific on Kay’s feelings about inhertiance, and I found this great post by him on Quora

                                1. 3

                                  These days Alan Kay says that OO is about message passing. I’ve seen some debate about whether that was always his message. hwayne probably has a more informed opinion on this than I do.

                                  1. 5

                                    Given the history, I suspect that Alan Kay did a very poor job of explaining what he did and has been yelling at us ever since for failing to read his mind correctly.

                                    1. 1

                                      Dealers of Lightning has a chapter on how Kay was disappointed with Smalltalk and wanted to start again, but the team basically told him no.

                                      BTW: When I asked him about Self, he told me:

                                      I liked Self. “Good OOP” is still waiting for a much better notion to replace the idea of a “Class”

                                  2. 1

                                    If OO is about dynamic/late binding etc, C++ without virtual functions shouldn’t be OO at all. But to the contrary, C++ without virtual functions looks and feels like C with OO, and virtual functions feel like a superfluous addition.

                                    1. 3

                                      C++ without virtual functions shouldn’t be OO at all.

                                      Alan Kay would agree: “I made up the term object-oriented, and I can tell you I did not have C++ in mind.”

                                    2. 1

                                      BTW: Only when I’ve read the book Moldable tools, I’ve finally got some of Kay’s points of OOP and “modeling”.

                                    3. 1

                                      The point was about “enterprise environment”, design patterns and Java eating our lunch. Ohh, how many times have I seen people complaining about perl using an arrow instead of a dot.

                                      1. 1

                                        An arrow? That’s so silly — ooRexx uses a ~ twiddle.

                                    1. 14

                                      I’ve been really tempted to buy a remarkable2. But the reviews I see say it’s great for note taking but not so great for just reading PDFs. Mostly I want to read PDFs. I’m still on the fence.

                                      1. 14

                                        As long as your PDFs don’t require color, it is 100% worth it. Definitely one of my favorite devices at the moment.

                                        1. 5

                                          Same. In the month or so I’ve had one, it hasn’t caused me a single frustration (and I’m the kind of person who gets annoyed at the user interfaces of my own Apple products). It works exactly as advertised. Anyone who thinks it might be worth the price tag should watch a third party review video and check out the official and awesome list projects. It has been awhile since I’ve stayed this excited about a new device so long after buying it.

                                        2. 12

                                          I picked one up recently hoping that I could migrate a lot of my ebooks and pdfs to it. I don’t plan on returning it, but I wouldn’t recommend it.

                                          I was a huge fan of the kindle dx, but I’ve managed to break the buttons on a couple which renders them practically useless. I was on the fence with the first remarkable device but figured I’d given the latest iteration a shot. I figured it’d be a good DX substitute. It’s not. I want to like it, the physical design is really good, but the software sucks.

                                          I have a large collection of documents (epub/pdfs) that I was looking forward to getting on the device. Largely a mix of books published in electronic formats from regular publishers (O’Reilly, Manning, PragProg, etc.) as well as a few papers and docs I’ve picked up here and there.

                                          First, the reMarkable desktop/mobile app that you have to rely on for syncing is a little wonky. Syncing between the device and mobile/desktop versions of the app works, but leaves a little to be desired. Second, I have yet to load a pdf or epub that isn’t brutally slow to navigate (just page by page). If the document has images or graphics (even simple charts and illustrations) it will affect navigation performance. Occasionally a document will load relatively quickly, and navigate reasonable well, only to slow down after a few page turns. Epubs tend to be a little more difficult to work with - particularly if you decide to change the font. All I have to compare this device to is my broken DX, which, everything considered, positively smokes the reMarkable.

                                          It’s usable. It works alright for PDFs, less so for epubs. On the positive side, the battery life is quite good.

                                          1. 3

                                            I agree with your analysis in most regards. Syncing a lot of ebooks and pdfs to it is not something at which it would excel by default. I have a large Calibre library, and I haven’t synced it over for that reason. However, it’s something I’m looking forward to investigating with KOReader, which supports the reMarkable.

                                            I haven’t experienced the lag that you talk about, but can understand that that would be bothersome – though I definitely have experienced the “wonkiness” of the companion apps.

                                            1. 1

                                              My understanding is that epubs are converted to PDF before being synced? Is that actually the case?

                                              1. 4

                                                It renders the epub to pdf for display but that’s all in-memory. It’s still an epub on disk.

                                                1. 1

                                                  I don’t know. I’ve got a couple books that are both pdf and ePub, and the pdf version behaves a little better. You can also resize and change fonts for ePub doc, but not for PDFs.

                                                  1. 1

                                                    Along these lines, another interesting observation I’ve made has to do with the way some kinds of text get rendered. In particular, I’ve encountered epubs with code listings that render fine in other apps and on other devices, but render horribly on the remarkable2 device. Interestingly, in some of those cases I will also have a publisher provided PDF that renders just fine.

                                                    Further, epubs and PDFs are categorized differently in both the app and the device. With epubs you can change the justification, page margins, line spacing, fonts, and font size. With PDFs you have fewer options, but you do have the ability to adjust the view (which is great for papers since you can get rid of the margins).

                                                  2. 2

                                                    I don’t think so – from my playing around with ssh, there are definitely some epubs stored on device. I actually think the browser extension generates epubs, rather than pdfs which was surprising.

                                                    1. 2

                                                      Huh. Cool. Hmmm. The real reason I shouldn’t get one is that I always fall asleep with my e-reader and it often bounces off my face.

                                                      1. 3

                                                        That’s a pro, for the device, it weighs next to nothing. I’ve damn near knocked myself out dropping an iPad Pro on my head when reading in bed.

                                                        1. 1

                                                          For me, it’s more the fact that the Kobo then ends up falling onto the floor. I’m not crazy with that with a $120 device, so …

                                                2. 7

                                                  I own Gen 1 and Gen 2. I love the simplicity and focus of the device. It’s an amazing… whiteboard.

                                                  Note taking is not suuuper great. Turns out marking up a PDF to take notes actually isn’t that great because the notes quickly get lost in the PDF. It’s not like in real life, where you can put a sticky note to jump to that page. The writing experience is fantastic though. I have notebooks where I draw diagrams/ideas out. I like it for whiteboarding type stuff.

                                                  Reading is terrible. I mean, it works. Searching is painfully slow. The table of contents doesn’t always show up (even though my laptop PDF reader can read the TOC just fine). When you do get a TOC, the subsections are flattened to the top level, so it’s hard to skim the TOC. PDF links don’t work. Text is often tiny, though you can zoom in. EPUBs appear to get converted to PDFs on the fly and their EPUB to PDF conversion sucks. Though, I’ve found doing the conversion myself in Calibre is way better.

                                                  Overall, I like the device for whiteboarding. But it’s kinda hard to recommend.

                                                  1. 2

                                                    Marking up PDFs works better in color, since you can pick a contrasting ink color. I do it in Notability on my iPad Pro (which is also great for whiteboarding / sketching.)

                                                    I was tempted by reMarkable when the first version came out, but I couldn’t see spending that kind of money on something that only does note taking and reading. I’m glad it’s found an audience though, it’s a cool device.

                                                    1. 1

                                                      Turns out marking up a PDF to take notes actually isn’t that great because the notes quickly get lost in the PDF. It’s not like in real life, where you can put a sticky note to jump to that page.

                                                      So far the best experience I’ve seen for this is LiquidText on an iPad Pro. While you can write on the PDF as any other annotator, there’s also a lot of more hypertext type of features, like collecting groups of notes in an index, or writing separate pages of notes that are bidirectionally hyperlinked to parts of the document they refer to. Or do things like pull out a figure from a paper into a sidebar where you attach notes to it.

                                                      The main downside for me is that you do more or less have to go all-on on LiquidText. It supports exporting a workspace to flat PDFs, but if you used the hypertext features in any significant way, the exported PDFs can be very confusing with the lack of expected context.

                                                      1. 1

                                                        Agreed that it is hard to find notes. There should be a way to jump to pages that have notes on them (this is how Drawboard PDF works, for example).

                                                        1. 1

                                                          What is the advantage over drawing on a piece of paper or on a whiteboard, then taking a photo of what you’ve drawn, if needed?

                                                          1. 1

                                                            I tried paper note books, but I’m too messy and make too many mistakes. Erasing, moving, and reordering is hard on paper.

                                                            A whiteboard is pretty good for temporary stuff and erases better than paper. But, it can be a bit messy.

                                                            I also tried Rocketbook for a while. I got the non-microwaveable (yes you read that right) one. That was okay. A little meh for me.

                                                            And of course, you can’t read PDFs on any of these.

                                                      1. 2

                                                        Listing Python’s strength as being computationally efficient seems non-intuitive, but it’s true. In common usage, Python is really just a high level scriptable linker for optimized C/C++ packages and Java/Scala applications (e.g. Spark). Python will take you a very long way in terms of scaling.

                                                        1. 1

                                                          For some things that’s true, but I’ve found it fairly brittle. You really have to find a built-in library function that does exactly what you want. Performance tanks as soon as you have to do anything on the Python side. NumPy in particular is full of gotchas if you don’t think carefully about the execution model. Some numpy functions take a Python function as a parameter (key functions, custom filter functions, etc.), and there are conveniences like numpy.vectorize, but any of these kill your performance, because they’ll end up doing Python function calls in an inner loop, and Python function calls are very slow.

                                                          1. 1

                                                            The numpy.vectorize example is a straw man. The docs are very clear that it’s not performant.

                                                        1. 4

                                                          For whatever it’s worth: I wrote something vaguely similar a while back, which produced a decent amount of discussion here.

                                                          1. 11

                                                            This is so incredibly neat. It downloads all dependencies on the fly, so you don’t need a whole TeX Live installation, but you still have access to all of the goodies should you need them. The TeX engine itself is implemented in Rust. Only drawback right now is that it doesn’t support -shell-escape — I hope someone can hack that in at some point, to enable support for minted and other stuff like that (the authors have a rationale for not including that, in that they want reproducible builds, but it does mean some things can’t be generated right now). Hope someone else finds this as interesting as I did.

                                                            1. 2

                                                              The TeX engine itself is implemented in Rust.

                                                              I was surprised by this claim (reimplementing a TeX engine is a huge amount of work!), so dug into the source a little. From what I can tell, at least for now, only the tectonic front-end is implemented in Rust. That includes stuff like the CLI, the glue that repeatedly calls latex/bibtex, the dependency-downloading code, etc. The actual typesetting is still done by the venerable TeX engine, written in C semi-automatically converted from Knuth’s WEB sources, and then improved by the TeX community over the years. Specifically, tectonic imports a fork of the XeTeX C sources.

                                                              For example, this is tex.rs, which does some front-end work but mostly calls out to tex_simple_main in core-bridge.c, which in turn calls out to tt_run_engine in xetex-ini.c, which then calls into the rest of the XeTeX engine to do the typesetting.

                                                            1. 4

                                                              This is cool and all but it’s a bit of a downer that strictly worse mainstream languages are adopting features from better languages and implementing them in a strictly worse way. E.g. in OCaml you would get an exhaustiveness error like:

                                                              Warning 8: this pattern-matching is not exhaustive.
                                                              Here is an example of a case that is not matched:
                                                              Scheduled
                                                              

                                                              Instead of the inscrutable:

                                                              error: Argument 1 to "assert_never" has incompatible type "Literal[OrderStatus.Scheduled]";
                                                              expected "NoReturn"
                                                              

                                                              (Also, you wouldn’t need to remember to put an else: assert_never(...) at the end of the code.)

                                                              1. 6

                                                                In OCaml pattern matching is a core feature, here it’s being emulated by clever users. There’s a proposal to add pattern matching to Python 3.10, in which case presumably Mypy would be able to do exhaustiveness checking.

                                                                (Also, calling Python “strictly worse” is a bit of a strong statement. Strictly worse how?)

                                                                1. 2

                                                                  Yes, eventually–if and when the proposal is accepted, then ships in Python, then gets implemented and ships in Mypy.

                                                                  Strictly worse how?

                                                                  • Performance
                                                                  • Type safety
                                                                  • Language design (including modules), e.g. the recently-added controversial walrus operator in Python is something that automatically and naturally occurs in OCaml due to its original design

                                                                  But since this is a programming-language-oriented thread, it’s reasonable to assume that we’re comparing mostly language design.

                                                                  1. 7

                                                                    On the other hand, Python runs natively on Windows, so it’s strictly superior to OCaml as a language.

                                                                    (F# is another story…)

                                                                    1. 5

                                                                      Difficult to tell whether this is a serious assertion, but on the off chance:

                                                                      • OCaml has no fewer than three ports to Windows. If this seems like a disadvantage, remember that many people actually recommend Anaconda as an alternative to the official Python installer on Windows.
                                                                      • The most well-known port, OCaml for Windows, provides recent compiler versions and almost all opam (equivalent of PyPI) packages.
                                                                      • Opam is planning full Windows support out of the box in an upcoming release.
                                                                      1. 6

                                                                        It was tongue-in-cheek up until I read your response, which made it suddenly an excellent argument about why “strictly superior” is nonsense. All three of your ports need “a Unix-like build environment”. Package manager support is only “planned” in the next version, much as pattern matching is “planned” for 3.10. Comparing it to Anaconda, which is recommended to scientists and data scientists who aren’t programmers, is flat wrong. The only people who could think that OCaml Windows support is as good as Python Windows support, which doesn’t require you to install cygwin, are OCaml partisans.

                                                                        Admitting that OCaml has worse Windows support than Python is the easiest thing in the world. If you’re not willing to even do that, why should I trust you about any of your other claims?

                                                                        1. 2

                                                                          Comparing it to Anaconda, which is recommended to scientists and data scientists who aren’t programmers, is flat wrong.

                                                                          Let’s actually look past the marketing blurb and check how Anaconda describes itself:

                                                                          ’Anaconda Individual Edition is a free, easy-to-install package manager, environment manager, and Python distribution with a collection of 1,500+ open source packages with free community support.

                                                                          ’Anaconda Commercial Edition is the world’s most popular open-source package distribution and management experience, optimized for commercial use and compliance with our Terms of Service.

                                                                          ’Anaconda Team Edition is our latest generation repository for all things Anaconda. With support for all major operating systems, the repository serves as your central conda, PyPI, and CRAN packaging resource for desktop users, development clusters, CI/CD systems, and production containers.

                                                                          ‘Anaconda Enterprise is an enterprise-ready, secure, and scalable data science platform…’

                                                                          (From https://docs.anaconda.com/ )

                                                                          So interestingly, they position only their enterprise edition as a data science platform, and the others as general-purpose Python distributions.

                                                                          Now we can make the argument that it’s still used mainly by data scientists. But that just proves my point! Data scientists, like most other computer users, are mostly using Windows. So why would they prefer Anaconda, which is (if we ignore the Enterprise edition) just a Python distro? Because it has an easy Windows installer which loads you up with commonly-used libraries! I.e. specific distribution for a specific need, just like the OCaml Windows ports.

                                                                          Admitting that OCaml has worse Windows support than Python is the easiest thing in the world. If you’re not willing to even do that, why should I trust you about any of your other claims?

                                                                          That’s a non sequitur. I was responding to your claim that:

                                                                          Python runs natively on Windows

                                                                          …with the implication that OCaml does not. You’re a smart guy, you probably already know that it does and has for a long time. I was just expanding on that and saying that the surrounding tooling is also catching up.

                                                                          Also, since when is Windows (a non-free operating system) support a measure of the quality of a programming language? Is this a portability argument in disguise? Are you suggesting in turn that OCaml, which has been ported and released for Apple Silicon, is better than languages which have not, and which due to resource constraints, won’t be for a while? Seems like a nonsensical argument.

                                                                          1. 2

                                                                            And let’s take at the OCaml for Windows page:

                                                                            opam-repository-mingw provides an opam repository for Windows - and an experimental build of opam for Windows. It is work in progress, but it already works well enough to install packages with complex dependencies (like core_kernel) and packages with external dependencies (e.g lablgtk).

                                                                            The repository is forked from the standard version. It contains Windows specific patches and build instructions. Especially build related tools (ocamlfind, ocamlbuild, omake, oasis, opam and OCaml itself) were modified, so that most unix-centric build instructions will also work with the native Windows/OCaml toolchain and I can sync the repository with the main repo from time to time (and without too much hassle).

                                                                            I don’t know about you, but this doesn’t stir confidence in me that I’ll be up and confidently running OCaml in Windows. Which brings up the question again, what does it mean to be “strictly superior”?

                                                                            Also, since when is Windows (a non-free operating system) support a measure of the quality of a programming language?

                                                                            What does Windows being non-free have to do with being strictly superior?

                                                                            Is this a portability argument in disguise

                                                                            Since nobody in this thread has defined what “strictly superior” means, why can’t it be?

                                                                            1. 1

                                                                              What does Windows being non-free have to do with being strictly superior?

                                                                              Non-free operating systems don’t have anything to do with language ‘superiority’ at all.

                                                                              nobody in this thread has defined what “strictly superior” means

                                                                              ‘Strictly superior’ is a subjective, and therefore, opinionated argument. So to each their own, but to me:

                                                                              • Statically typed (meaning, no typechecking at runtime)
                                                                              • Pattern matching
                                                                              • Exhaustivity checking built in
                                                                              • Every object/method/value is not a key/value in a bunch of dicts which can be runtime-patched
                                                                              • No nulls
                                                                              • Immutable values
                                                                              • Expression-oriented syntax
                                                                  2. 2

                                                                    Fwiw, Java supports that kind of exhaustiveness checking now too, added in JDK 13 as part of switch expressions. Seems to be becoming a more common feature even outside of functional programming.

                                                                    1. 1

                                                                      That’s the thing–all features from functional programming (starting from garbage collection in the ’50s) eventually find their way into ‘mainstream’ programming languages, all the while FP is called ‘academic’ and ‘impractical’ every step of the way 😂

                                                                      1. 1

                                                                        You can say the same about OO languages. After all, Simula and Smalltalk were certainly not widely popular. C++ first appeared in 1985, which was a good 25 years after Algol-60 was released. And FP languages have gotten plenty wrong. Haskell’s laziness has been criticized for years, and I’d be happier if I didn’t have to deal with Scala implicits in my code again.

                                                                  3. 3

                                                                    “Not exhaustive” error messages when dealing with non-trivial types are amazing. I sit there and look at the example trying to figure it out then when I do half the time it is a bug that would take a lot of time to find.

                                                                    1. 2

                                                                      It’s a much better value proposition to add something like this that provides, let’s be conservative, half the utility, to an existing codebase than it is to switch languages (and ecosystems, tools, and so forth) in order to get the full utility.

                                                                      At some point the definitions of “better” and “worse” need to be weighted based on how much work is actually accomplished with a given tool / language.

                                                                      1. 2

                                                                        Hopefully also weighted by a fair accounting of the amount of bugs prevented by the language.

                                                                        1. 2

                                                                          Don’t listen to him! It’s a trap! It’s totally not fair to allow OCaml and friends to put the bugs prevented by the language on their side of the ledger because those are several infinite classes of bugs.

                                                                      2. 2

                                                                        My counterpoint is that OCaml only provides this feature through having a purpose-built ADT for it.

                                                                        Mypy (and typescript) generalize these concepts by slowing arbitrary union of types. You can have “string or bytes” instead of “Either String Bytes” (with the wrapping required at call sites)

                                                                        From a usability perspective OCamls stuff is way more restrictive and verbose

                                                                        1. 1

                                                                          I guess it’s possible with a suitable type system, e.g. in Scala you can have an implicit that coercing the value into Either, or in C++ it could work via an implicit constructor. But yeah, to me was somewhat surprising to find myself (as a big fan of types!) enjoying mypy and other gradual typing systems more than ‘stricter’/‘static’ ones.

                                                                          1. 1

                                                                            Arbitrary unions of types turn out to be a bad idea as soon as you need to deal with generics. Difficult to work with A | B if A and B are generics.

                                                                        1. 4

                                                                          Hm I just noticed that the gg paper shares 2 authors with this 2020 paper POSH: A Data-Aware Shell

                                                                          POSH optimizes shell scripts by bringing code to data, sort of in the style MapReduce. It is for what I’d call big data / small computation (that is, I/O intensive workloads). So it is actually complementary to gg, which is small data / big computation (CPU intensive).

                                                                          Although POSH doesn’t share a runtime with gg, which I was sort of surprised by. The gg IR looks very shell-like, so I thought it would be natural to extend that to pipelines somehow.


                                                                          Coincidentally I learned about POSH from another shell optimization project which I just learned about, Pash! (also 2020)

                                                                          PaSh: Light-touch Data-Parallel Shell Processing

                                                                          Notes on these projects here: https://github.com/oilshell/oil/issues/867

                                                                          And I read the gg paper more carefully and put some notes here: https://github.com/oilshell/oil/wiki/Distributed-Shell

                                                                          There is a great video about it on YouTube, a Hacker News thread with some interesting experiences with lambda for astronomy, and more etc.

                                                                          I really like the framing of this work: low latency, cold vs. warm clusters, etc. They don’t mention the shell connection that much, but I think there is an obvious relation, and the tool itself is very shell oriented (the gg infer wrapper, etc.). It also makes me a lot more interested in Lambda, which has apparently supported arbitrary containers since late 2018.

                                                                          1. 2

                                                                            Sounds related to Joyent Manta (or whatever it’s called these days).

                                                                            1. 3

                                                                              Oh yeah thanks for mentioning that! I remember going to talk at Joyent in SF about Manta when it launched (2011 or 2013?) It’s a good idea: Unix-style polyglot MapReduce. (Hadoop has something similar but not widely used.)

                                                                              In fact I’m pretty sure I remember Bryan Cantrill saying in another talk that part of the reason Samsung acquired Joyent is because they had a huge Manta bill? Like it was cheaper to acquire the whole company LOL… (I think Samsung was allergic to Google Cloud for competitive reasons, and apparently AWS too ?)

                                                                              I don’t have a reference, but all the press releases mention Manta:

                                                                              https://news.samsung.com/global/samsung-to-acquire-joyent-a-leading-public-and-private-cloud-provider


                                                                              FWIW they used Solaris Zones, and this was before there were any standard container formats. So I don’t think they had anything like what AWS Lambda or Kubernetes has with Docker containers.

                                                                              1. 1

                                                                                FWIW they used Solaris Zones, and this was before there were any standard container formats. So I don’t think they had anything like what AWS Lambda or Kubernetes has with Docker containers.

                                                                                No, but for some use cases what they had was even better imo, and something I haven’t seen replicated elsewhere. They used some fancy ZFS overlay stuff to give every zone built-in access to thousands of pkgsrc packages from a given pkgsrc quarterly revision, as if they had been installed in the zone. They didn’t count as part of your zone size and didn’t slow down zone startup, so were basically a “free” 10gb or so of dependencies you didn’t have to drag around yourself as part of a container.

                                                                                Especially for data analysis work and one-off jobs this was nice. You could write a script assuming that basically anything reasonable was already installed in the zone: compilers and interpreters for almost any language, the whole Python data-science stack, R, CLI utilities like jq and xmlstarlet, etc., etc. You could also then specify additional pkgsrc packages to install at startup, or .tar.gz’s to copy into the zone, but you only had to do that for more unusual or non-open-source stuff.

                                                                                1. 1

                                                                                  Yeah I don’t doubt that it was better in many ways, especially if you compare it to Docker at that time [1]. Although when I went to that particular talk, close to launch time, they didn’t have a good solution yet. I think they just had a fixed Zone that resembled the host Zone. It was a canned set of packages.

                                                                                  Actually I just tried NearlyFreeSpeech, which uses FreeBSD jails, and it’s kind of similar. They have something called “realms” which is a big blob of software they update once in awhile.

                                                                                  https://faq.nearlyfreespeech.net/q/siterealm

                                                                                  I think this probably works for a lot of use cases, but it’s less flexible than what’s becoming the standard with Linux, e.g. with Lambda and Kubernetes (the latter of which feels just as clunky as Docker).


                                                                                  Having fast startup is definitely a problem with these types of systems, and I think their solution addresses that. However I think there is still the “local dev environment problem”.

                                                                                  If they have Python 3.4, and you want Python 3.8,. you might be out of luck, or you might be in for a lot of work.

                                                                                  This is problem extends up the stack, especially with the scientific software stack, e.g. NumPy/SciPy/Pandas and R. They move faster than distros move. Packaging them is a huge pain. The upstream devs really only work on Linux or Windows.

                                                                                  The HN thread on gg has a good example of that with respect to astronomy, and I’ve experienced similar problems for a long time: https://news.ycombinator.com/item?id=20433315

                                                                                  So Lambda is still a pain, but I’m virtually certain the pain would be just as bad or worse on Manta, even now..


                                                                                  BTW I took another look at Manta: https://apidocs.joyent.com/manta/job-patterns.html

                                                                                  e.g. for https://github.com/oilshell/oil/issues/867

                                                                                  My reaction is that I think they need structured data over pipes. I love the shell, but I’ve also done a lot of MapReduce stuff, and having records is sort of a “must have”. Still working on it in Oil: https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil

                                                                                  [1] I’m glad Docker is getting refactored into something more stable / refactored away

                                                                          1. 4

                                                                            I work at a university, and at least here they explicitly told us they were going to be looking at keycard swipes for contact-tracing. That was one of the justifications for all buildings being card-access only this semester, so they have a list of who was inside each building on each day (the other justification is increased risk of theft/vandalism due to many buildings being largely empty).

                                                                            1. 4

                                                                              I can’t find any mention of input or interactivity in Gemini. Is it possible to build something like lobster.rs or wikipedia?

                                                                              1. 3

                                                                                A page can request user input in a single text field, which has the semantics that the same page is re-fetched with the contents of the input field added as a query parameter (see 3.2.1 in the spec). So far I’ve seen this used for: 1) search boxes, and 2) text input to interactive fiction games.

                                                                                I don’t think it’s designed for more wiki-style editing where the entire document itself is edited in the same viewer.

                                                                                1. 2

                                                                                  Just like the WWW in 1994. If Gemini catches on, it will quickly (d)evolve into the web of 1996.

                                                                              1. 4

                                                                                So what I didn’t get: What is FaaS ? What exactly is the use case for this service ? Defining some function which can then be called from anywhere (probably “sold”) ?

                                                                                1. 9

                                                                                  It’s like CGI billed pr request on a managed server.

                                                                                  1. 7

                                                                                    NearlyFreeSpeech.net does something sort of like that. Not per request, but based on RAM/CPU minutes. I find it pretty convenient.

                                                                                    1. 5

                                                                                      Yeah, I was hoping on billing by resource use (RAM, CPU and data transfer through syscalls) in a way that would give you a more precise view into how long your programs were taking to run. This would also give people an incentive to make faster code that uses less ram, which i would really love to see happen.

                                                                                  2. 3

                                                                                    I think this whole FaaS is a very interesting movement. Combined with more edge pods we deployed through Fastly / Cloudflare, we are onto something quite different than the big cloud services we saw with Facebook / Twitter / Gmail today.

                                                                                    Reimagining how you are going to do email today (or really, any personalized messaging system like Whatsapp). These edge pods with deployment bundles like wasm enables us to implement the always online inbox, potentially with high availability right at the edge. So your device syncing will be extremely fast. At the same time, it is hosted online so everyone can reach you with minimal central coordination.

                                                                                    It is unlikely in the beginning these things by their own will be successful. Decentralization is not a big consideration at this time. But it could deliver benefits today if implemented correctly, even for existing big players like Facebook. You could have a Wasm edge node close to the device materialize the feed GraphQL fragments in anticipation for a new user request. And since the edge node will know what’s the last synced feed fragment, it can also do so incrementally.

                                                                                    I am optimistic about this. Would love to see wasm based deployment taking off, especially for edge nodes.

                                                                                    1. 1

                                                                                      This is an approach and idea that DFINITY (https://dfinity.org/) is pursuing, to provide a fully decentralized computing platform. The system is running wasm as the basic unit of execution, and charges for the cycles, memory, and bandwidth used. Currently, it is in beta, but should become available next year.

                                                                                      Disclaimer: I work for DFINITY.

                                                                                      1. 1

                                                                                        Thanks! Yep, I looked at DFINITY before. One thing would be compelling to me is the closeness to the customers. With our cloud computing moved to the low-latency territory (most significantly, the cloud gaming), closeness of the edge nodes is a necessity. This is often overlooked by many decentralized movements from cryptocurrency space (probably because these Dapps have different focuses).

                                                                                    2. 2

                                                                                      Functions as a Service. Basically the usecase is for people that want to run code that doesn’t run often enough to justify having a dedicated box for it, and just often enough that you don’t want to set up anything for it beforehand. In this case, I plan to start using it for webhook handlers for things like GitHub and Gitea.

                                                                                      1. 2

                                                                                        So then you plan to be administering/running Wasmcloud? The idea is that people can just upload code to you? What hosting service are you using?

                                                                                        This reminds me that I need to write about shared hosting and FastCGI. And open source the .wwz script that a few people are interested in here:

                                                                                        https://lobste.rs/s/xl63ah/fastcgi_forgotten_treasure

                                                                                        Basically I think shared hosting provides all of that flexibility (and more, because the wasm sandbox is limited). I do want to stand my scripts up on NearlyFreeSpeech’s FastCGI support to test this theory though…

                                                                                        I think the main problem with shared hosting is versioning and dependencies – i.e. basically what containers solve. And portability between different OS versions.

                                                                                        I think you can actually “resell” shared hosting with a wasmcloud interface… that would be pretty interesting. It would relieve you of having to manage the boxes at least.

                                                                                        1. 4

                                                                                          So then you plan to be administering/running Wasmcloud?

                                                                                          I have had many back and forth thoughts about this, all of the options seem horrible. I may do something else in the future, but it’s been fun to prototype a heroku like experience. As for actually running it, IDK if it would be worth the abuse risk doing it on my own.

                                                                                          The idea is that people can just upload code to you?

                                                                                          If you are either on a paid tier, uploading the example code or talked with me to get “free tier” access yes. This does really turn into a logistical nightmare in practice though.

                                                                                          What hosting service are you using?

                                                                                          Still figuring that part out to be honest.

                                                                                          I think the main problem with shared hosting is versioning and dependencies – i.e. basically what containers solve.

                                                                                          The main thing I want to play with using this experiment is something like “what if remote resources were as easy to access as local ones?” Sort of the Plan 9 “everything is a file” model taken to a logical extreme just to see what it’s like if you do that. Static linking against the platform API should make versioning and dependencies easy to track down (at the cost of actually needing to engineer a stable API).

                                                                                          I think you can actually “resell” shared hosting with a wasmcloud interface… that would be pretty interesting. It would relieve you of having to manage the boxes at least.

                                                                                          I may end up doing that, it’s a good idea.

                                                                                          1. 1

                                                                                            (late reply)

                                                                                            FWIW I have some experience going down this rabbithole, going back 10 years. Basically trying to make my own hosting service :) In my case part of the inspiration was looking for answers to the “polyglot problem” that App Engine had back in 2007. Heroku definitely did interesting things around the same time period.

                                                                                            Making your own hosting service definitely teaches you a lot, and it goes quite deep. I have a new appreciation for all the stuff we build on top of. (And that is largely the motivation for Oil, i.e. because shell is kind of the “first thing” that glues together the big mess we call user space.)


                                                                                            To be a bit more concrete, I went down more of that rabbithole recently. I signed up for NearlyFreeSpeech because they support FastCGI. I found out that it’s FreeBSD! I was hoping for a “portable cloud” experience with Dreamhost and NearlyFreeSpeech. But BSD vs. Linux probably breaks that.

                                                                                            It appears there are lots of “free shell” providers that support CGI, but not FastCGI. There are several other monthly providers of FastCGI like a2hosting, but not sure I want to have another account yet, since the only purpose is to test out my “portable cloud”.

                                                                                            Anyway, this is a long subject, but I think FastCGI could be a decent basis for “functions as a service”. And I noticed there is some Rust support for FastCGI:

                                                                                            https://dafyddcrosby.com/rust-dreamhost-fastcgi/

                                                                                            ( I’m using it from Python; I don’t use Rust)

                                                                                            It depends on how long the user functions will last. If you want very long background functions, then FastCGI doesn’t really work there, and shared hosting doesn’t work either. But then you have to do A LOT more work to spin up your own cloud.

                                                                                            It’s sort of the “inner platform problem” … To create a platform, you have to solve all those same problems AGAIN at the level below. I very much got that sense with my prior hosting project. This applies to packaging, scheduling / resource management, and especially user authentication and security. Security goes infinitely deep… wasm may help with some aspects, but it’s not a complete solution.

                                                                                            And even Google has that problem – running entire cluster managers, just to run another cluster manager on top! (long story, but it is interesting)

                                                                                            Anyway I will probably keep digging into FastCGI and shared hosting… It’s sort of “alternative” now, but I think there is still value and simplicity there, just like there is value to shell, etc.

                                                                                      2. 1

                                                                                        So what I didn’t get: What is FaaS ?

                                                                                        FaaS is a reaction to the fact that the cloud has horrendous usability. If I own a serer and want to run a program, I can just run it. If I want to deploy it in the cloud, I need to manage VMs, probably containers on top of VMs (that seems to be what the cool kids are doing), and some orchestration framework for both. I need to make sure I get security updates for everything in my base OS image and everything that’s run in my container. What I sctually want is to write a program that sits on top of a mainframe OS and runs in someone’s mainframe^Wdatacenter, with someone else being responsible for managing all of the infrastructure: If I have to maintain most of the software infrastructure, I am missing a big part of the possible benefit of outsourcing maintenance of the hardware infrastructure.

                                                                                        Increased efficiency from dense hosting was one of the main selling points for the cloud. If I occasionally need a big beefy computer but only for a couple of hours a month and need a tiny trickle of work done that wouldn’t even stress a first-generation RPi the rest of the time, I can reduce my costs by sharing hosting with a load of other people and having someone else manage load balancing across a huge fleet of machines. If; however, I have to bring along a VM, container runtime, and so on, then I’m bringing a fixed overhead that, in the mostly-idle phases, is huge in comparison to my actual workload.

                                                                                        FaaS aims to provide a lightweight runtime environment that runs your program and nothing else and can be scaled up and down based on load and billed by RAM MB-second, CPU-second and network traffic (often with some rounding). It aims to be a generic and scalable version of the kind of old-school shared hosting, where a load of people would use the same Apache instance with CGI: the cost of administering of the base environment that executes the scripts is shared across all users and the cloud provider can run the scripts on whatever node(s) in the datacenter make sense right now. The older systems typically used the filesystem for read-only data and a database for persistent data. With FaaS, you typically don’t have a local filesystem but can use cloud file / object stores and databases as you need them. Again, someone else is responsible for providing a storage layer that can scale up and down on demand and you pay for the amount of data that’s stored there and how often you access it but you don’t need to overprovision (as you do for cloud VM disks, where you’re paying for the maximum amount of space you might need for any given VM).

                                                                                        TL;DR: FaaS is an attempt to expose the cloud as a useful computer instead of as a platform on which you can simulate a bunch of computers.

                                                                                      1. 8

                                                                                        The head of Red Hat’s desktop team (Christian F.K. Schaller) posted a comment in June 2019 warning that this was likely to happen soon:

                                                                                        Once we are done with this we expect X.org to go into hard maintenance mode fairly quickly. The reality is that X.org is basically maintained by us and thus once we stop paying attention to it there is unlikely to be any major new releases coming out and there might even be some bitrot setting in over time. We will keep an eye on it as we will want to ensure X.org stays supportable until the end of the RHEL8 lifecycle at a minimum, but let this be a friendly notice for everyone who rely the work we do maintaining the Linux graphics stack, get onto Wayland, that is where the future is.

                                                                                        Though there’s a bit of contradiction there, because RHEL8 is officially supported through 2029. I guess they will just do absolute minimum support (e.g. patching CVEs)?

                                                                                        1. 16

                                                                                          I fully expect by 2027 that Windows will either be Open Source or will have swapped out the NT kernel for Linux, and Microsoft’s primary products will be cloud-based distributions of Office, a gaming-centric desktop operating system, the Xbox, SQL Server, and cloud services.

                                                                                          You heard it here first, folks.

                                                                                          1. 4

                                                                                            Microsoft’s primary products will be cloud-based distributions of Office

                                                                                            Isn’t this already the case with O365?

                                                                                            1. 2

                                                                                              Right but my implication was that you wouldn’t even be able to buy standalone Office anymore. Maybe that’s already come to pass; I don’t keep tabs on that too much.

                                                                                              1. 2

                                                                                                Maybe that’s already come to pass

                                                                                                Not yet; you can still buy standalone Office 2019 for both Windows and Mac. But yes they clearly are moving away from it. The product page introduces it somewhat condescendingly as a product for “customers who aren’t ready for the cloud”.

                                                                                            2. 2

                                                                                              After hearing Brian Cantrill rant about the Linux kernel a billion times, something makes me wonder if we’re missing out by having the Linux kernel be the “winner” here, compared to some more featureful OS kernels.

                                                                                              Granted he probably has many opinons on this

                                                                                              1. 2

                                                                                                I, for one, await the return of glorious Solaris and Solaris Zones

                                                                                                1. 2

                                                                                                  Linux is generally more featureful than other options. That’s a result of thousands of people banging on it and adding the warts they want for their use cases.

                                                                                                  The downside is that it feels to me like Linux is built out of warts, glued together with workarounds. It’s not a fun codebase to interact with.

                                                                                                2. 1

                                                                                                  I agree. It is clear that we will, at long last, win the war of Linux vs Windows - but we need to make sure that our victory is not a pyrrhic one.

                                                                                                  1. 13

                                                                                                    but we need to make sure that our victory is not a pyrrhic one.

                                                                                                    Too late!

                                                                                                    The massive footprint of FOSS code, and its centrality to the computing ecosystem as a whole, has arrived largely as a function of its utility to megacorp-scale business in service of utterly defeating the very things that made open code and software freedom attractive and vital propositions to so many of us in the first place. There’re few clearer examples of this than the general role of the Linux kernel and friends in the current software economy.

                                                                                                    We lost by winning, or won by losing, or however you want to frame it. At any rate, ca. 2020 the bad guys are in charge and the end user / subject of computational systems is in the general case basically fucked.

                                                                                                    1. 3

                                                                                                      I don’t know if the end user is any more fu$%^& by the corporations than in the 90s early 00s. We mostly won on the war on the code being readable and modifiable by anyone. We lost it on the system’s architecture and the data.

                                                                                                      But the main thing is that we as users let ourselves get fu%^&* by the corporations by not treating technology as a useful tool but more and more as an integral part of our lives.

                                                                                                    2. 8

                                                                                                      “Meet the new boss, same as the old boss”

                                                                                                      Some context to digest while you mash that unkind or troll flag button: I’ve used Linux, off and on, for 20 years or so now. I’ve never really used Windows, especially not professionally. At this point it’s been a few years since I’ve used MacOS, even at home, but that’s OK since I doubt I’ll ever go back. But at this point, I’d have to say that Linux is my Windows. I don’t use it because I love it, I use it because it works well enough, is very compatible with all the stuff I need to work with, and I’m already familiar with it. I’m much more fond of the BSDs, and even more enamored of relatively obscure and impractical systems like Genode or Haiku or Redox or even Oberon; things that I don’t really have time to play with at this point in my life. So that’s where I’m coming from.

                                                                                                      But even back when I was more zealous, framing the choice of OS as a “war” seemed wrong to me. There are so many different use cases, so many different needs, and the whole field is so young still. Why does it have to be winner-take-all? Why does it have to be a monoculture? Do all you opinionated expert technical people really desire that level of violence in pursuit of homogeneity, or is it just clumsy rhetoric wielded in ignorance of what war is really like?

                                                                                                      Now, I’d be the first to point out that “Linux” is hardly just one thing. There are so many distros and desktops and init systems and so on; even Android is Linux if you squint at it right. But the fact remains, the Linux kernel (and more broadly, POSIX) is a paved cowpath that’s become a fenced-off megahighway. It makes some choices fast and efficient while making others nearly impossible, or at least highly dangerous. It’s not a stream you can cross, and it is very much driven by its corporate sponsorship. It’s a big improvement over Windows or MacOS in that anyone can read the code, at least in theory. But the number of people in the world who can actually do anything significant with that seething mass of exceedingly complex “open source” systems software is vanishingly small, and the number of people who actually contribute even smaller. If we’re just talking about the kernel, it’s a tiny cadre of highly disciplined corporate engineers who live and die by a code of backwards compatibility. Is it a big improvement that instead of just one company developing the OS and its APIs, it’s a loose hegemony of mutually distrustful competitors? I don’t know. Maybe, but it sure enforces technological lock-in, and that’s not necessarily a good thing.

                                                                                                      In a networked world, compatibility should be driven by protocols and specs with independent implementations. What we have instead, too often, is arbitrary de-facto standards. Linux might be less of an offender than MS in this regard, but it’s still quite culpable. You won’t catch me cheering for the new boss.

                                                                                                      1. 9

                                                                                                        Linux doesn’t have >30 years of crushing opposing services and removing business threats.

                                                                                                        Microsoft has been practicing Embrace, Extend, Extinguish for over 30 years. It has been practicing Fear, Uncertainty, Doubt for over 30 years.

                                                                                                        In fact, they intended to do it in relation to Linux, as of the late 90s (As the Halloween Memos show

                                                                                                        In discussing ways of competing with open source, Document I suggests that one reason that open source projects had been able to enter the server market is the market’s use of standardized protocols. The document then suggests that this can be stopped by “extending these protocols and developing new protocols” and “de-commoditiz[ing] protocols & applications”.

                                                                                                        They also channeled large amounts of money to SCO in the hope that the copyright lawsuit would be won by them:

                                                                                                        The document describes, among other points, Microsoft’s channeling of $86 million (equivalent to $116 million in 2019) to SCO.

                                                                                                        For decades, they have deliberately hampered the market and new technologies to maintain an upper hand.

                                                                                                        I do not think anyone with the full history can consider The Linux Foundation to be “slightly less bad” than Microsoft

                                                                                                        1. 2

                                                                                                          That’s not my argument, though. My argument is, monocultures are bad. War is bad. The relative badness of Party A and Party B is irrelevant.

                                                                                                        2. 8

                                                                                                          While I agree with a lot of what you’ve said, I’d like to point out that for me personally its not been a war o as in “Linux is better, Windows should diaf”. For me it’s been about not rewarding the bully and the absolutely horrible behavior.

                                                                                                    1. 9

                                                                                                      Johnathan Edwards built a prototype visual programming language around decision tables, called “Subtext 2”. Here’s the OOPLSA ‘07 paper. Interesting work, but he abandoned it. And its successors… looks like he’s up to “Subtext 10” now.

                                                                                                      1. 1

                                                                                                        “built” seems like a strong statement, looks like an academic project? So it’s all papers and demos and blog posts, but nothing anyone could actually try out…

                                                                                                        1. 4

                                                                                                          No, there were (are) definitely some functioning prototypes of all those systems, but nothing he wanted to release or support. Research is like that. You probably could get some source code if you sent him a nice email… but if you’re genuinely interested in the ideas, you should probably just read the papers, understand them as best you can, and implement them yourself. Research!

                                                                                                          1. 2

                                                                                                            From the demo videos, “built a prototype” seems accurate enough, though it’s not “released”. Plenty of people and companies don’t release every prototype they build.

                                                                                                            1. 2

                                                                                                              Ah, I’ll take your parsing. I read it as ((built) (a prototype language)) but ((built a prototype) [of a] language) makes much more sense here

                                                                                                        1. 3

                                                                                                          What do you think about visual representations of the same data, in forms like decision trees or flowcharts?

                                                                                                          I have teaching experience with decision tables in an admittedly pretty specific domain: Decision tables are one of the canonical ways of specifying a finite state machine (called a state-transition table in this context). Typically there will be a column for states, a column for inputs, and a column for successor state. Each row of the table represents one possible transition.

                                                                                                          The same information can be visualized as a state diagram, where each row of the decision table is represented by a directed edge in a graph instead. Students seem to almost invariably find the diagrammatic representation more intuitive to both read and write, to the point where they regard the table version as an arcane implementation detail at best (e.g. you can shove the table into a 2d array and write a table-driven FSM simulator), but not as a representation suited to humans. This might not generalize to more general contexts than FSMs though, where I could imagine the diagrams getting more hairy.

                                                                                                          1. 2

                                                                                                            In my personal opinion, diagrams tend to scale much better for state machines than tables do. I like to use DTs for single point-in-time decisions, and the more you go away from that the worse they become.

                                                                                                          1. 1

                                                                                                            I’ve been spending some time writing Observable notebooks and the experience really feels like a paradigm shift. Observable gives literate programming, interactive live coding and social & collaboration feature all within one space. It really feels like the single major successful innovation in the field for a long time. The article could also mention Smalltalk, Self, HyperCard or even VisualBasic as environmental that pushed the envelope and went beyond the compiler.

                                                                                                            1. 3

                                                                                                              I’m curious, can you say anything about how Observable differs from running Jupyter notebooks?

                                                                                                              1. 3

                                                                                                                The Observable folks wrote a bit on how they differ from their perspective.

                                                                                                                1. 1

                                                                                                                  Thanks, that’s helpful

                                                                                                                2. 1

                                                                                                                  Observable uses a data flow approach to automatically update the cells independently of their order, and the kernel runs directly in the browsers. You can also import any symbol from other notebooks, making them more like literate modules.

                                                                                                              1. 21

                                                                                                                Kudos to the original author, not only for the high quality original content but for the creative use of phpBB as their blogging platform. Not only does this give simple content mark up, inline attachments and search for free, it also provides an all in one user commenting and subscription mailing capability by changing some basic forum settings, great idea!

                                                                                                                Another pleasant side effect is that it didn’t bring my current gen CPU to its knees rendering drop shadows, rounded corners and scrolling effects! :D

                                                                                                                1. 2

                                                                                                                  Agner’s been using forum software for his blog for a long time, although oddly enough the move to phpBB is recent (2019). From 2009–2019 he used a more minimalist forum that I don’t recognize. Possibly his own software?

                                                                                                                  1. 2

                                                                                                                    Interesting, the generator suggests AForum 1.4.2, but didn’t find anything further.

                                                                                                                    I first hosted and managed phpBB back in 2001, which is I guess why I find its use here so refreshing between the posts about frameworks and ways to bring back ‘the old web’.

                                                                                                                    With all the content management systems and hosted solutions out there, phpBB is a great example of an opensource community and product that has provided a reliable platform and consistent migration path for user owned, self hosted content for close to 20 years.

                                                                                                                    Sure it’s using PHP, which appears increasingly out of vogue with the modern wave a web development. But the stack is so ubiquitous it can be hosted reliably for pocket change.

                                                                                                                  2. 1

                                                                                                                    this is such a cool idea, i wonder why i haven’t seen it before. gotta investigate if it will work for me.

                                                                                                                    also it didn’t have a full screen banner image to scroll past, just a little BBS menu.

                                                                                                                    1. 5

                                                                                                                      i wonder why i haven’t seen it before.

                                                                                                                      I take it you didn’t surf the web much 10-15 years ago. Web forums software was king and there was all sorts of usage in all creative ways. With forum software authors themselves encourage such creative use and always pointing out that you could use it as a blog, a news site, issue tracker, an archive front end, etc.

                                                                                                                      Back in the day, I had this private phpbb forum in which I would archive thousands of posts per day I scrapped from a curated list of blogs via RSS. PHPbb shipped with its own custom made full text index. Searching large corpus was a breeze.

                                                                                                                      Good memories.

                                                                                                                      1. 2

                                                                                                                        The PHP forum wars of the 00’s is an era that shouldn’t be forgotten! and phpBB survived through it all.

                                                                                                                        Not only the inbuilt full text searching, the built in template and caching engine is also worth a mention.. a great piece of work that dramatically dropped resource usage in tightly packed shared hosting environments.

                                                                                                                    2. 1

                                                                                                                      Not a fan of php, but no complaints about the simplicity of the page phpbb generates.

                                                                                                                      1. 1

                                                                                                                        It made so much sense to me that I didn’t realize at all I was looking at an instance of phpBB.

                                                                                                                        Awesome idea.

                                                                                                                      1. 3

                                                                                                                        Last week of the summer before I start teaching fall semester next Monday, so wrapping up some loose research ends and prepping courses. I’m teaching Design & Organization of Programming Languages, which is going to use the new (free) textbook Crafting Interpreters, and Artificial Intelligence, which is going to use a mix of readings and my own course notes in the form of Colab notebooks (hosted Jupyter notebooks).

                                                                                                                        Fortunately, we’re all online, so a bit less stressful than at some other universities. The administration was still claiming we’d be doing hybrid in-person/online classes (using video simulcasting) until about 3 weeks ago, but they gave up on that plan after the District of Columbia government added more restrictions and quarantine rules.

                                                                                                                        1. 0

                                                                                                                          I’d argue that types are a terrible solution to the actual problem.

                                                                                                                          I’d argue the actual problem is, we need to check variables in our languages, for instance for someone’s age, we almost certainly don’t want -32768 as a possible age, as that makes zero sense. Also an age of 32768 is also probably equally stupid.

                                                                                                                          we want age to be an integer between 0 and 120 or so. Certainly < 200. That’s what we actually want.

                                                                                                                          We can argue about the upper and lower bounds of the age variable, depending on the program being written(i.e. an age of 120 for a program about the romans would be laughable, since they never lived remotely that long). But I hope we can all agree there should be upper and lower bounds for our variables.

                                                                                                                          Types, basically don’t do that, and practically every language ever written skips over any sane way to have limits on our variables besides basically int8, int16 and string.

                                                                                                                          There are some attempts, Liquid Haskell has a way. Postgres does this pretty well with the check constraint(though it’s not a programming language, obviously). Nim recently got ‘DrNim’ that attempts the same thing. Most other languages fail.

                                                                                                                          1. 8

                                                                                                                            Types do that. Commonly the pattern looks like having a constructor function that validates the machine type, like smart constructors in Haskell. (Though, as cgenschwap already noted, dependent types can do this in a quite different way.)

                                                                                                                            1. 2

                                                                                                                              I think you meant types might be able to do that. so I agree, in theory, types can grow to specify constraints, but other than Haskell(with multiple competing solutions apparently) and now Ada(thanks @weinholt), nobody’s type system has this ability that I’m aware of(I’m sure there are some weird esoteric languages out there that do..).

                                                                                                                              But no commonly available/used language has anything resembling a way to sanely do constraints. The best you can do is writing a check function and calling it all over the place, hoping you don’t miss a place and your constraint fails to hold, with no warnings if you miss a spot. Languages aren’t immune to the problem, even most databases, except some SQL ones(like PG’s check) pretty much punt on constraints, so it’s a very wide problem.

                                                                                                                              Even web frameworks, mostly fail at checking constraints, and it’s rare to see web code in the wild do much beyond the occasional JavaScript-based client-side validation of inputs. Otherwise the most you will see is ‘yes input validation is important’, but almost nobody is doing it. It’s hard, it’s not taught in programming classes or books or documentation(for the most part) and languages by and large punt on the issue.

                                                                                                                              I’ve read a few haskell learning sites, and a book, though I’ve never programmed anything useful in Haskell, and I’d never heard of smart constructors before, so even when it’s available, nobody apparently uses it.

                                                                                                                              Anyways, the goal here, from my perspective, is we desperately need to validate inputs, and store data with known constraints, and with rare exception, no code does this, and no commonly used languages make this remotely easy. so I think my point stands, constraint checking is still very much not done in the wild.

                                                                                                                              Thanks for teaching me about Smart Constructors! :) I <3 learning new things!

                                                                                                                              1. 8

                                                                                                                                Pascal also supports it:

                                                                                                                                type
                                                                                                                                  Age = 0..200;
                                                                                                                                
                                                                                                                                1. 4

                                                                                                                                  Smart constructors are definitely not in the “nobody uses it” category. They are one of the most pervasive patterns in strongly typed languages (especially ML descendants like Haskell or Scala).

                                                                                                                                  1. 2

                                                                                                                                    Smart constructors are just functions that build values of the required type, but perform some extra checks when the value is constructed…

                                                                                                                                    It seems like you could employ the same smart constructor pattern in many languages. For the age example, you’d create a type for the valid age range and use this type elsewhere in your code:

                                                                                                                                    class Age {
                                                                                                                                        final int value;
                                                                                                                                        Age(int value) {
                                                                                                                                            if (value < 0 || value > 120) {
                                                                                                                                                ...raise exception
                                                                                                                                            }
                                                                                                                                            this.value = value;
                                                                                                                                        }
                                                                                                                                    }
                                                                                                                                    

                                                                                                                                    This is along the lines of cgenschwap’s recommendation of preferring parsing over verifying.

                                                                                                                                    1. 1

                                                                                                                                      You’d have to make sure the setters and getters did the checking as well along with making sure there was never any access to the internal value other than by the setters and getters. Which is a lot more faff than having it supported natively.

                                                                                                                                2. 7

                                                                                                                                  Aren’t you just talking about dependent types? I don’t think this is a situation where languages “fail” per se, but more that it is a technically difficult thing to do (while also having a language that is easy to use for regular programmers/programs).

                                                                                                                                  Dependent types are just an extension of types – so they are certainly a good solution to the actual problem!

                                                                                                                                  1. 1

                                                                                                                                    Dependent types is one way to do this, though so far I’ve never seen that built into a language and usually a very complicated way.

                                                                                                                                    But who uses dependent types? no languages really have them built-in, certainly no commonly available/used languages. Haskell, has Liquid Haskell and per pushcx smart constructors also does this, apparently. Are Smart Constructors built-in? It seems like they are, but my Haskell on this machine won’t compile it, but it’s also a really old Haskell, so that could be why.

                                                                                                                                    I agree bolting constraints onto our existing type system(s) is very complicated and messy.

                                                                                                                                    I’ve never seen any code out in the wild that does anything like this.

                                                                                                                                    We all know we can’t trust whatever the user gives us, and the giant list of security vulns just enforce the point, and yet basically no language in common use has any nice sane way to handle these sorts of issues.

                                                                                                                                    1. 3

                                                                                                                                      If dependent types interest you, the answer to “who uses dependent types” is idris

                                                                                                                                      1. 2

                                                                                                                                        It is unfortunate that few languages have dependent types – I completely agree with you here. However, putting constraints on user input isn’t really an issue in my opinion. You should always parse the info from the user rather than verify it.[1]

                                                                                                                                        Raw user input is stored in a Raw{var} type (which can’t be trusted) and is parsed (with constraints) into a {var} which can be used without fear. Of course, this doesnt guarantee the data is valid – your code might still screw it up – but this concept can just be used every time the data is mutated.

                                                                                                                                        Dependent types are fantastic because they turn this into a compile-time guarantee. Otherwise I don’t really see the benefit of having a language feature like this at runtime, it is very easy to do yourself.

                                                                                                                                        [1] https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/

                                                                                                                                        1. 1

                                                                                                                                          I think you and I are in complete agreement here about the end goal. I’ve read that post before, and while what they write is possible in Haskell, basically never used(from my very limited Haskell experience), in other languages it’s a giant PITA.

                                                                                                                                          No popular JSON parser will let you add constraints to your parsing of JSON(that I’m aware of).. The most they will do is sort of let you say this must be an int64, or this must be a string. Obviously JSON is just one example here. I’m sure there is one or two very generic parsers that allow for constraints at parsing time, but I’m not aware of any in popular languages that make adding actual constraints easy, or even possible.

                                                                                                                                          1. 1

                                                                                                                                            Certainly with strong, static typing this is easier to enforce (functions only take the parsed version of input). However every language can do this, it just becomes a matter of code hygeine. If JSON parsers dont let you add constraints then you just add another parsing step which can apply constraints.

                                                                                                                                            Sure this adds boilerplate and complexity, but I would argue that this is fairly easy to do.

                                                                                                                                    2. 6

                                                                                                                                      I’d argue that types are a terrible solution to the actual problem.

                                                                                                                                      I’d argue the actual problem is, we need to check variables in our languages…

                                                                                                                                      …we want age to be an integer between 0 and 120 or so. Certainly < 200. That’s what we actually want.

                                                                                                                                      Let me make sure that I understand what you’re asserting here: you believe that there’s only value in type systems insofar as they can allow for constraining numeric types to a bounded range? And, all languages with static type systems which fail to do this are fundamentally failures?

                                                                                                                                      1. 0

                                                                                                                                        No, Types are awesome for compilers, to make things go fast and optimize their implementation. There is a reason types got invented in the first place. It’s hugely easier for a compiler to optimize number math if they don’t have to concern themselves with if it might also be a string “happy”.

                                                                                                                                        I argue though that they are pretty terrible for end programmers, because they lack constraints. There is a reason Python and friends, who mostly ignore types are hugely popular.

                                                                                                                                        I only picked age as a very easy, hard to dispute example, but I believe constraints should apply to all data types, not just numeric types. pay period date constraints on a timesheet would be another easy use-case that doesn’t use numbers, but dates.

                                                                                                                                        PostgreSQL’s check syntax is what I’m after here, before anything can get stored in the table, it has to be verified for sanity. but PostgreSQL is the wrong place to do this, it needs to happen way out near the user, so it’s way, way easier to say “hey user, “first” is not a valid number in the range 0-100, since this is an input for age.” So when we go to store the variable in our program, we need to be able to parse it and constrain it and verify it’s sane before anything else can happen. We as an industry completely fail at this, for the most part.

                                                                                                                                        1. 3

                                                                                                                                          Yes and no. There is a lot of benefits to types, they help when writing code by pointing out bugs before the code is running (contrary to Python for example). Depending on the type model you even get theorems for free, i.e. you can prove theorems about functions just from it’s type signature! That’s pretty neat.

                                                                                                                                          Another problem is the overly structured data you’re proposing: What if I’m 106 and want to register for your service? What if I don’t have a first name? What if I don’t have a street address and live “2 mi N then 3 mi W of Jennings, OK 74038”? What if, in a family tree program, a person wants to input a child which he had with his daughter?

                                                                                                                                          I’m not saying no constraints is a better approach, but there are a lot of edge cases to be thought of, even for something as simple as an age or a name, which maybe shouldn’t be restricted at all.

                                                                                                                                          1. 2

                                                                                                                                            There are new things being added to types to make it better, I agree, dependent types one of them. Like the OP post says, many type systems are hard to reason about.

                                                                                                                                            I agree gross errors are caught trying to shove a string into an int8 variable. But academic papers disagree over how useful types are in practice, it’s mostly just yelling opinions across the internet these days, as far as I can tell. Perhaps over time, academic papers will have better science involved and eventually figure it out, but I think we can all agree the answer , scientifically speaking, is murky at best[0]. That said proving theorems sounds pretty neat!

                                                                                                                                            Of course there are lots of edge cases with data validation/parsing, the known issues here, are multitude. I don’t have a street address most of the time. That doesn’t mean we should punt and give up on the problem, and let everything be free-form text fields. :) It’s very, very hard to reason about free-form text fields.

                                                                                                                                            Every application will likely need unique constraints around particular data, depending on the context, but I think sane defaults can be found for many use cases. In my OP I covered this already. We can argue where the limits need to be, but for most applications I think we can agree negative ages, and ages over 150[1] are probably sane defaults.

                                                                                                                                            If you buy into the parse vs. validate debate[2], we can’t even currently parse with any sanity. OpenBSD mostly punts on sane parsers and just assumes they are totally broken by sandboxing the parsers from the rest of the application(sshd, tcpdump as examples). I also mentioned in a different comment about JSON parsers as an example.

                                                                                                                                            0: http://danluu.com/empirical-pl/

                                                                                                                                            1: https://en.wikipedia.org/wiki/Oldest_people

                                                                                                                                            2: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate

                                                                                                                                      2. 5

                                                                                                                                        Types, basically don’t do that, and practically every language ever written skips over any sane way to have limits on our variables besides basically int8, int16 and string.

                                                                                                                                        I believe that Ada handles this in a good way. You can declare an age type that has a valid range of 0 .. 120. I don’t have direct knowledge of this, but AFAIK it uses a combination of static and dynamic checking. Those cases which are impossible to verify with a static checker are verified at runtime.

                                                                                                                                        1. 3

                                                                                                                                          Pascal and Common Lisp are two other languages with this feature, just for completeness. They’re called “subrange types” in Pascal.

                                                                                                                                          1. 2

                                                                                                                                            Yep, Ada can do this. You can also use a regular integer type with contracts and use Spark (the Ada subset) to prove that the integer is in the range you expect it to be, thus guaranteeing that your program is free from runtime exceptions.

                                                                                                                                            Ada has some pretty cool features, it’s sad that nobody knows about them.

                                                                                                                                          2. 4

                                                                                                                                            You might find Racket’s contract system interesting. Also this talk if you have ~40 minutes (assuming 1.5x speed).

                                                                                                                                            1. 1

                                                                                                                                              Neat! So I agree in non-popular languages, this is sort of only recently being addressed, to some degree or other, with varying degrees of implementation. Most documentation ignores the problem. I’ve never used Racket, but Haskell is the other commonly known language that has this ability to some degree, yet I’ve never seen any Haskell code in production that uses any of the features available. Is using the contract system actively used in Racket?

                                                                                                                                              I went looking through github, seems Pollen is popular for something written in racket, and it uses ‘provide’! [0]

                                                                                                                                              It seems however, it’s limited to modules only.

                                                                                                                                              0: https://github.com/mbutterick/pollen/search?q=provide&unscoped_q=provide&type=Code

                                                                                                                                              1. 3

                                                                                                                                                Yes, the majority of Racket code uses contracts extensively and they are also used in documentation. Some examples:

                                                                                                                                                provide is just Racket’s way of exposing bindings from one module so that they can be required by another. The contract library provides the contract-out provide transformer that attaches contracts to bindings when they are provided. You’re right that when someone uses contract-out, then that contract will only be enforced at module boundaries. However, that’s not the only way to attach a contract to a value. The same library also provides define/contract among other things so you can do stuff like this within a single module:

                                                                                                                                                #lang racket
                                                                                                                                                
                                                                                                                                                (define smallint/c (integer-in 0 255))
                                                                                                                                                
                                                                                                                                                (define/contract (f x)
                                                                                                                                                  (-> smallint/c smallint/c)
                                                                                                                                                  x)
                                                                                                                                                
                                                                                                                                                (f 42)
                                                                                                                                                (f 1024)
                                                                                                                                                

                                                                                                                                                Running the above yields:

                                                                                                                                                42
                                                                                                                                                f: contract violation
                                                                                                                                                  expected: (integer-in 0 255)
                                                                                                                                                  given: 1024
                                                                                                                                                  in: the 1st argument of
                                                                                                                                                      (-> (integer-in 0 255) (integer-in 0 255))
                                                                                                                                                  contract from: (function f)
                                                                                                                                                  ...
                                                                                                                                                

                                                                                                                                                Or even stuff like:

                                                                                                                                                #lang racket
                                                                                                                                                
                                                                                                                                                (define smallint/c (integer-in 0 255))
                                                                                                                                                
                                                                                                                                                (define/contract (f x)
                                                                                                                                                  (->i ([in smallint/c])
                                                                                                                                                       [out (in) (=/c (add1 in))])
                                                                                                                                                  x)
                                                                                                                                                
                                                                                                                                                (f 42)
                                                                                                                                                

                                                                                                                                                That yields:

                                                                                                                                                f: broke its own contract
                                                                                                                                                  promised: (=/c 43)
                                                                                                                                                  produced: 42
                                                                                                                                                  in: the out result of
                                                                                                                                                      (->i
                                                                                                                                                       ((in (integer-in 0 255)))
                                                                                                                                                       (out (in) (=/c (add1 in))))
                                                                                                                                                  contract from: (function f)
                                                                                                                                                  blaming: (function f)
                                                                                                                                                   (assuming the contract is correct)
                                                                                                                                                

                                                                                                                                                It’s been a while since I’ve done any Haskell, but I don’t think there’s anything quite like this in the language. I definitely recommend watching the talk I linked to because it explains some of these ideas in a visual way and much better than I could do here.

                                                                                                                                                I believe Clojure is also adopting some of these ideas now via spec, but I haven’t looked into it.

                                                                                                                                                1. 1

                                                                                                                                                  <3 Thanks!! This looks very awesome! Go Racket!

                                                                                                                                                  1. 1

                                                                                                                                                    <3 Thanks!! This looks very awesome! Go Racket!

                                                                                                                                              2. 2

                                                                                                                                                I believe you’re looking for dependent types as in eg Idris: https://en.wikipedia.org/wiki/Dependent_type