1. 6

    Waiting for @andyc comments on JS vs. Shell.

    1. 7

      I think there’s definitely a place for internal DSLs in every language, but we also need a new Unix shell :)

      http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#what-about-this-shell-like-library-embedded-dsl

      1. 7

        I want to chime in with a somewhat different opinion. Not entirely contrary, but with some more detail that I hope will be useful for people to think about the tradeoffs.

        To start, let’s agree: your link to your site mentions that the shell is a low-level component that needs to be available with few dependencies. There are various things you want to do with shell scripts with absolute minimum dependencies, like early boot scripting or user-level scripts to install your basic environment dependencies (eg. to install another shell!). And we absolutely should have a better shell than Posix Shell or Bash for this stuff.

        However, I think most shell scripts don’t need to be so spartan, and for these cases an embedded shell language can be much more powerful.

        Note that not all embedded shell DSLs are the same or even in the same category. This zx looks like it’s in the category of Plumbum for Python or Scsh for Scheme. With these, you have a library to easily write shell commands or pipelines inside a host language, but I wouldn’t consider these “shells” as much as “shell libraries” because they really aren’t suited for interactive use. In other words, nobody really fires up Python with Plumbum (or Scsh) to play around in the shell and find the solution, then copy/paste it to a script. Instead, they start directly at the script level. Thus these libraries don’t capture the full shell scripting process that people commonly use. This is why “just start with python rather than writing a shell script” often doesn’t happen. You really want that interactive first step for many things, and translating interactions to another language really seems to be a hurdle (especially when eg. you are interacting with Bash but some features, eg. maybe globbing, don’t translate directly or easily to the embedded DSL).

        Other embedded shells, like Rash in Racket (my project) or Xonsh in Python, are a much more thoroughly modified DSL designed with live shell interactions in mind. For example, they use custom parsers to have an interactive-friendly terse syntax more like Bash while being able to escape to the host language (though the quality and flexibility of the embedding varies). This much more fully captures the spirit and lifecycle of shell programming – you can iterate with commands live in a repl, copy them to a script, and grow the script from there. However, with an embedded shell DSL, when you start growing the script you have the whole host language available. Not only does this include core language design that’s done poorly in popular shells (eg. control flow, error handling, data structures, etc), but you get the entire language package ecosystem (this is where Xonsh really shines) and any advanced features that a stand-alone shell language wouldn’t have (eg. Rash gets to use Racket’s macro system, interoperates with Typed Racket and other languages embedded in Racket, has first-class delimited continuations, etc). This allows scripts to grow gradually (without a full rewrite) not only past the “Bash has impoverished data structures and is full of landmines” stage where most shell scripts are scrapped, but also past the “this part would be more {efficient, complete, clean, etc} if done natively in a general-purpose language” phase where some shell scripts need to graduate to become “real” programs.

        Current embedded DSL shells have various shortcomings. Eg. while Rash is designed to be a language for live interaction, it desperately needs a new line editor to even begin to compete with Zsh or Fish on interactive completion, history management, or even basic command editing. (Completion is a common problem for shells, and there have been some discussions that @andyc has hosted about building re-usable tooling for it.) Embedded shells tend to lack a lot of polish because they are all just hobby or grad school projects. (If I were financially independent I could focus on Rash full time for a few years and really polish it up, but alas! Maybe after I graduate and work in industry for a while.) Xonsh is probably the most polished for interactive use, though I think it’s weaker than some other embedded shell DSLs in other respects. (Not to disparage Xonsh – it is one of the oldest in its class and later ones had the benefit of hindsight.)

        Despite their current issues, I think the brightest future for shell programming is high quality embedded shell DSLs. In the long term, or at least in some idealized long term where sufficient manpower is given to both stand-alone shells and embedded shells, stand-alone shell languages won’t be able to compete with embedded shells in terms of features, libraries, customizability, etc.

        (Just to be clear about the thread’s original topic, I think zx looks cool an useful, but it is clearly not an attempt to be this higher level of embedded shell DSL I’ve discussed here.)

        1. 2

          Hm interesting, I do see your point about the difference between between shell libraries and internal shell DSLs. Although I use neither of them so I guess I don’t have much to say about them!

          I also haven’t encountered them “in the wild” very much; when I look at open source projects I tend to see shell scripts. For example almost all the docs and examples for Github Actions or Travis CI is shell-in-YAML.


          As for whether embedded DSLs are the brightest future of shell programming – I think it depends on the definition of “shell”. It’s become very clear to me that not everyone uses it the same way or thinks of it the same way.

          Some people mean “an interactive REPL” and actually Jupyter notebooks more or less qualify as a “better shell” for them.

          For me, shell is the lowest common denominator between languages, and it deals directly in kernel concepts like byte stream and processes, without abstraction.

          I use it to glue together Python and R, R and C++, Python and JavaScript, etc.


          I had a long exchange with @xiaq, designer of Elvish a few days ago, on the architectural ideas of the Perlis-Thompson Principle and the “narrow waist”:

          https://lobste.rs/s/ww7fw4/unix_shell_history_trivia#c_epslyn

          This probably needs to be in blog posts and not lobste.rs comments, but the claim is that each language is defined by its core data structures.

          • POSIX sh: everything is a string
            • bash: everything is a string, with arrays as secondary
            • Oil: everything is a string, with JSON-like data structures as secondary
          • Elvish: Everything is a JSON-like value, which are primary. These values can travel directly over pipelines without serialization.
          • Nushell: Everything is table.
          • Rash: I haven’t used it, but presumably everything is a cons cell like in Racket?

          If you define shell as the “lowest common denominator”, then my claim is that sh/bash/Oil is actually the lowest common denominator between an Elvish, Nushell, and Rash script :)

          I’m not necessarily making a claim about which is better or which is easier to use. I’m just saying it’s a fact of software that there are divergent mental models and meta-models, and the lowest common denominator will always be literal operating system concepts like byte streams and processes. At least until something else supplants Unix as the narrow waist of essentially all computing :)

          So that is my view on the future of shell programming. Basically computing is being applied to more areas, and used by more people. So it’s getting more diverse and will never consolidate. That even includes more shell-like languages that needs to be bridged :) And it includes C++, Rust, and Zig. And Matlab and Julia, etc.

          This post has a lot of color on those views, with a bunch of slogans and fallacies (shell is the language of diversity and heterogeneity; processes aren’t slow; programming with strings isn’t slow, etc.)

          http://www.oilshell.org/blog/2021/07/blog-backlog-1.html


          As an example of the difference, I’d ask the same question I did of Elvish. How do you parallelize Rash functions with xargs -P 8, or is that not idiomatic?

          I have another “alternative shell challenge here”: https://www.oilshell.org/blog/2020/02/good-parts-sketch.html#the-alternative-shell-challenge

          This is not to say that other definitions of “shell” aren’t valid. I’m only saying that we do need exactly this kind of shell. We need a narrow waist that’s a tiny layer on top of the kernel.

          1. 2

            Lack of adoption so far doesn’t mean embedded shell DSLs aren’t a good idea. Every good idea that has become popular (electricity, personal computers, the internet) initially had no users.

            Lack of adoption of embedded shell DSLs may perhaps be because there are relatively few of them, they are relatively unknown, most of them are relatively immature, and people don’t trust that they will be maintained in 5 years? These are just growing pains of a new technology. If there is continued work on high quality shell DSLs we can move past this phase.

            That said, I have seen libraries like Plumbum used in various places, and Rash has a few users (including obviously myself). I haven’t seen anybody using Oil shell, but that doesn’t mean it’s a technological dead end, or even that there really aren’t people using it. Over time I hope to see more people using Rash and Oil.


            I agree that the future of “shell” is divergent, and there is room for Oil, Rash, Jupyter notebooks, and more. We probably need better ways of discussing “shells” to help people understand the differences and tradeoffs between these different visions, and which one(s) are more suitable to their goals.


            Rash: I haven’t used it, but presumably everything is a cons cell like in Racket?

            Everything is not a cons cell in Racket. Racket of course has cons cells, but also dictionaries, and various kinds of atomic data (numbers, strings, …), and procedures, and user-definable structures that can be properly encapsulated with APIs, generic interfaces, and higher order contracts… So values in Racket/Rash can be whatever kind of rich data structure you want. And of course Racket has libraries for JSON, CSV, XML, etc, so you can communicate with things that read or write those formats.

            Now, when sending data to a subprocess you probably care about serializability, and you can’t serialize things like procedures or continuations. However, Rash pipelines can have subprocesses as well as Racket procedures, so pipelines with Racket procedures don’t have that limitation. Also, Rash has arbitrary user-definable “substitutions”. IE Bash has substitutions like $(...) (replace the argv element with the result of this code), and <(...) and >(...) (replace the argv element with a path to a file descriptor for this pipeline). One of the demos of user-definable substitutions I made is closure substitution, where the program gets a string that is a path to an executable script written on-the-fly that connects back to the original script via socket so you can use Racket procedures with programs like find -exec.

            (Substitutions are yet another language feature that should be user-definable and extensible. Maybe someone wants temporary directory substitution, or mount point substitution where you mount a virtual file system for the duration of a command, or… you could take this to silly extremes, but user-extensibility is important. This is basically the same argument that a language should have a good macro system so a user can build a for-each loop on top of a regular for loop without waiting for language designers to drag their feet. Of course, the main difference between substitutions in Rash and just using a procedure to compute what the arguments should be is that substitution objects get a clean-up hook.)


            I agree that a “narrow waist” shell that just supports the raw OS primitives should exist. However, I disagree that a large percentage of what I consider “shell scripting” should be done in that shell. Abstraction is powerful, convenient, and helpful. Just like most programming has gradually moved to higher-level programming languages with only specific things remaining firmly in the domain of “use assembly” or “use C”, I think many visions of shell scripting are better served by a powerful, generous language rather than a “lowest common denominator”.

            To me, “shell languages” are mostly about live interactions and automating things. Eg. getting programs to work together, finding ways to do tasks at the command line then saving that in a script if you want to do it again, etc. The idea of using the lowest common denominator tool, or the purity of encapsulating specifically the “narrow waist” of Unix, are generally of little concern or even anti-values. If higher-level tools with richer abstractions make it easier to live code, automate, and maintain automation, they are probably better choices most of the time. I want to use a powerful tool that lets me easily and powerfully interact with my environment, and that lets me write a script and get on with other things (but then also lets me come back and grow my script when I realize that I need to improve that automation).

            Live coding and automation aren’t my whole view of shell programming. Eg. I think it’s also often for quick prototyping (which I think is Rash is well suited to), and about making one-liner alias-like command wrappers (which frankly I don’t use Rash for because it has a slow start-up), and lots of other things, some of which would definitely be better in something like Oil rather than something like Rash, or perhaps in a Jupyter notebook, etc. But I think things like automation and prototyping are very significant chunks of shell programming that benefit more from rich, flexible, powerful languages rather than languages that are trying to be a lowest common denominator. And I think embedding a shell into a host language that is already designed to be flexible and powerful with strong abstraction mechanisms is the easiest and most reasonable way to have all of that in a shell, and the only way to provide a gradual upgrade path for scripts that over time need to become more fully featured programs suited to a general-purpose language.

            The live coding part is still in many ways better in Zsh or Fish than in Rash, but that’s for lack of time and effort into a line editor, not because writing command completions and interactive widgets in Zsh is a better idea than writing them in Racket. I think that should be pretty obvious to anybody who has tried to write shell completion functions or Zsh widgets. One beauty of embedding in Racket is that eventually one could write an emacs-like line editor that can have editing functions (including completions, interactive searching/narrowing widgets, etc) written in Racket, or any Racket-hosted language, including DSLs tailored specifically to writing completions or interactive widgets.


            How do you parallelize Rash functions with xargs -P 8?

            You can use closure substitution (mentioned above), along with Racket’s (honestly somewhat poor) means of parallelization.

            Eg. xargs -P 8 (closure-substitute my-racket-function)

            Or you could pipeline the data directly to a racket function that handles parallelization rather than going through xargs. You could write it directly or make an xargs-like function inside Racket that parallelizes a function that you pass it, reading input from a pipeline line by line and processing it. I would probably do that instead of using an xargs subprocess to paralellize code that I’m already writing in Rash/Racket.

            Eg. data-generating-command |> parallel-xargs-in-racket #:jobs 8 my-function

            That said, parallelizing a shell function with xargs -P 8 is something that, I believe, most shells including Bash and friends can’t do. Does Oil have an xargs builtin or something like a fork server substitution that sends xargs a path to a program that forks the shell to service subprocess calls?

            (A user could write a fork server substitution in Rash, though I think Racket isn’t necessarily fork-safe in all situations. I’ve used fork in Racket programs, but I don’t actually recommend using fork in Rash. Actually, I think using fork directly is generally an anti-pattern, though sometimes it is the only way to do certain things in Unix. If you really need to fork, Rash may not be the right tool. Again, this is about different visions of what shell programming is – in any shell script I’ve ever written or worked with, use of fork in the shell was an implementation detail, not a critical necessity. For those to whom “shell scripting” is all about using fork in specific ways, Oil is probably the better fit. That said, in a fork-safe language, or if Racket exposes a user-level way to fork with a guarantee of safety, an embedded shell DSL might still be the right choice there.)

            Oh, now I look more closely at your post and see your $0 dispatch pattern. Yes, Rash can do that. And… any language where you can access the full argv parameter and get the path to the current executable should be able to. However, $0 dispatch can’t capture dynamic state like closure substitution can. Closure substitution allows you to use higher-order and stateful functions with things like xargs and find.

            I think Rash performs quite well against “shell challenges” like you post above (it can do both the xargs challenge as well as the challenges linked in the blog post). If you’ve looked at a bunch of embedded shell libraries and formed an opinion about Rash by lumping it in with them without much inspection (not an irrational thing to do), Rash is much more powerful and flexible than you would assume. I looked at the related work and made real improvements. It’s not perfect, and there are mistakes I want to correct in future versions. And there are a few limitations of the platform I haven’t yet fixed (eg. I’ve started a patch to Racket necessary to allow Rash to include interactive job control, but I haven’t gotten around to finishing it. Maybe another improvement should be a general user-level safe fork, which I could use instead of Racket’s process spawning API to implement job control and be able to do all low-level Unix stuff with processes. This would lose the benefits of the portability of Racket’s process spawning API, though, and I like that Rash works on Windows.). But with time and work all these issues can be fixed. They aren’t intrinsic limitations to the idea of an embedded shell DSL.

            While you could come up with some challenges that do some specific low-level adjustment to how a process runs that Racket’s process execution API doesn’t cover, showing that Rash isn’t the best fit for certain low-level Unix-specific things, you could also come up with high level abstractions that are useful for live interaction and automation that low-level shells can’t do. Again, different visions of what the shell is and should be.


            As you argue, there is room for both our shells and more. I hope people can embrace nicer things than Posix shells for the various different visions of what “shell” means. My main hope in writing all this is that people don’t write off the idea of embedded shell DSLs because there have been so many failures and flawed implementations. There are very real and serious potential benefits of using embedded shell DSLs instead of stand-alone ones! There are also potential downsides, but the status-quo is generally that people use stand-alone shells when an embedded one could be better, not the reverse.

            1. 1

              Thanks for your thoughtful comments. Short reply below, but on a different forum we could go deeper into this. Everyone is welcome to join https://oilshell.zulipchat.com and I’ve had conversations with other shell authors there.

              I agree that a “narrow waist” shell that just supports the raw OS primitives should exist. However, I disagree that a large percentage of what I consider “shell scripting” should be done in that shell. Abstraction is powerful, convenient, and helpful.

              This is a long argument, but I’d say I’m more concerned with composition than abstraction, because it makes programs shorter and more flexible. Shell functions are obviously a form of abstraction, but to me the key property is that they compose with external processes (e.g. the xargs example and the redirect example).

              Pipelines are not particularly abstract, but they compose very well.

              I would also say that certain forms of abstraction can inhibit composition.

              The first two posts here are about composition more than abstraction I’d say, although it’s not either-or:

              https://www.oilshell.org/blog/tags.html?tag=shell-the-good-parts#shell-the-good-parts

              Also, a key point is that the old stuff has to compose with the new stuff. I think a lot of alternative shells suffer from a bunch of new concepts that don’t compose with the OS primitives and lead to longer programs with more edge cases.

              The idea of using the lowest common denominator tool, or the purity of encapsulating specifically the “narrow waist” of Unix, are generally of little concern or even anti-values.

              It has to be elaborated on the blog more, but the narrow waist is a practical architectural concept that reduces the amount of code in a system and makes code shorter. It’s desirable to aim for O(M + N) amounts of code vs. O(M * N) amounts. A real example of this is that there are R-Python bridges, and I’ve seen people try to write R-Java bridges, etc. There is a clear O(M * N) problem there that is addressed by simply using CSV (or Oil’s QTT).

              If you’ve looked at a bunch of embedded shell libraries and formed an opinion about Rash by lumping it in with them without much inspection (not an irrational thing to do), Rash is much more powerful and flexible than you would assume.

              Yes I’d like to read more about it. I did read the Rash paper a few years ago.

              To me the key benefit of the embedded shell DSLs is for people who know that language. That is, JavaScript programmers might prefer zx, and Python users might prefer plumbum, etc. for obvious reasons. It’s hard to learn a new language.

              So I think you are arguing that there are strong reasons to use Rash even if you DO NOT know Racket?


              Other Notes

              • Oil is a rich language with convenience, but it is kind of designed “around” the narrow waist, rather than trying to replace it with something else. As mentioned, JSON-like data structures are secondary to byte streams, and I believe that’s a feature and not a bug. I know pretty much everyone disagrees on first glance, so that’s why I’m devoting so much space to this on the blog.
              • I do think the language runtimes are an issue as well; I remember reading that the Shill shell was originally written in Racket but they started to write their own intepreter when they wanted to productionize it. (I haven’t kept up with it though). One of my blog posts also links to a discussion about the Go runtime and fork().
              1. 1

                Re: O(M + N) vs O(M * N)

                Yes, this is an important point. And if you are creating an embedded shell language within every general-purpose language, it is clearly an O(M * N) situation. Maybe only a few languages will end up with good embedded shells. But a shell embedded in any language can still be the glue language for programs written in any language.


                Re: Narrow Waist

                The Narrow Waist of Unix is too narrow. Lisp Machines had some much better ideas here, and PowerShell shows what some of that can be for a shell. Of course, .NET is not the narrow waist I want either, and PowerShell is basically useless outside of Windows. But it has some great ideas.


                Re: composition vs abstraction

                The key strength of embedded DSLs is that they compose better with their host language than they would as separate DSLs. I think embedded shells are a win for composition and abstraction.

                There are limits to composition where you have rich data and maybe want to put it through a Unix pipe. But this is just dealing with the fact that the Unix waist is too narrow. So you have to hack around it by (potentially lossy) serialization, by passing functions by $0 reference or closure substitution, etc.

                You mention that Oil is rich but designed around the narrow waist. I would say that Rash is designed to be a language that embraces a much wider waist, but that gracefully cooperates with the Unix waist where necessary. IE it’s designed so that one could write an ls in Racket that returns the listing as objects rather than strings, then the next pipeline stage could filter based on attributes of the objects like PowerShell, etc, but also has special consideration for Unix subprocesses and byte-stream pipes. While I can dream of a future with modern Lisp Machines, I live in a world with Unix. I designed Rash with both in mind.


                So I think you are arguing that there are strong reasons to use Rash even if you DO NOT know Racket?

                Yes, basically for 2 reasons: (1) because embedded shells have advantages, and Rash is, in my opinion, currently the best design of an embedded shell that can provide those advantages, and (2) because of strengths that arise specifically from using Racket. People who know Python and not Racket can pick up Xonsh more easily to get some of #1, but they will miss out on #2.

                #2 is sort of a double reason, though. There is currently no other language that can host an embedded shell as well as Racket due to various language features that are unique to Racket (or relatively rare but found in Racket). These features are important to the implementation of an embedded shell that make the integration tighter and cleaner, as well as providing power and flexibility to live interactions and scripting.

                (The most important feature is Racket’s macro system, which is truly world-class and far ahead of any competition.)

                Ultimately encouraging someone to learn Rash implies encouraging them to learn at least a little bit of Racket, because you can’t really use Rash without some basic knowledge of Racket. Or in other words, if in Rash you want to use a data structure, or an if expression or loop, you have to use Racket because Rash just punts everything to Racket except a few features that are specific to shell programming. But I argue that features of Rash, features of Racket, and the synergy between them, provide things for shell programming that you can’t currently get elsewhere.

                I do think the language runtimes are an issue as well.

                Yes. While Racket is world-class in some ways, it also has weaknesses. The worst one for Rash is probably the slow startup time. Racket is reasonably fast once it’s running (eg. it’s faster than Python, which isn’t saying much. Scripts that I’ve ported from Bash to Rash end up being faster if they aren’t dominated by startup time.). But the startup is painful for scripts that are otherwise fast. I really want other languages to start taking key features like macros seriously so they can compete. Right now Racket really stands alone in its advantages.

                I haven’t followed Shill that closely, though I read up on it several years ago. I know their first hurdle was that Shill required a custom kernel module that was only available for FreeBSD (I think). Shill is going in a very different direction than Rash. It is much more concerned with security and information flow, and much less concerned with other aspects of programming expressiveness or convenience. Shill issues with the runtime I think mostly revolve around the fact that they needed fine-grained control over (custom) OS security stuff while not benefitting very much from the expressiveness of embedding in a host language (with respect to their core goals).

                That said, the runtime can also be helpful. If you are writing a shell language from scratch you have to write all the runtime stuff yourself. With Rash I got to just lean on Racket. Thus, Rash’s implementation is pretty small, just a few thousand lines (about half of that is the core library for pipelining and stuff, which can be used by itself like a Python user would use Plumbum, and half about macro stuff and parsing for the interaction-friendly syntax). If I write the line editor that I want to, I’m confident it will be much bigger than the entire DSL implementation.


                It’s hard to learn a new language.

                And whatever shell you learn is… another language to learn. An embedded shell gets to piggy-back. If you already know the host language, there is less to learn. If you don’t already know the host language, you need to learn some of it, but then you are learning a shell and a general purpose language for the price of one (or maybe 1.5).

                I think embedded shells have a good story here for education. I think we both agree, and I think this phrase that I often use is actually a near-quote from you or another alternative shell author, that “it’s hard to tell someone to learn Bash with a straight face”. The shell is so useful, and unlocks so much practical power for people to harness. But not only is it yet another thing to learn, but Posix shells are particularly bonkers, full of weird gotchas for a learner (and even seasoned users) to trip over! This is a real problem – I see plenty of people graduate with a BS in CS, but with no ability to use the shell because it’s a weird thing that is pretty big and doesn’t really fit well in the school’s curriculum. These students have been kept in the dark about one of computing’s most empowering secrets that lets people, with relatively minimal work, write programs that actually do things they care about! (I could go on about my love for the shell and how empowering it was to learn it. I’m probably preaching to the choir here.) Now, a well designed modern shell of any type could improve the situation. But an embedded shell in a language the students already learn? It could help a lot.

                Now, Rash is pretty complicated, to be honest. It has a lot of features that are there because they make it more powerful, but aren’t exactly pedagogical home runs. One thing I like about Racket is its simplified student languages. If you aren’t familiar with them, Racket has a series of simplified languages that remove various features. This allows the student languages to have better error messages and make it easier to learn both the syntax and semantics of the language in gradual steps. IE you can’t accidentally use some syntax for a more advanced feature and therefore get some error message that you can’t understand. I hope to some day make a “Student Rash” language that is less flexible, but provides an easy way for a student to learn the key concepts of shell programming as a “diff” to the programming they already know. Students could learn about processes, argv, pipelining, etc, but not need to learn yet another form of conditional, loop construct, syntax for the data structures they know, etc. They can keep using all of the things they know, in exactly the same syntax, and just learn the new concepts and syntax for running pipelines.


                but I also want to encourage you to write a blog!

                I do have a blog, but sometimes I feel more motivated to write when its in response to some specific question or comment. I have a lot of blog posts (about Rash and other things) that I’ve been meaning to write (along with a revamp of Rash’s documentation), but… I guess I’m more prone to putting off writing when there isn’t a specific person I’m responding to right now. Anyway, I do intend to write a bunch more about Rash and shell thoughts in my blog and in Rash’s documentation and introductory material. At any rate, I’ll file away my comments here as a starting point for writing future blog posts that explore the ideas more thoroughly (and hopefully more clearly).

                That said, besides bombing lobsters threads about shells and using it to write all of my shell scripts, Rash doesn’t have a lot of my attention for the moment. I’m trying to wrap up my dissertation and find a job, and I’m not sure when I’ll have much time to focus on shells again. Hopefully soon. We’ll see. When I do I’ll definitely take another good look at Oil and other shells for good ideas.

                Also, thanks again for maintaining such a great catalogue of shells on your wiki. It’s such a great reference to keep up on the shell world and what other people are doing.

              2. 1

                Sorry, my posts always end up really long when I’m talking about shell stuff. I should have edited that to be shorter. But I have a lot to say about shells…

                1. 1

                  No problem, I will respond to this, but I also want to encourage you to write a blog! I would like to see how others are approaching their designs.

                  In particular I would like to know how a Racket-based embedded shell language compares with a Clojure, Common Lisp, or Elisp one.

                  And how those compare with Python/JS and Haskell/Ocaml ones etc.

                  I’m sure there is a lot we can learn from each other – I have read almost everything I can about shell, but there is still a lot missing IMO

              3. 1

                Somehow I missed that oil has json-like structures as a feature. With json being used more and more to glue languages together, I would imagine this fits well with your notion of shell as a glue language.

                Certainly shell is used this way, but is that its purpose? It seems to me like shell and kernel are related words, the shell being a low level user interface on the outside, and the kernel on the inside. If shell is to be simple and powerful, then perhaps usefulness as a glue language emerges as a side effect.

                1. 1

                  Well shell has many purposes and people have many viewpoints on it, as I mentioned in some long comments on this thread.

                  But while something like Jupyter notebooks could replace some use cases for a shell (not for me but for many), I’d say that the glue / narrow waist / least common denominator role is unique to shell.

                  Evidence I retweeted today :)

                  https://twitter.com/ozaed/status/1428421032096960516

        1. 1

          This article describes how DesqView works with XMS, but that implies a 286+ with 1Mb+ of memory. How did it perform task switching on an 8088 with 640Kb? I thought the MS-DOS task switcher was capable of swapping to disk, but that’s only feasible because it’s a full screen task switcher so having a noticeable delay while switching tasks is expected - a windowing environment presumably needs to be able to switch tasks in less than a second.

          1. 2

            Good question… memory management options on DOS machines is a topic that deserves a full post (I’ve put it on my list), but I probably elided a little too much from this message. A quick rundown on the topic: the 8086 thru 80186 only had a 20-bit address bus and so could map 1MB, a good chunk of which was “reserved-ish” for hardware purposes, leaving 640k. The 286 had a bigger address bus and could support 16MB, the 386 even more so, but this matters less than you would think in practice because virtually all DOS software at this point ran in real mode without the ability to use any of that extra memory. The solution was (skipping some complexity here) a device driver that used the MMU to remap sections of the past-1MB memory down into the real mode address range for programs to use. And that’s what we usually call XMS, although there were other conventions as well.

            That makes sense, but what about, say, the 8088? Well, it turns out the solution is basically the same: remap “extended” memory into the 640k standard range. But since the 8088 has only 20 address wires and no MMU, it can’t be done onboard. Instead, it was done off-die by an expansion card that basically intercepted the memory bus to remap different sections of expanded memory into a 64kb “window” in the real mode memory space. Running software had to hit specific interrupts to prompt the expanded memory card to change the mapping. Such controllers on ISA cards were pretty readily available in 4MB and 8MB sizes, and while I don’t know this for sure, I’m making a pretty confident assumption that DESQview on 8088 required one.

            The standards for these get confusing due to competition and fast change… EMS (extended memory specification) is typically associated with these 8088 systems, but I think there were at least a few other less common standards, and IBM had XMA which you could say is the ancestor to XMS. In late pre-286 machines there was sometimes an on-motherboard EMS controller that almost felt like an MMU. But in short, an “expanded memory manager” defined broadly was software on the 80386 but hardware on the 8088. Further complicating things a lot of 80386 memory managers could emulate the old-style hardware memory page switching so that you could use software written against EMS. These didn’t play nicely with each other so you might have to manually choose to allocate certain memory ranges to either extended memory or expanded memory, and trying to keep “extended” vs “expanded” straight kind of typifies how annoying this must have been for users. I got a desktop technician certification on the very trail end of any of this being relevant and remember serious headaches over all this.

            DESQview was also limited on non-80386, although I’m not sure of the exact issues. I know for example that later versions of DESQview were capable of windowing direct-drawing programs (e.g. vga raster programs) on the 386 by use of the MMU to remap the framebuffer, but could only run such programs full-screen on earlier processors.

            1. 2

              For the top of the third paragraph, you mean expanded, right? (EMS == Expanded, XMS == Extended?)

              XMS worked by throwing the CPU into protected mode so it could address more memory temporarily, then thunking back to real mode. That’s what HIMEM.SYS was for. No remapping is needed here. But as you suggest, this isn’t possible on an 8088, since it didn’t have physical addressing.

              Remapping memory down into real mode was possible on the 386, which had something resembling an MMU. That’s what EMM386 was for, which could use the CPU to implement EMS. As far as I know the 286 didn’t have enough support for remapping, but still had protected mode so could still use pure XMS.

              Hardware EMS existed but never seemed widespread. It was a way to upgrade an 8088 system without replacing it. But by the time 1Mb+ of memory started to be desirable, there was no reason to get an 8088, so at least in my neighborhood, these were rare. I had an 8088 laptop with 128Kb of EMS, but was never able to use it due to lack of a driver (sigh.)

              The way I read their literature, DESQview supported a pure 640Kb 8088 environment, no EMS. Quarterdeck were the masters of PC memory management in their day, and they may have had a lot of tricks I don’t know about. Obviously it’s possible to physically copy memory around within 640Kb, but because that memory is effectively a stack, the only way I can imagine this working is to firstly reserve a swap area upfront then allow applications to run in the remainder, and applications can then be copied from the remainder to the swap area. But that means the size of this region has to be statically provisioned before any application can run, which seems gross. Putting the swap area at the other end of RAM is dangerous since DOS can’t describe it as allocated and an app would just grow into it. Since it’s already a TSR, perhaps they intercepted calls to query memory size and just lie, and hope that programs won’t inadvertently use unprotected but theoretically unavailable addresses?

              1. 1

                Putting the swap area at the other end of RAM is dangerous since DOS can’t describe it as allocated and an app would just grow into it. Since it’s already a TSR, perhaps they intercepted calls to query memory size and just lie, and hope that programs won’t inadvertently use unprotected but theoretically unavailable addresses?

                This doesn’t sound right. DOS must know both where the beginning and and of allocatable memory lies, otherwise, it might try to overallocate on a computer that has only 512KB instead of 640. I recall that if you had a VGA card, there was a way to convince DOS to use video RAM beyond 640KB all the way to b800h where text mode memory starts, for an extra 96KB of conventional memory.

                1. 1

                  DOS must know both where the beginning and and of allocatable memory lies, otherwise, it might try to overallocate on a computer that has only 512KB instead of 640.

                  Right, I agree - that’s what I was trying to say in the last sentence. It has to know the end of memory, but has no way to describe randomly allocated regions within the conventional area because it was single tasking so memory was allocated as a stack. I don’t know what this call looks like and what semantics it has though. Since application memory access isn’t really controlled by DOS, all this can be is a guideline though - a program can just access an address and see what happens, and if they do, they’d trash anything that’s there.

                  Edit: Another possibility - did DOS start allocating high addresses and work downwards? This would seem logical and would make using those addresses above 640Kb relatively straightforward by just loading DOS at the desired address and applications don’t change. But if that happened, it really would be impossible to allocate memory at the “other end”, since the end of memory is a well known value (zero.)

                  I recall that if you had a VGA card, there was a way to convince DOS to use video RAM beyond 640KB all the way to b800h where text mode memory starts, for an extra 96KB of conventional memory.

                  I didn’t know this was possible with VGA. I remember doing it with CGA under OS/2, even to the point of running real mode Windows with 700Kb of conventional memory, although CGA is … CGA.

                  Edit: Wikipedia suggest that this was possible on MDA/Hercules/CGA, but not with EGA; didn’t VGA include EGA compatibility?

                  1. 1

                    Memory for Hercules graphics is at B000h, so you can safely map memory into A000h-AFFFh.

                    Color text mode memory is at B800h, so you can map memory below that as long as you stay out of graphics mode. That’s true for CGA, EGA, and VGA.

                    What I remember doing (and I could be mistaken on this; it was a long time ago and I was in high school) is using VGA video memory at A000h while in text mode. But maybe I remember wrong and just mapped memory there with EMM386.

                    To get DOS to use that memory as conventional memory you also have to relocate the 1KB EBDA.

                2. 1

                  Oh yeah, I definitely confuse expanded and extended memory at least a couple times every time I bring it up. That brochure is a great find, I too am surprised that it could run on an 8088 without an Intel Above card or something. I’m probably going to spend some time messing around with DESQview myself in the future because I want to understand the later features better, so maybe I’ll figure something out… I can see moving around in the 640kb but that would be really risky. It’s possible they relied only on disk swapping in that case. I previously wrote about Visi On for which disk swapping was arguably a major weakness, but this was a few years later and it was probably an easier sell.

            1. 2

              I wonder how feasible would be to bypass the pseudoterminal choke-point entirely, and just integrate terminal and shell directly?

              1. 3

                It’s trivial in terms of MVP (just a repl where e is string split + Command::run), moderately annoying in terms of UX feature parity (using fish gives you autosuggestions, customizable prompt, and other things you’d have to implement yourself otherwise), and rather challenging in terms of ecosystem adoption (people use vim, readline, ncurses).

                But yeah, terminal/shell split generally makes no sense today, I hope they add some kind of “built in” shell in the future, and generally fulfill my new shell program :-)

                1. 1

                  I would rather have a well defined set of escape sequences that the terminal can interpret for non-traditional formatting. For example, what if there were an escape sequence for an expandable bulleted list? Many programs could take advantage of something like that; why should the shell be special?

                  1. 1

                    Yes, there’s no reason to not define an IPC mechanism for that. Though I’d use output mimetypes rather than escape sequences.

                2. 1

                  Isn’t that kinda tmux?

                  1. 1

                    It also uses a pty.

                  2. 1

                    ioctl.

                    1. 1

                      eshell is maybe reasonably close, if you consider emacs as your terminal

                    1. 15

                      The post says performance is a key requirement, but never once mentions the word “latency”, which is what actually matters in a terminal. fps is completely irrelevant since a blur of text is totally unusable anyway, so you lose nothing by just discarding some of the display. But a terminal that doesn’t respond swiftly to user input is very frustrating compared to one that does.

                      1. 5

                        It’s not completely irrelevant. I appreciate that even rxvt-unicode can, as I’ve found out, do 60 FPS, since I got to have a fairly responsive spectrum visualiser (although going that high imposes a high load on the X server). It’s in no way a blur—60 FPS is over 16 ms, even the slow panels for photo editing can do that.

                        1. 2

                          I agree that latency is the most important performance metric for a terminal (specifically upper percentile latency, since a terminal with good best case latency will still feel sluggish if the tails are long).

                          It’s also good to know that a terminal is capable of processing updates quickly; a terminal that has good round trip latency (from keyboard input to a character being displayed) can still feel sluggish if I cat a 64KB text file and don’t get a terminal prompt back for many seconds.

                          1. 1

                            Oh of course, that’s part of the responsiveness. But you don’t have to actually render to achieve that… in fact, insisting on actually rendering everything might even be harmful depending on how you do it. Input responsiveness is still what’s actually important: if you ctrl+c in the middle of it, that should happen instantly.

                            My terminal vs xterm for example, doing the same thing: cat mylib/*.d, 192,000 lines of code being dumped. My terminal does it in 1.5 seconds, with ctrl+c being instant at any point in it. xterm does it in 4 seconds. My code actually renders an individual frame more slowly than xterm… but xterm tries to render it all, so it wastes my time showing a blur of text. Mine just detects that it is spammed output and only actually draws a few representative frames. It redraws periodically just to indicate to me that it is still working, but it doesn’t waste time drawing every detail that are impossible to see anyway. (I concede there might be exceptions where you’re doing like an ascii art animation in the terminal, where my approach is probably sub-optimal, but I’ve never seen it relevant myself)

                            rxvt does the same thing in 0.2 seconds, so they do an even better job :) I’m not sure if they just frame drop too or what though, they could legitimately be much faster.

                            But anyway I guess my main point is you can afford to drop a lot of frames and deal with a slow render, but you can’t afford to be slow in input processing at all. You wanna actively respond to the user in single-digit milliseconds, but that response doesn’t necessarily mean “re-render all the lines”.

                        1. 5

                          Judging by the comments here I’m not interested in reading the article.

                          But, why use ls | grep foo at all instead of *foo* as the argument for rm?

                          1. 6

                            I was also distracted by using the output of ls in scripting, which is a golden rule no-no.

                            1. 1

                              Is this not what ls -D is for?

                            2. 5

                              Despite “The UNIX Way” saying that we have all these little composable command line tools that we can interop using the universal interchange language of plaintext, it is also said that we should never parse the output of ls. The reasons for this are unclear to me, patches that would have supported this have been rejected.

                              Definitely the glob is the right way to do this, and if things get more complex the find command.

                              1. 5

                                “Never parse the output of ls” is a bit strong, but I can see the rationale for such a rule.

                                Basically the shell already knows how to list files with *.

                                for name in *; do  # no external processes started here, just glob()
                                   echo $name
                                done
                                

                                That covers 90% of the use cases where you might want to parse the output of ls.

                                One case where you would is suggested by this article:

                                # Use a regex to filter Python or C++ tests, which is harder in the shell (at least a POSIX shell)
                                ls | egrep '.*_test.(py|cc)' | xargs -d $'\n' echo
                                

                                BTW I’d say ls is a non-recursive special case of find, and ls lacks -print for formatting and -print0 for parseable output. It may be better to use find . -maxdepth 1 in some cases, but I’m comfortable with the above.

                              2. 3

                                why use ls | grep foo at all instead of *foo* as the argument for rm

                                Almost always, I use the shell iteratively, working stepwise to my goal. Pipelines like that are the outcome of that process.

                                1. 2

                                  I gave an example below – if you want to filter by a regex and not a constant string.

                                  # Use a regex to filter Python or C++ tests, which is harder in the shell (at least a POSIX shell)
                                  ls | egrep '.*_test.(py|cc)' | xargs -d $'\n' echo
                                  

                                  You can do this with extended globs too in bash, but that syntax is pretty obscure. You can also use regexes without egrep via [[. There are millions of ways to do everything in shell :)

                                  I’d say that globs and find cover 99% of use cases, I can see ls | egrep being useful on occasion.

                                  1. 1

                                    If normal globs aren’t enough, I’d use extended glob or find. But yeah, find would require options to prevent hidden files and recursive search compared to default ls. If this is something that is needed often, I’d make a function and put it in .bashrc.

                                    That said, I’d use *_test.{py,cc} for your given example and your regex should be .*_test\.(py|cc)$ or _test\.(py|cc)$

                                    I have parsed ls occasionally too - ex: -X to sort by extension, -q and pipe to wc for counting files, -t for sorting by time, etc.

                                    And I missed the case of too many arguments for rm *foo* (for which I’d use find again) regarding the comment I made. I should’ve just read the article enough to know why ls | grep was being used.

                                  2. 1

                                    That’s clearly just a placeholder pipeline. No one actually wants *foo* anyhow.

                                  1. 42

                                    Eh, there are some problems with xargs, but this isn’t a good critique. First off it proposes a a “solution” that doesn’t even handle spaces in filenames (much less say newlines):

                                    rm $(ls | grep foo)
                                    

                                    I prefer this as a practical solution (that handles every char except newlines in filenames):

                                    ls | grep foo | xargs -d $'\n' -- rm
                                    

                                    You can also pipe find . -print0 to xargs -0 if you want to handle newlines (untrusted data).

                                    (Although then you have the problem that there’s no grep -0, which is why Oil has QSN. grep still works on QSN, and QSN can represent every string, even those with NULs!)


                                    One nice thing about xargs is that you can preview the commands by adding ‘echo’ on the front:

                                    ls | grep foo | xargs -d $'\n' -- echo rm
                                    

                                    That will help get the tokenization right, so you don’t feed the wrong thing into the commands!

                                    I never use xargs -L, and I sometimes use xargs -I {} for simple invocations. But even better than that is using xargs with the $0 Dispatch pattern, which I still need properly write about.

                                    Basically instead of the mini language of -I {}, just use shell by recursively invoking shell functions. I use this all the time, e.g. all over Oil and elsewhere.

                                    do_one() {
                                       # It's more flexible to use a function with $1 instead of -I {}
                                       echo "Do something with $1"  
                                       echo mv $1 /tmp
                                    }
                                    
                                    do_all() {
                                      # call the do_one function for each item.  Also add -P to make it parallel
                                      cat tasks.txt | grep foo | xargs -n 1 -d $'\n' -- $0 do_one
                                    }
                                    
                                    "$@"  # dispatch on $0; or use 'runproc' in Oil
                                    

                                    Now run with

                                    • myscript.sh do_all, or
                                    • my_script.sh do_one to test out the “work” function (very handy! you need to make this work first)

                                    This separates the problem nicely – make it work on one thing, and then figure out which things to run it on. When you combine them, they WILL work, unlike the “sed into bash” solution.


                                    Reading up on what xargs -L does, I have avoided it because it’s a custom mini-language. It says that trailing blanks cause line continuations. Those sort of rules are silly to me.

                                    I also avoid -I {} because it’s a custom mini-language.

                                    IMO it’s better to just use the shell, and one of these three invocations:

                                    • xargs – when you know your input is “words” like myhost otherhost
                                    • xargs -d $'\n' – when you want lines
                                    • xargs -0 – when you want to handle untrusted data (e.g. someone putting a newline in a filename)

                                    Those 3 can be combined with -n 1 or -n 42, and they will do the desired grouping. I’ve never needed anything more than that.

                                    So yes xargs is weird, but I don’t agree with the author’s suggestions. sed piped into bash means that you’re manipulating bash code with sed, which is almost impossible to do correctly.

                                    Instead I suggest combining xargs and shell, because xargs works with arguments and not strings. You can make that correct and reason about what it doesn’t handle (newlines, etc.)

                                    (OK I guess this is a start of a blog post, I also gave a 5 minute presentation 3 years ago about this: http://www.oilshell.org/share/05-24-pres.html)

                                    1. 10

                                      pipe find . -print0 to xargs -0

                                      I use find . -exec very often for running a command on lots of files. Why would you choose to pipe into xargs instead?

                                      1. 12

                                        It can be much faster (depending on the use case). If you’re trying to rm 100,000 files, you can start one process instead of 100,000 processes! (the max number of args to a process on Linux is something like 131K as far as I remember).

                                        It’s basically

                                        rm one two three
                                        

                                        vs.

                                        rm one
                                        rm two
                                        rm three
                                        

                                        Here’s a comparison showing that find -exec is slower:

                                        https://www.reddit.com/r/ProgrammingLanguages/comments/frhplj/some_syntax_ideas_for_a_shell_please_provide/fm07izj/

                                        Another reference: https://old.reddit.com/r/commandline/comments/45xxv1/why_find_stat_is_much_slower_than_ls/

                                        Good question, I will add this to the hypothetical blog post! :)

                                        1. 15

                                          @andyc Wouldn’t the find + (rather than ;) option solve this problem too?

                                          1. 5

                                            Oh yes, it does! I don’t tend to use it, since I use xargs for a bunch of other stuff too, but that will also work. Looks like busybox supports it to in addition to GNU (I would guess it’s in POSIX).

                                          2. 11

                                            the max number of args to a process on Linux is something like 131K as far as I remember

                                            Time for the other really, really useful feature of xargs. ;)

                                            $ echo | xargs --show-limits
                                            Your environment variables take up 2222 bytes
                                            POSIX upper limit on argument length (this system): 2092882
                                            POSIX smallest allowable upper limit on argument length (all systems): 4096
                                            Maximum length of command we could actually use: 2090660
                                            Size of command buffer we are actually using: 131072
                                            Maximum parallelism (--max-procs must be no greater): 2147483647
                                            

                                            It’s not a limit on the number of arguments, it’s a limit on the total size of environment variables + command-line arguments (+ some other data, see getauxval(3) on a Linux machine for details). Apparently Linux defaults to a quarter of the available stack allocated for new processes, but it also has a hard limit of 128KiB on the size of each individual argument (MAX_ARG_STRLEN). There’s also MAX_ARG_STRINGS which limits the number of arguments, but it’s set to 2³¹-1, so you’ll hit the ~2MiB limit first.

                                            Needless to say, a lot of these numbers are much smaller on other POSIX systems, like BSDs or macOS.

                                          3. 1

                                            find . -exec blah will fork a process for each file, while find . | xargs blah will fork a process per X files (where X is the system wide argument length limit). The later could run quite a bit faster. I will typically do find . -name '*.h' | xargs grep SOME_OBSCURE_DEFINE and depending upon the repo, that might only expand to one grep.

                                            1. 5

                                              As @jonahx mentions, there is an option for that in find too:

                                                   -exec utility [argument ...] {} +
                                                           Same as -exec, except that ``{}'' is replaced with as many pathnames as possible for each invocation of utility.  This
                                                           behaviour is similar to that of xargs(1).
                                              
                                                1. 4

                                                  That is the real beauty of xargs. I didn’t know about using + with find, and while that’s quite useful, remembering it means I need to remember something that only works with find. In contrast, xargs works with anything they can supply a newline-delimited list of filenames as input.

                                                  1. 3

                                                    Yes, this. Even though the original post complains about too many features in xargs, find is truly the worst with a million options.

                                          4. 7

                                            This comment was a great article in itself.

                                            Conceptually, I think of xargs primarily as a wrapper that enables tools that don’t support stdin to support stdin. Is this a good way to think about it?

                                            1. 9

                                              Yes I’d think of it as an “adapter” between text streams (stdin) and argv arrays. Both of those are essential parts of shell and you need ways to move back and forth. To move the other way you can simply use echo (or write -- @ARGV in Oil).

                                              Another way I think of it is to replace xargs with the word “each” mentally, as in Ruby, Rust, and some common JS idioms.

                                              You’re basically separating iteration from the logic of what to do on each thing. It’s a special case of a loop.

                                              In a loop, the current iteration can depend on the previous iteration, and sometimes you need that. But in xargs, every iteration is independent, which is good because you can add xargs -P to automatically parallelize it! You can’t do that with a regular loop.


                                              I would like Oil to grow an each builtin that is a cleaned up xargs, following the guidelines I enumerated.

                                              I’ve been wondering if it should be named each and every?

                                              • each – like xargs -n 1, and find -exec foo \; – call a process on each argument
                                              • every – like xargs, and find -exec foo +` – call the minimal number of processes, but exhaust all arguments

                                              So something like

                                              proc myproc { echo $1 }   # passed one arg
                                              find . | each -- myproc  # call a proc/shell function on each file, newlines are the default
                                              
                                              proc otherproc { echo @ARGV }  # passed many args
                                              find . | every -- otherproc  # call the minimal number of processes
                                              

                                              If anyone has feedback I’m interested. Or wants to implement it :)


                                              Probably should add this to the blog post: Why use xargs instead of a loop?

                                              1. It’s easier to preview what you’re doing by sticking echo on the beginning of the command. You’re decomposing the logic of which things to iterate on, and what work to do.
                                              2. When the work is independent, you can parallelize with xargs -P
                                              3. You can filter the work with grep. Instead of find | xargs, do find | grep | xargs. This composes very nicely
                                          1. 9

                                            For the past two years I’ve been using a Raspberry Pi Zero W (yeah, the $10 one) as my daily driver. I do occasionally get to do extraneous stuff on my uni-issued laptop, but for the most part I use the Pi, with Alpine Linux (which I found to be much more lightweight than Arch or Debian-based distros) and i3wm.

                                            Suffice to say that most things simply don’t work on the Pi. Browsing is limited to Netsurf with JS disabled, meaning I have to use my phone to comment here (HN being the only site I’ve seen so far that works perfectly without JS). I practically live in xterm, because most GUI stuff is too slow to be usable. I do occasionally use some lightweight GTK or SDL applications though.

                                            By far the most annoying thing about using the Pi Zero, though, is that any software that relies on GPU acceleration is completely off-limits. That means that pretty much all Vulcan/OpenGL/GLU software is unusable, which includes all Qt applications (I haven’t found a way to disable OpenGL in Qt; if you know a way, by all means let me know). Even simple things like music players and pixel art editors that lack any kind of animations or videos or any other godforsaken garbage that would benefit from hardware acceleration in the least uses Vulcan or OpenGL for some reason. Why? Just why?

                                            1. 5

                                              Regarding hardware acceleration in 2D applications. Using a dedicated chip for rendering is more power efficient so it should be used if possible. For example in a video player the image needs to be interpolated to match screen resolution and a colorspace conversion might be involved as well, which are both an ideal match for a GPU.

                                              1. 4

                                                I’ve wondered this myself. My home computer is an 8-core AMD FX with a 1GB GT720, and anything that uses opengl for compositing feels sluggish. I have to configure Firefox to not use the GPU for compositing, else I run up to the 1GB limit after opening just a few web pages, and then everything starts to feel slow.

                                                If it’s not for compositing then I don’t know what else Qt would be using opengl for. 20 years ago I ran Qt apps on machines that didn’t even have mesa installed, so what changed?

                                                1. 2

                                                  As someone who uses a ThinkPad T41 (Pentium M, 1GB RAM), I recommend Opera 12. It starts slower than Dillo or Netsurf, but it handles relatively modern JavaScript. Many sites like Lobsters and old Reddit work perfectly. The T41 is probably a bit more powerful than the Pi Zero W, but it might work for you too.

                                                1. 5

                                                  At home I’m using a small ARM device (Raspberry Pi 4 with an SSD) as my main computer

                                                  I was interested to learn the why behind this choice but I didn’t think it was covered in the post?

                                                  I toyed with a Pi 4 as a desktop for a bit too, out of curiosity mainly. It works pretty well—especially native binaries. Browsing and web applications, as well as things implemented in scripting languages like Python were where it fell down for me. You could only keep a limited amount of tabs active at a time and it became quite clear in things like modern chat applications that a lot of JS was running on every keystroke.

                                                  It was great for finding slow paths in my usual set up though. The multi-second lag starting a new zsh instance prompted me to replace nvm with fnm for instance.

                                                  1. 6

                                                    You could only keep a limited amount of tabs active at a time and it became quite clear in things like modern chat applications that a lot of JS was running on every keystroke.

                                                    modern software is an absolute travesty

                                                    1. 3

                                                      and it became quite clear in things like modern chat applications that a lot of JS was running on every keystroke.

                                                      This describes my snappy-feeling. I have teams running and although it works, it isn’t pleasant. Switching from one chat to another takes 5 seconds. I really hope that someday someone makes a teams plugin for pidgin.

                                                      I was interested to learn the why behind this choice but I didn’t think it was covered in the post?

                                                      The why behind it has to do with my commute and the weight of my backpack. For the past few years I’d only have work computers at home, sometimes workstations and sometimes laptops (heavy due to the specs). Nothing that actually was mine. We are going back to the office instead of full remote and what drew me over the line was the weight of my current work laptop. When cycling to work its really noticeable if you have 3 kilograms less weight in your bag. Without laptop and power brick, I’m only carrying lunch, a tire repair kit and my wallet. Why not leave the laptop at work? Well, it’s uncertain if the next day I’ll be at work due to covid. Maybe a coworker has to quarantine and so do we or maybe the rules change.

                                                      1. 2

                                                        Even lighter would be a VM that you hibernate to a thumb drive, and performance would be comparable.

                                                        (Edit: on my initial read I thought you were toting the pi instead of the laptop, but after a second read, it sounds like maybe you’re just using it as a home computer and not toting any computing device?)

                                                        1. 1

                                                          Seems like a great solution. Less to cart on the bike and a super cheap home computer.

                                                          1. 1

                                                            Have you any idea what KDE is doing that would cause issues with snappiness - assuming you are noticing some effect on non-KDE apps?

                                                        1. 3

                                                          One of the best feelings in the world is when you realize what makes lisp “LISP” rather than just a programming language is code as data, data as code. Because then you’ll see it everywhere, and suddenly you realize everything is a lisp, and all it’s like no one learned the lesson of the turing machine.

                                                          1. 3

                                                            Concatenative languages aren’t a LISP. They’re based on function concatenation, not function application, which is slightly but importantly different than lambda calculus. There’s a good explanation here: http://evincarofautumn.blogspot.com/2012/02/why-concatenative-programming-matters.html

                                                            “Everything is Turing complete” is not the same as “everything is a LISP”.

                                                            1. 2

                                                              (…) what makes lisp “LISP” rather than just a programming language is code as data, data as code. (…) it’s like no one learned the lesson of the turing machine.

                                                              “Everything is Turing complete” is not the same as “everything is a LISP”.

                                                              The reference was to a single tape, holding data and instructions - not to the idea of Turing completeness.

                                                            2. 1

                                                              If everything is a lisp, then lisp is just another programming language. You can’t have both uniqueness and ubiquity.

                                                              1. 1

                                                                This is the exact conflict that I enjoy experiencing :)

                                                            1. 4

                                                              I’m familiar with tagged pointers, as so many interpreters use them (particularly Ruby, which I’ve hacked on quite a bit). NaN boxing is new to me, though.

                                                              My first thought after reading about the technique is whether signaling NaNs could be used to make arithmetic more efficient. Since using a signaling NaN would generate an exception, generated code could just use the number as-is and catch the exception, falling back to a slow path in that case.

                                                              My second thought is that we’ve had tagged pointers for decades, so why aren’t there hardware instructions for working with them?

                                                              1. 4

                                                                What I do to make arithmetic more efficient in my nan-boxed interpreter is something like this:

                                                                Value add_op(Value a, Value b) {
                                                                    double result = a.number + b.number;
                                                                    if (result != result) goto slow_path;
                                                                    return Value{result};
                                                                slow_path:
                                                                    ...
                                                                

                                                                I optimistically add the two nan-boxed values using float arithmetic. If the result isn’t a nan, I return it. The overhead for the fast path is a nan test and a conditional branch that isn’t taken. The slow path takes care of adding things that aren’t doubles, and throwing an exception if the arguments have the wrong type.

                                                                Using signaling nans and traps would make the code significantly more complicated. And I don’t see why it would be faster. Setting up and tearing down the trap handler for the add case would probably be at least as expensive as the above code, and probably more so.

                                                                1. 2

                                                                  That’s interesting; I very much like that technique.

                                                                  I wouldn’t expect the trap handler to be set up and torn down for every evaluated opcode, just once at startup. The exception would certainly be slower than a failed branch prediction. The fast path would save a comparison, which may or may not be faster on a modern processor.

                                                                  1. 3

                                                                    The tail-threaded interpreter passes around its state in registers. In the case of “tails”, there are two registers containing sp and pc. How does a generic trap handler determine what opcode is being executed, and how does it access the interpreter state? If this information is stored into global variables before executing the add instruction, then that setup code is more expensive than the nan test in my code. Is there a way to use a trap that is cheaper than the nan-test code? Without using assembly language or other CPU and ABI specific code, I mean.

                                                                    1. 1

                                                                      Hmm, that’s a good point. I had imagined it walking the stack like a debugger would, but that would definitely be CPU/ABI specific.

                                                                  2. 2

                                                                    That’s essentially what I do. The constructor that takes a double checks whether it’s already a NaN, which would of course be misinterpreted, and substitutes the value that means “null”. A handy side effect is that arithmetic on non-numbers produces null, which is similar to SQL semantics.

                                                                  3. 2

                                                                    My second thought is that we’ve had tagged pointers for decades, so why aren’t there hardware instructions for working with them?

                                                                    There are some, these are generally called “tagged architectures”. Things like Burroughs Large Systems and LispMs did this. ARM, PowerPC, and a few others also have support if you opt into it, but these are usually exposed to compiler writers, so your compiler has to know about & use them. It’s definitely an interesting area.

                                                                    1. 3

                                                                      CHERI is a tagged architecture, with a single one-bit non-addressable tag to differentiate between memory capabilities (which the C/C++ compiler uses to represent pointers and have strong integrity properties) and other data. Arm is providing us with an implementation early next year.

                                                                  1. 3

                                                                    IMO the separation of code and data doesn’t make a design any less OO; the end result is a data object and a processor object (not unlike MVC).

                                                                    I do agree that as OOP languages evolve they become more functional. Good OO code is often very functional, because functional design imposes limits such as reducing side effects, which makes a program easier to reason about. That doesn’t mean the code is any less OO, unless OO is a synonym for “bad”; rather, as OO programs and languages evolve, they fall toward their functional destiny.

                                                                    In the end, we will end up with a plethora of languages where we can no longer tell whether the language is intended to be OO or functional. You can come at it from the functional side too, and end up building a better CLOS. Would a functional language that acquires OO traits be any less OO than an OO language that acquires functional traits? It is my belief that both are heading toward a shared destiny.

                                                                    1. 13

                                                                      the separation of code and data doesn’t make a design any less OO

                                                                      🤔

                                                                      Anton van Straaten:

                                                                      The venerable master Qc Na was walking with his student, Anton. Hoping to prompt the master into a discussion, Anton said “Master, I have heard that objects are a very good thing - is this true?” Qc Na looked pityingly at his student and replied, “Foolish pupil - objects are merely a poor man’s closures.”

                                                                      Chastised, Anton took his leave from his master and returned to his cell, intent on studying closures. He carefully read the entire “Lambda: The Ultimate…” series of papers and its cousins, and implemented a small Scheme interpreter with a closure-based object system. He learned much, and looked forward to informing his master of his progress.

                                                                      On his next walk with Qc Na, Anton attempted to impress his master by saying “Master, I have diligently studied the matter, and now understand that objects are truly a poor man’s closures.” Qc Na responded by hitting Anton with his stick, saying “When will you learn? Closures are a poor man’s object.” At that moment, Anton became enlightened.

                                                                    1. 65

                                                                      I got that banner once too, after a string of good faith but slightly controversial comments which got many upvotes and a few flags. From what I can tell, the banner is less serious than it sounds, but being nudged towards deleting your account definitely leaves a sour taste in the mouth.

                                                                      I don’t think the ratio of upvotes to flags is taken into account, just the pure number of flags. That would mean that people who post frequently are likely to see the banner, unless everything they post is 100% uncontroversial.

                                                                      The banner suggests talking to a mod, I tried sending a PM to a moderator asking what I should change in my behavior to not be prompted to delete my account again. I never got a reply. I suppose there are higher priority things to take care of.

                                                                      1. 42

                                                                        Yep. I’ve seen it twice.

                                                                        The first time, I had no clue what it was about, and tried to PM every mod. I got a response from one about a week later I think, and then later from pushcx saying that he’d added a page to show you your own ‘standing’.

                                                                        The “flagging” for me was based on just two comments:

                                                                        1, where I expressed an opinion about systemd, that got 30+ upvotes and 1 “incorrect” downvote, and then replied to myself to question how an opinion can be “incorrect”, and that comment got 5 ‘off topic’ down votes.

                                                                        2, I make a sarcastic comment about the irony of using docker to distribute all-in-one Go binaries, and of course, it was downvoted -5 as ‘trolling’.

                                                                        I am convinced from my use here, HN, /. - using user input via ‘downvotes’ as any kind of trusted heuristic about the quality of a comment, is a stupid fucking idea. It’s never been done well, even here, which is already better than most places.

                                                                        The only thing that makes it better here is that the banner is all that happens automatically - there’s no automatic banning or anything. But it really doesn’t make that clear, and the hint about closing your account is just ridiculous IMO.

                                                                        1. 5

                                                                          I remember that discussion (I was the one who responded). I did find it to be an important example of how sensitive the system is, precisely because it’s based on user input. I looked over those two comments carefully at the time, and I still bear them in mind every time we’re discussing feature changes.

                                                                          1. 4

                                                                            And I’m still thankful for your efforts!

                                                                            I know moderation is generally a thankless task.

                                                                            I hope it’s clear from my comments, that the actual moderation (ie the actual actions you take) on 🦞 is among the best Ive seen, if not the best.

                                                                            My issue is very much with the way regular users use the tools available to them, and thus the tools that are available.

                                                                            Lobsters doesn’t have downvotes, so (some) people use flags to signify stuff they disagree with (although I think someone mentioned the incorrect flag is gone now?) as well as flagging actually “bad” stuff.

                                                                            In theory the orange site should do better at this because it has both downvotes and flagging - the problem is the downvote action has consequences that actively encourage the production of echo chambers.

                                                                            🦞 still manages this much better overall, but I also wonder how much of that is down to sheer size. Would unpopular opinions be flagged to death here if the population grew ten-fold?

                                                                            I don’t know. But I wouldn’t bet my carefully hoarded internet upvote points on it!

                                                                            1. 3

                                                                              Good thoughts, and it’s very welcome criticism.

                                                                              Yes, “incorrect” is gone now.

                                                                              1. 1

                                                                                “Incorrect” is gone, but “unkind” remains?

                                                                                That’s sus.

                                                                          2. 5

                                                                            (edit: I’ve removed a quote here, since the post I was replying to had been deleted by its author.)

                                                                            (Slightly later edit: I suspect the author removed that post because it brought up the problem of flagging abuse, and mentioned their own post as an example. Taking a look at my standing page, I absolutely understand why that’s the case though. I have two flags this month: one flagged “troll”, which I don’t wanna argue about, and another one flagged “spam”. It’s on a post that’s on-topic, contains no links, no mention of any product except the one the topic is about, no self-promotion – actually no promotion of any kind. I’d love to know why someone thought it might be spam, but since there’s no way to appeal flags and no way to know who flagged a comment, that’s anyone’s guess.

                                                                            That being said, since some decisions are taken based on flags, there should be a way to appeal them, just sayin’ ;-))

                                                                            It’s also my experience that metrics really don’t help. Way back when phpBB was still a thing I was part of a moderator team that tried to test-drive one of those up/downvote plugins that started popping up around 2006 or so (or maybe earlier? I don’t recall anymore… anyway, it was a long time ago). It was terrible, and we tried a couple of “soft” versions (one that allowed upvotes but not downvotes, another one that allowed for various positive flags like helpful, funny, whatever, the way Slashdot did) – we eventually turned them off for good.

                                                                            All of them had a negative impact not only on the community – which, 15 years of Facebook later, is probably easy to understand – but they had a negative impact on the moderation process, too. We routinely had to weigh obvious decisions against metrics that didn’t tell the whole story (e.g. banning users who slipped subtle neo-Nazi crap into their posts on a 400-page topic and rarely got called out on it because there were like 10 people keeping that topic alive anyway vs. not banning users expressing an unpopular opinion about a recent game or whatever and getting downvoted to hell).

                                                                            There’s always a point at which you admit that the metrics just don’t tell the whole story, like the one above (I remember the systemd post, too). After that point, no metrics tell the whole story anymore. If you have to decide, for every post, if the metrics are trustworthy or not, then you’re already in subjective land, and the metrics only enforce a decision that’s already made. Dumping the metrics entirely at least enables some more freedom to articulate your thoughts and make a good call.

                                                                            Large social networks, like Facebook and Twitter, need to show metrics because they depend on projecting an image of an objective information dissemination platform, otherwise the ad funding dries up. And even in their case it’s bullshit, and they gather more stats, and make more automated decisions, than anyone else. I don’t think acknowledging the subjective nature of mod actions would be hurtful for lobste.rs in any way. Many of us are here precisely because things don’t work the way they work on FB or HN – doing things exactly the way those platforms don’t do them, and could never do them, sounds like a good call ;-).

                                                                            1. 3

                                                                              It’s funny you mention Slashdot: I’ve always been interested in their moderation/meta-moderation approach and wonder why it hasn’t caught on in more places.

                                                                              I think metamoderation could work well here. Present comments with churn, positive or negative, and have the community decide if the vote was appropriate. If the community says “you, user, are using your flags incorrectly”, perhaps they decay in value?

                                                                              1. 5

                                                                                I wasn’t specifically talking about the metamoderation part, but Slashdot’s system has a problem related to that of downvote-based systems: when allowed to do something other than reply, butthurt users will abuse the flag system, too. If someone expresses an unpopular opinion, it won’t be downvoted, but it will be flagged as “flamebait”, “troll”, “redundant” and “offtopic” without any consideration, because that’s just what mobs do – if booing (downvotes) isn’t available, they’ll take slogans (flags) as well.

                                                                                I know this sounds cold-hearted but it is what it is, I’ve been a butthurt user who acted irrationally more times than I can count, and I still find myself indulging in flamewars and the like even when I’m perfectly aware it’s stupid. I try to keep away from flags for this precise reason. Flames are a whole different story :).

                                                                                That being said, it’s worth remembering that Slashdot’s policy dates from a whole other era. I think many younger people have slightly different exectations from moderation these days, shaped by large social media platforms and the like.

                                                                                Any moderation policy is going to have some impedance mismatch. Some people are going to find it intolerable, and many of them will be smart people who have a lot of interesting things to say. Such is life.

                                                                              2. 1

                                                                                I’d love to know why someone thought it might be spam, but since there’s no way to appeal flags and no way to know who flagged a comment, that’s anyone’s guess.

                                                                                I imagine one could have a system where the flags would be required to be accompanied by a motivating comment by the flagger, only visible to the receiver, and perhaps moderators.

                                                                              3. 1

                                                                                I am convinced from my use here, HN, /. - using user input via ‘downvotes’ as any kind of trusted heuristic about the quality of a comment, is a stupid fucking idea. It’s never been done well, even here, which is already better than most places.

                                                                                How would you do it then? You can’t expect moderator(s) to be on top of every thread, so you need some way to alert them that something’s up. AFAIK using downvotes (which Lobsters doesn’t really have, just “flags”) is still the best way to do that.

                                                                                1. 12

                                                                                  I agree, I think lobste.rs is basically the best possible solution I can imagine. It has upvotes to show support, but not downvotes to show “disagreement”. Instead, it lets you flag comments for being off-topic, spam, troll, unkind or me-too; I suspect that the threshold for “I disagree with you, so I’m gonna flag your comment as a troll” is much higher for most people than the threshold for “I disagree with you, so I’m gonna downvote you”. It’s not like actual moderation decisions are made automatically based on the flags.

                                                                                  To be honest, the only real issue I see with the lobste.rs system is that the message is so harshly worded. I wouldn’t even be against the current low flag limit if it just said something like, “Your recent comments have been heavily flagged. Maybe you should consider taking a break to cool down”. Maybe it would be nice if it also took into account the ratio between flags and upvotes, rather than just the flags. But it’s where the banner suggest deleting your account that it really goes off the rails IMO. It’s also a bit weird that it suggests contacting a moderator, when the moderation team clearly doesn’t have the time/patience/desire/whatever to respond to inquiries about it.

                                                                                  1. 33

                                                                                    If we must have such a warning, which I wouldn’t be in favour of really, then I’d phrase it as “Your comments have been flagged a lot, this may or may not be a problem but here are your comments to review; kindly take a look and see if there’s something you could have done better” or something to that effect. Right now, and in your suggestion as well, it kind of assumes you did something wrong, which may not be the case at all.

                                                                                    1. 9

                                                                                      I would also change the background from red to something more neutral. It would still be visible, just not that “alarming”.

                                                                                      1. 5

                                                                                        This is really nicely worded and I can’t agree with you more. Cooling off assumes it was a heated debate, which itself might not be bad, yet the original text was quite aggressive in blaming the recipient, instead of following though that it might have been an honest mistake, or even no mistake at all.

                                                                                      2. 4

                                                                                        I suspect that the threshold for “I disagree with you, so I’m gonna flag your comment as a troll” is much higher for most people than the threshold for “I disagree with you, so I’m gonna downvote you”

                                                                                        I suspect so too, but maybe i’m just projecting.

                                                                                        What feels weird to me about Lobsters flagging system (or at least of the impression i get of it form reading this thread) is that while flags definitely have a cost for the flagged person, they seem to not have any cost on the flagging one? If that’s true, then i think that’s a glaring problem and should be addressed. Someone that’s mass-flagging comments should be as much a red flag for moderation as someone who gets mass flagged.

                                                                                        Regarding the wording of the flag notice itself, i completely agree with others in that it’s quite bad and should be improved in tone and clarity (and probably in color too).

                                                                                        NB: take this comment with a grain of salt, as i’m mostly a lurker here and not much of an active member :)

                                                                                        1. 2

                                                                                          while flags definitely have a cost for the flagged person, they seem to not have any cost on the flagging one?

                                                                                          I got told-off once for flagging something that I shouldn’t have (the mod was correct), though I also didn’t understand that flagging actually got a human involved. My mental model was more like a downvote (from other comments here, I don’t think I’m the only one). So I felt bad about wasting a mod’s time. It probably says how flags work somewhere on the site, but it’s not on the “flag” dropdown :)

                                                                                          Maybe there should be a low limit on flags-per-week, or something like that. You could get really economicsy and a debt system - if you’re out of flags, and you see something you really think should be flagged, well that’s going to put you in flag-debt and cost you two of your flags from next week. With a limit on how far into flag-debt a user can go.

                                                                                          And or some down-weighting of users who flag heavily vs those who flag rarely.

                                                                                      3. 7

                                                                                        As @hauleth correctly points out, the issue is not that people can signal they see a problem with a post.

                                                                                        The problem is assuming that signal is (a) trustworthy and (b) even remotely close to correct.

                                                                                        The idea that people don’t downvote (or flag, in lobster’s terminology, the effect is the same) comments simply because they disagree with the view, is laughable. It definitely seems to happen less here than the orange site, but that’s like saying America has less gun violence than a literal war zone, so there is no problem.

                                                                                        As I said, Lobsters gets a lot of things right, and it wouldn’t take much to improve it.

                                                                                        The threshold for the warning message definitely should be smarter - a fixed number of flags makes zero sense if the goal is to have other users feedback be meaningful. 100 people upvoting and 11 people marking as ‘off topic’ or ‘troll’ because they don’t like the content, doesn’t really signify “you should reconsider your words” to me, unless of course the goal is an echo chamber.

                                                                                        If a post doesn’t go below “0” effective ‘score’, it should not qualify, and even then I’d expect a much higher threshold than 11 such instances.

                                                                                        As multiple people have said, the message itself is just bizarre right now. If you have to have a message it should be a lot more aware of the context of how it’s probably being read.

                                                                                        If someone is genuinely just trolling, the message won’t mean shit to them anyway, they’re not likely to stop because of it.

                                                                                        1. 4

                                                                                          That this warning is not great because of the wording and threshold is something we can quickly agree on, as I already expanded on in some other messages in this thread.

                                                                                          I’m not so cynical about the value of downvotes in general though;I don’t think that “flag for disagreement” are “flag for unconstructive” mutually exclusive. We’re all more sensitive when someone is being an ass about something we disagree with and less sensitive when we do agree with it.

                                                                                          It’s a bit of a tricky balance; early today I saw a thread where an author was just being obtuse IMHO, responding with short messages that were essentially just refutations instead of actually having a conversation. I considered flagging it as it’s not a constructive conversation, or … because I don’t agree with it? Kind of both. I didn’t end up flagging it btw.

                                                                                          I did flag another comment a few days ago which consisted of little more than “oh, you must not have understood it, let me explain it to you again like you’re an idiot who needs to be educated”. It was actually a fairly substantive comment, but also very condensing and not constructive. At that point you’re just being an ass, so -1 troll. I probably wouldn’t have if I had agreed with the gist of the post.

                                                                                          And this also works two ways. I mean, Drew DeVault got plenty of upvotes as well even when he was demonstrably outright lying in his posts just because people vaguely agreed with the sentiment.

                                                                                          Overall, “+1 because I agree with it” is actually a far bigger issue than “-1 because I disagree” IMHO.

                                                                                          1. 3

                                                                                            You’re right. I don’t really agree with the concept of simple ‘upvoting’ or ‘downvoting’. I think it encourages knee-jerk “ooh I agree with one sentence so clicky clicky” responses, rather than requiring people to actually articulate when they agree or disagree with something.

                                                                                            Of course then you end up with “+1” etc type comments.

                                                                                            But the bigger issue IMO is still the conflation of agree/disagree and “this is inappropriate”.

                                                                                            Flagging something means a moderator should intervene and review the comment. It should be unrelated to whether the person agrees or disagrees with the content.

                                                                                            1. 4

                                                                                              I don’t really agree with the concept of simple ‘upvoting’ or ‘downvoting’. I think it encourages knee-jerk “ooh I agree with one sentence so clicky clicky” responses, rather than requiring people to actually articulate when they agree or disagree with something.

                                                                                              How would you sort comments then? Perhaps Lobsters is small enough that it can get away without them, but on HN with hundreds of comments it gets hard and you need some form of mechanism. In spite of the problems with voting, I don’t really know of anything better.

                                                                                              1. 3

                                                                                                Posting order (i.e. chronological) wouldn’t be terrible.

                                                                                                On a small site like this, sorting by ‘agree’/‘disagree’ is probably not necessarily as bad, it that’s all it’s used for.

                                                                                                On the orange site the problem is that they encourage ‘downvote to disagree’ and then hide downvoted comments both visually, and then literally.

                                                                                                If that’s not the poster child for group-think and echo-chambers, I don’t know what is.

                                                                                                1. 4

                                                                                                  I want to say that I’m reading this whole discussion and I appreciate it, though I’ll resist the temptation to weigh in on every single point.

                                                                                                  Lobsters is what it is and I think removing the use of upvotes to sort comments would significantly change the site and the kinds of discussions it’s good for. Many people would leave. When I see people building their own new spaces, however, I do advocate for chronological order. I think it’s very difficult to have anything that even remotely resembles a game mechanic, without attracting users who treat social interaction as a game that they want to win.

                                                                                                  1. 1

                                                                                                    You’re probably right that removing upvotes would change lobsters somewhat. Perhaps then the solution is the downvote mechanism that I detest so much, because if it’s purely attached to sorting order, that’s still a better outcome than having posts flagged simply because someone disagrees.

                                                                                                    1. 1

                                                                                                      Would it be possible to display separate upvote and downvote counts, but still sort by the sum (“overall score”)? That way voting still does affect reading order, but the UI makes it feel less onerous if you’ve been downvoted. In essence, hopefully normalizing a UI pattern to indicate disagreement quickly.

                                                                                            2. 1

                                                                                              And this also works two ways. I mean, Drew DeVault got plenty of upvotes as well even when he was demonstrably outright lying in his posts just because people vaguely agreed with the sentiment.

                                                                                              Is the lying still demonstrable, or was it only demonstrable back then?

                                                                                              1. 7

                                                                                                Stuff like this (and some more on the same topic at HN), and stuff like this. There have been some other discussions as well. I can’t be bothered to track them all down.

                                                                                                1. 2

                                                                                                  I’m sorry to say it, but the web browser scope looks less like lies and more like methodology that you do not like. I cannot judge the other one, though.

                                                                                                  1. 5

                                                                                                    I’ve looked at now 100+ documents from that list. Not a single one has had actual content related to the web standard.

                                                                                                    That’s not a minor methodology error; that’s just completely wrong and he’s more than smart enough to see that too, especially after multiple people pointed it out to him. An honest response would be to fix and correct it.

                                                                                                    1. 1

                                                                                                      I disagree. At the very least, a person still has to sift through all of those documents, even if they don’t have to read them.

                                                                                                  2. 2

                                                                                                    “Demonstrably outright lying” doesn’t fit what I see here. Maybe there were better examples at the time; I know much of the record was lost when drewdevault was banned due to Lobsters’s policy of deleting posts.

                                                                                                    1. 5

                                                                                                      He deliberately said things that were not true and defended it with “but my point us correct anyway”; I don’t know how to call that anything else than lying. Granted, he’s probably just a victim of his own anger and bias, but that’s a poor excuse and the end result is the same.

                                                                                                      Anyway, this thread isn’t about Drew, so I’ll leave it at that.

                                                                                                      1. 3

                                                                                                        Deliberate outright lies could be demonstrated by a direct quote of the lie. What I see is a disagreement over whether an estimate is fair and what it means.

                                                                                                        1. 9

                                                                                                          The estimate makes a really serious methodological error, one that can scupper an otherwise strong paper. It’s fine to make mistakes, but doubling down on them is bad.

                                                                                            3. 3

                                                                                              I think the keyword there is trusted heuristic. No-one says that downvotes/flags cannot be one of the metric that there is something wrong, just it shouldn’t be used to “automatically notify users”. Human interaction for showing “big red banner with “go f… delete account” should be required.

                                                                                              1. 3

                                                                                                It seems to me that the problem is determining who are the trolls - eg the person criticising Go, or the Go fans downvoting the criticiser. One solution might therefore be to limit the downvote to never less than -1, and greyed. Downvote trolls might be satisfied by this enough to not flag, and anything serious would still get flagged, and genuine criticisers wouldn’t be harassed by people who can’t handle criticism of their favourite whatever.

                                                                                                1. 2

                                                                                                  I don’t know if you’re talking about a specific discussion, but in general I find Go discussions tiring because it’s always the same “Go bad” arguments over and over again, whether that’s relevant to the story posted or not. It’s really tiresome.

                                                                                                  If you don’t like Go then that’s fine, but maybe you should be careful interjecting on Go stories unless you really have something of value to add. I don’t go around “C++ bad” or “PHP bad” on every C++ or PHP story.

                                                                                                  I think the same applies to Rust from what I’ve seen on occasion, although I don’t read most Rust stories so I’m not sure.

                                                                                                  1. 1

                                                                                                    Your response seems to have nothing to do with my comment.

                                                                                            4. 12

                                                                                              Previous: Flagged comment warning should have tiers.

                                                                                              IIRC there were some other discussions as well, but I can’t find them right now. I actually thought this warning got removed last year, but it seems not.

                                                                                              I have the impression false positives are fewer since the “incorrect” flag reason got removed; I used to get this warning fairly regularly before that, but haven’t since.

                                                                                              According to the standing page (https://lobste.rs/u/arp242/standing, put in your own username in there) you need to net 11 in the last month to see that page. End up in two or three conversations where one or two people get their panties all in a knot, and yeah … say hello to your 11 flags.

                                                                                              Either way, automatic moderation never really works all that well IMO, or at least should have a really high bar. There is automatic moderation on Stack Overflow (and other SE sites), but the bars are so high that if you hit them, you almost certainly deserved it. And before you get to that point, you usually get flagged in the moderator interface first. IIRC there isn’t even a way for moderators to override these bans, because it’s simply not needed.

                                                                                              I don’t know the details of the single person who has 59(!) flags, but that’s really a excessive outlier (and almost certainly not burntsushi, who is probably close to those 11 flags) so doing something about that automatically might be okay. But 11 flags in 30 days seems really low. And given there are just 7 users in total who see this warning it seems to me that automating this costs most than it’s worth. Lobsters isn’t that large.

                                                                                              1. 2

                                                                                                IIRC there were some other discussions as well, but I can’t find them right now. I actually thought this warning got removed last year, but it seems not.

                                                                                                It was shown on the Replies page, which was offline for performance reasons for a long time.

                                                                                                I’m probably one of the few who took a peek at my /standing page nevertheless.

                                                                                              2. 5

                                                                                                I can promise we’re aware that the voting/flagging system penalizes controversy. The formula doesn’t have any way of knowing which posts and comments are difficult-but-necessary conversations vs. which ones are behavior that should be changed, but we do take it into account behind the scenes.

                                                                                              1. 6

                                                                                                Which network will host the lobsters irc channel in the future?

                                                                                                1. 6
                                                                                                1. 7

                                                                                                  As I’ve got to understand some time ago, common language is just a tool to communicate ideas between individuals. Not the place where perfection needs to be attained at all costs. That’s especially hard for us programmers who normally work in a context or almost supernatural perfection.

                                                                                                  Sure saying “HTTP API” is technically better but if saying that we are going to build a RESTful API makes the concepts clear to the whole team (from the new intern to the snarky senior), that’s what we will use.

                                                                                                  As the article presents, since APIs implemented exactly as presented in the original paper (“trulu” RESTful) call themselves something else than REST, that means that there is no really confusion to begin with.

                                                                                                  This meand that if someone nowadays says REST API they actually mean HTTP API, that’s it.And if someone says Hydra, HyperMedia or ChockoMangoTM API then the other person might simply say “can you show me what you mean?” and move from there.

                                                                                                  1. 9

                                                                                                    saying that we are going to build a RESTful API makes the concepts clear to the whole team

                                                                                                    I feel like it doesn’t. Most people just think of HTTP methods bolted on to any kind of API. If you say HTTP API, and maybe clarify by saying “yknow, an API designed to respond to HTTP messages”, it would be considerably clearer since there’ll be no conflict between people’s varying understandings of REST.

                                                                                                    1. 7

                                                                                                      Right, exactly. If someone says a “REST API” I don’t know if they actually mean REST as originally defined or if they’re just using it to mean ‘any API that uses HTTP as a transport’.

                                                                                                      1. 9

                                                                                                        I made this mistake once. I was asked to design a REST API, and so I did. Turns out they wanted RPC over http.

                                                                                                        1. 2

                                                                                                          Turns out they wanted RPC over http.

                                                                                                          There are a ton of folks who seem to think that REST == JSON-over-HTTP, and honestly don’t understand why RPC with JSON-over-HTTP isn’t RESTful.

                                                                                                          1. 2

                                                                                                            LOL that made me dribble my drink i’m so sorry

                                                                                                          2. 2

                                                                                                            Definitely in the minority there though.

                                                                                                            Not to say that you are wrong but you are underestimating how many people actually think like that, at least from my own experience. This is supported even further by the fact that this post was even made in the first place.

                                                                                                      1. 1

                                                                                                        Yeah, but can it run Doom?

                                                                                                        1. 2

                                                                                                          Doom needed four floppies. Doom 2 took 5.

                                                                                                          1. 1

                                                                                                            Perhaps it could be loaded into memory from a DAT cassette.

                                                                                                            1. 1

                                                                                                              Most of the Doom data is game assets and levels though, I think? You might be able to squeeze the game engine and a small custom level in 400k.

                                                                                                              1. 1

                                                                                                                Yes it is. But the engine itself is about 700k. The smallest engine (earliest releases) were a little over 500k. You could probably build a smaller one with modern techniques and a focus on size though.

                                                                                                                1. 2

                                                                                                                  Good news, ADoom for Amiga is only 428k! Bad news, Amigas only have double density FDDs so you only have 452k for the rest of the distro.

                                                                                                          1. 20

                                                                                                            For context, see https://www.kline.sh/

                                                                                                            As far as I can see, the birds have finally come home to roost from the sale of Freenode many years ago.

                                                                                                            Edit this submission can probably folded into https://lobste.rs/s/p50qbz/freenode_now_belongs_andrew_lee_i_m

                                                                                                            1. 24

                                                                                                              When I heard about the acquisition several years ago, I assumed the eventual result would be a lot more dire.

                                                                                                              Having a clean break and switching over to Libera Chat is a much better outcome than I expected. My hat is off to all the former freenode volunteers who have put in so much work to keep the network running and have successfully navigated this transition in a way that will protect the network for its users over corporate interests going forward.

                                                                                                              1. 10

                                                                                                                This resignation letter also has some more backstory and lots of links: https://mniip.com/freenode.txt

                                                                                                                And admin access to the freenode servers has been lost: https://twitter.com/freenodestaff/status/1395046345145307140

                                                                                                                1. 10

                                                                                                                  This part is by far the worst:

                                                                                                                  Some users reported[6][7] that Andrew Lee had already started assembling a replacement team, and that people were bribed to join with the prospect of money, power, vanity, and even revenge on former staff[8].

                                                                                                                  1. 5

                                                                                                                    Oh, so our friendly neighborhood freenode admins will be replaced by people who are only in it for the power trip and the money. That’ll definitely end well.

                                                                                                                    1. 1

                                                                                                                      Maybe I’m dense or maybe those chat logs were too long and I missed it. Where was the bribe?

                                                                                                                      1. 5

                                                                                                                        Not sure where money comes in, but here’s the power thing:

                                                                                                                        [23:06:08] <nirvana> and the opers will just /wallops and global notice their new network
                                                                                                                        [23:07:35] <Ariadne> i mean, i have important work to do.  dealing with an IRC network is not really something i want to be doing this decade outside of fucking around for fun with IRCX
                                                                                                                        [23:07:51] <Ariadne> i have code running on two planets
                                                                                                                        [23:07:53] <nirvana> but
                                                                                                                        [23:07:57] <nirvana> you can ban your enemies.
                                                                                                                        [23:08:01] <nirvana> im turning tomaw
                                                                                                                        [23:08:01] <Ariadne> i don't have enemies
                                                                                                                        [23:08:07] <nirvana> and jess in to a bot.
                                                                                                                        [23:08:22] <nirvana> i know you don't, we're older now
                                                                                                                        [23:08:25] <nirvana> more mature
                                                                                                                        [23:08:31] <nirvana> i was just trying to make you laugh
                                                                                                                        [23:08:52] <nirvana> dont tell me that sweet revenge of a kline /fuckyou - just ONE to those people who dissed on charybdis
                                                                                                                        [23:08:59] <nirvana> wouldn't make you feel good
                                                                                                                        [23:09:01] <Ariadne> i don't really care
                                                                                                                        [23:09:14] <nirvana> i think about it, i wouldn't do it though
                                                                                                                        [23:09:18] <nirvana> have to walk the higher path
                                                                                                                        [23:09:29] <nirvana> im still banned from snoonet lol
                                                                                                                        [23:10:58] <Ariadne> your employer owns snoonet
                                                                                                                        [23:11:45] <nirvana> i actually dont work for ltm
                                                                                                                        [23:11:50] <nirvana> still
                                                                                                                        [23:12:06] <nirvana> i work for his brother though
                                                                                                                        [23:12:09] <nirvana> alex lee
                                                                                                                        [23:12:48] <nirvana> i help where i can but im too busy at the office to do much else
                                                                                                                        [23:12:53] <nirvana> like have a girlfriend, mine left me ;/
                                                                                                                        [23:13:19] <nirvana> and actually i think its just prawnsalad that owns snoonet now
                                                                                                                        [23:13:21] <nirvana> i could be wrong tho
                                                                                                                        [23:15:44] <Ariadne> anyway, i think i've contributed as much as i can to the topic of IRC network governance with the charybdis and atheme software packages as well as the atheme operator workflow (of which freenode still uses 
                                                                                                                        to this day for training)
                                                                                                                        [23:16:07] <Ariadne> having an o:line means additional work for me.  it is a distraction.
                                                                                                                        [23:16:37] <nirvana> ill make sure you get +oO in #freenode
                                                                                                                        [23:16:40] <nirvana> so you can kick people
                                                                                                                        [23:16:44] <Ariadne> why the hell do i want that
                                                                                                                        [23:16:45] <nirvana> my gift to you pal
                                                                                                                        [23:16:56] <Ariadne> stop trying to bribe me
                                                                                                                        [23:16:59] <Ariadne> this is just pathetic
                                                                                                                        [23:17:05] <nirvana> might even give you flags +f
                                                                                                                        [23:17:08] <nirvana> so your friends cna join in
                                                                                                                        
                                                                                                                  2. 2

                                                                                                                    This is VERY interesting. Happy to see IRC is still around, at least.

                                                                                                                  1. 84

                                                                                                                    The network is the people, not a domain name. #lobsters is staying with Libera.Chat.

                                                                                                                    1. 6

                                                                                                                      freenode has taken over the #lobsters namespace*. I was in the channel directing people to the current place, but I’ve left to avoid any appearance of sanctioning freenode’s channel. Given how long we were present there may be some confusion, so I wanted to leave an explicit comment that, no, there is nothing on freenode that is in any way affiliated with any aspect of this website.

                                                                                                                      • Along with hundreds of others. And removed every banmask, which they’ll learn to regret pretty quickly.
                                                                                                                      1. 4

                                                                                                                        So basically Libera.Chat is an old-fashioned permanent netsplit, where every channel in the original network is also available there? I haven’t seen those in over twenty years!

                                                                                                                        1. 18

                                                                                                                          where every channel in the original network is also available there?

                                                                                                                          No. Communities that want to move to Libera will have to create the channels there as well, nothing is created/moved “automatically”.

                                                                                                                          1. 14

                                                                                                                            I don’t think it’s like that; I will be shutting down all the freenode channels I operate as soon as the matrix bridge to Libera is operational. There will be stragglers, but I don’t know of any channels which are seriously considering remaining on freenode.

                                                                                                                            1. 4

                                                                                                                              Can you please explain why Matrix is such a killer feature for you? I keep seeing it mentioned, but I don’t know much about it, particularly how it is better/different than other chat technologies that have come and gone over the years (e.g. jabber).

                                                                                                                              1. 5

                                                                                                                                I use Matrix to bridge all my other cats together. I can get Signal, Telegram, Hangouts and Matrix all in one place (FB too until I abandoned that).

                                                                                                                                My Google voice I recently moved to jmp.chat/XMPP and sadly the XMPP bridges to Matrix are lacking. I’ve started looking at biforst, but it’s super early alpha.

                                                                                                                                1. 5

                                                                                                                                  I personally don’t care much about Matrix, but in the channels I operate, we have a decent contingent of users connecting from Matrix because they prefer not to have to set up a bouncer to get messages when they’re disconnected. For my own use, the bouncer setup is way better, but I’m trying to be accommodating for the preference of others.

                                                                                                                                  In theory, the idea of truly federated chat is awesome, but in practice everyone uses the flagship server and setting up your own homeserver is intensely unpleasant. So they have a long way to go to actually achieve the benefits of decentralization really.

                                                                                                                                  1. 1

                                                                                                                                    IMO the benefits are here if you use it as a glorified IRC bouncer. Yes I know there are plenty of those already but I’m too lazy to host one and I don’t want to pay for one.

                                                                                                                                    1. 2

                                                                                                                                      I agree it’s a serviceable bouncer, but the benefits of decentralization have nothing to do with that.

                                                                                                                                  2. 1

                                                                                                                                    Simple answer for me: Matrix is the IRC bouncer I always wanted but never had the discipline to keep running successfully.

                                                                                                                                    Persistent presence on all the channels I care about. Huge win.

                                                                                                                            1. 1

                                                                                                                              I’m curious how much of this is perhaps driven to derail screen scraping.

                                                                                                                              1. 4

                                                                                                                                I’m scratching my head here. Do people actually screen-scrape Google Docs?

                                                                                                                                1. 3

                                                                                                                                  Even if they did, why would Google want to disable this? Are there docs out there being accessed by thousands of bots at the same time?

                                                                                                                                  1. 1

                                                                                                                                    Answer is: probably

                                                                                                                                2. 4

                                                                                                                                  Or adblocking.

                                                                                                                                  1. 3

                                                                                                                                    I am wondering if it is to get decent performance on low-spec Chromebooks.

                                                                                                                                    1. 1

                                                                                                                                      As someone who works on rich editors for the web (https://notion.so), this move makes complete sense for both performance and reliability. With Canvas/WebGL & a custom framework, Docs has much complete control over rendering, and render latency. They have much lower exposure to user agent differences. And, as a megacorp with infinite engineers, they can bear the much higher cost of building everything themselves.

                                                                                                                                    1. 12

                                                                                                                                      There’s a real interesting historical connection here. The author of this, Wouter van Oortmerssen, also created FALSE in 1993. Not only was FALSE one of the first “true” esolangs, it inspired Chris Penner’s Befunge and Urban Müller’s… brainfuck! I don’t think esolangs would be nearly as vibrant today without Wouter’s work.

                                                                                                                                      1. 3

                                                                                                                                        Wouter was huge in the Amiga scene. He created the E programming language, which was definitely my favorite language to use on the Amiga.

                                                                                                                                        EasyGUI (also written by Wouter, I believe), was the sweet spot for GUI development on the Amiga. It got you 80% of the way to MUI without needing MUI (and trust me, GUI development on the Amiga without something BOOPSI-based like MUI was terrible.)

                                                                                                                                        Wouter then went on to develop Flat Buffers at Google. He’s had an amazing career.

                                                                                                                                        1. 5

                                                                                                                                          He also did TreeSheets!

                                                                                                                                          1. 4

                                                                                                                                            which is written in Lobster, incidentally!

                                                                                                                                            1. 8

                                                                                                                                              That’s what I would do if I’d write it today, but it is actually in C++ (using wxWidgets), though nowadays has a way to script it in Lobster. TreeSheets actually predates Lobster by a few years (2008 vs 2010 or so).

                                                                                                                                          2. 1

                                                                                                                                            He created the E programming language

                                                                                                                                            I’m assuming this is a different E than the one of capabilities fame created by Mark Miller ?

                                                                                                                                            EDIT: nm, the wikipedia page links to AmigaE, which is the one you were referring to.

                                                                                                                                            1. 1

                                                                                                                                              Yeah, sorry. This one: https://en.wikipedia.org/wiki/Amiga_E

                                                                                                                                          3. 2

                                                                                                                                            Can you enlighten me on what an esolang is?

                                                                                                                                            1. 3
                                                                                                                                              1. 1

                                                                                                                                                Hey it’s not a “previously on” because I never submitted that one to lobsters :P

                                                                                                                                                (The lecture is definitely going up, if it doesn’t kill me first. One more week to go…)

                                                                                                                                                1. 2

                                                                                                                                                  Wouter’s homepage is worth a browse.

                                                                                                                                                2. 1

                                                                                                                                                  inspired Chris Penner’s Befunge

                                                                                                                                                  just to clarify (incase anyone else was confused like me), it’s Chris Pressey that invented Befunge; Chris Penner is someone else; at least I think so :)

                                                                                                                                                  1. 1

                                                                                                                                                    Mea culpa! No idea why I wrote “Penner”

                                                                                                                                                  2. 1

                                                                                                                                                    The visual programming language he did for his PhD thesis (late ’90s) is interesting as well: http://strlen.com/aardappel-language/

                                                                                                                                                  1. 3

                                                                                                                                                    I remember being confused and amazed the first time I came across Rspec years ago. After some years of using it as a black box I finally sat down to try to imagine how the nifty syntax even worked, and came up with this little gist.

                                                                                                                                                    That said, now that I’ve moved onto Elixir, I actually much prefer the ExUnit syntax:

                                                                                                                                                    describe "my_func/4" do
                                                                                                                                                      test "it works as expected" do
                                                                                                                                                        assert foo == blah
                                                                                                                                                      end
                                                                                                                                                    end
                                                                                                                                                    

                                                                                                                                                    It’s more minimal. When I use the expect(foo).toBe(...) syntax, I feel like I’m constantly having to look up the different matchers and what they mean. A simple matching assert is much clearer to me. It’s a bit of friction every time I have to work on our frontend jest tests.

                                                                                                                                                    That said, I got my start with RSpec and it taught me to think in terms of “expectations”, and how it’s even possible to write tests before the code, which I still do from time to time. So it’s a testament to how much RSpec changed programming that what was revolutionary at the time and needed these syntactical guidelines is now taken for granted and just feels clunky to me.

                                                                                                                                                    1. 4

                                                                                                                                                      Matchers are not just syntactic sugar; there’s another reason they exist.

                                                                                                                                                      Imagine writing a testing framework in Ruby where you can write assertions the way you suggest:

                                                                                                                                                      assert foo == blah
                                                                                                                                                      

                                                                                                                                                      How do you get Ruby to print the values of foo and blah if the assertion fails?

                                                                                                                                                      You can do it, by inspecting the AST or the bytecode of the calling method at runtime, but it’s really kludgy to do in Ruby, and the interpreter must withhold optimizations for it to work (e.g. foo and blah must be stored somewhere in the stack frame, not held in registers and then optimized away since they are no longer used). That’s why in Test::Unit (or minitest) we instead write:

                                                                                                                                                      assert_equal blah, foo
                                                                                                                                                      

                                                                                                                                                      But now you have a new problem: any new type of comparison you might want to do needs a new assertion method, so we end up with assert_equal, assert_not_equal, assert_instance_of, assert_match, assert_same, and so on. And they all end up looking something like this:

                                                                                                                                                      def assert_equal(expected, actual)
                                                                                                                                                        msg = build_message(expected, actual) {
                                                                                                                                                          "Expected #{expected} to equal #{actual}"
                                                                                                                                                        }
                                                                                                                                                        assert_block(msg) { expected == actual }
                                                                                                                                                      end
                                                                                                                                                      

                                                                                                                                                      What matchers do is take away some of the boilerplate involved in writing a new assertion. So instead of the above, we could eliminate some of the duplication like this:

                                                                                                                                                      def assertx(actual, matcher, *expected)
                                                                                                                                                        msg = build_message(expected, actual) {
                                                                                                                                                          "Expected #{actual} to #{matcher} #{expected}"
                                                                                                                                                        }
                                                                                                                                                        assert_block(msg) { @matchers[matcher].call(actual, *expected) }
                                                                                                                                                      end
                                                                                                                                                      
                                                                                                                                                      def define_matcher(name, &block)
                                                                                                                                                        @matchers[name] = block
                                                                                                                                                      end
                                                                                                                                                      
                                                                                                                                                      define_matcher :equal { |expected, actual| expected == actual }
                                                                                                                                                      

                                                                                                                                                      then use it like this:

                                                                                                                                                      assertx foo, :equal, blah
                                                                                                                                                      

                                                                                                                                                      Once you have that, it’s a few more steps to flip it around and make it read like Engilsh, which is how we end up with expect:

                                                                                                                                                      expect(foo).to.equal(blah)
                                                                                                                                                      
                                                                                                                                                      1. 4

                                                                                                                                                        Oooh, good point. I didn’t think of that. In Elixir assert is a macro, which I see now is how we get all the developer niceties of showing what was expected and given and all that. That plus asserting against a pattern match to select out map fields and the like makes it real slick and easy feeling. But I don’t think I had considered how much the underlying language contributes to what sort of testing is possible. Thanks, this was interesting!