1. 7

    I use Gnus, which is a news/mail reader in Emacs. It does threading and quoting correctly and display of HTML messages pretty well.

    In the past I went through a period of Mutt usage, and found the tutorial by Steve Losh at https://stevelosh.com/blog/2012/10/the-homely-mutt/ to be very helpful

    If you prefer Vim I think Mutt is a very good option. It can be configured to render HTML via lynx or w3m IIRC (I think it’s mentioned in the above tutorial) ISTR Losh’s preferred mutt keys are also quite vimmish

    1. 2

      I also use Gnus. I found it very strange to begin with, but got used to it. The splitting and scoring features are incredibly powerful, especially for high-traffic mailing lists

      1. 1

        Gnus here as well, but I blew away all the keybindings[1] and have a few of my own (that are intuitive to me having a history of mutt, pine, etc. and also because I use Evil mode).

        I use Gnus over other Emacs clients because it can do IMAP and so I do not need to depend on something like fetchmail (or whatever is popular these days), meaning it’s easy to bring up on different machines and platforms. UI-wise I’d prefer mu4e.

        [1] Gnus is a bit vi-like in it’s keybindings: there are a lot of them and accidentally touching the wrong one might delete or kill something you did not want to lose. (Or at least, I could never figure out how to recover.)

      1. 2

        What I really want is something like Python (or like Haskell, depending on the circumstances) where running shell commands is a “first-class” language construct, with ergonomic piping of stdout/stdin/files. Oil seems much-improved compared to Bash, but it (intentionally, for good reason) borrows many of the same bad design choices with regard to control flow and cryptic naming schemes.

        The best solution I’ve found so far is to call subprocess.run from a Python script and use string interpolation to pass arguments to the commands I want to run. But this is far from ideal.

        1. 8

          Why not Perl? It’s the obvious choice for a language where shell commands are a “first-class” language construct.

          1. 3

            Totes agreed!

            I find shell logic and data structures impenetrable and error prone, so I write all my “shell scripts” in Perl. It’s easy to pull shell output into Perl data structures using backticks, e.g. print the names of files in $TMPDIR iff they are directories

            chomp(my @out = `ls $ENV{TMPDIR}`); for (@out) { print "$_\n" if -d qq[$ENV{TMPDIR}/$_]; }
            

            For non-perl users, the backticks above mean “run this shell command”, chomp removes the newlines, qq[] means “wrap this in double quotes and handle the annoying escaping for me”, and $ENV{X} is how you access env var X, etc.

            The book Minimal Perl for UNIX and Linux People has a lot of information about how perl can be used to do everything on unices that awk, grep, sed, etc. can do

            also with Perl you get a real debugger (see also perldoc perldebtut or the book Pro Perl Debugging)

            1. 3

              I think it easily gets overlooked these days, since it is no longer hip and therefore less on the frontpage news

              1. 5

                Apropos of “no longer hip”, tcl is a nice scripting language with a clean syntax.

                1. 1

                  I used to mess with tcl scripts all the time in the early 2000s when I ran eggdrop irc bots. Good old days…

                  1. 4

                    Sqlite started as a tcl extension and still has a good tcl interface: https://sqlite.org/tclsqlite.html

                    1. -2

                      TIL!

            2. 3

              Depending on your use cases, have you ever tried Fish, or more on the programming side, Raku/Perl? Raku is awesome for shell-style scripting.

              1. 2

                Hm what do you mean by control flow? I’m working on more docs so it probably isn’t entirely clear, but control flow looks like this

                if test --dir $x {
                  echo "$x is a dir"
                }
                

                And you can use expressions with ()

                if (x > 0) {
                  echo "$x is positive"
                }
                

                For loops are like bash with curly braces

                for x in a b c { echo $x }  # usually written on 3 lines, being compact here
                for x in *.py { echo $x }
                for x in @(seq 3) { echo $x }  # explicit splitting rather than $(seq 3)
                

                I’m also interested in feedback on naming. Oil has long flags like test --dir instead of test -d. I am trying to get rid of all the one letter obscureness (or at least ALLOW you to avoid it if desired).


                It’s accurate to describe Oil as “Python where running shell commands is first class” (and pipelines and more).

                Although I would take care to call it perhaps the “David Beazley” subset of Python :) That is a prominent Python author who has been lamenting the complexity of Python.

                He writes some great code with just numbers, strings, lists, dicts, and functions. Oil is like Python in that respect; it’s not like Python in the sense that it has a meta-object protocol (__iter__ and __enter__ and __zero__ and all that).

                I’m interested in language feedback and if that matches your perception of Oil :) Does the code look cryptic?

                1. 3

                  I’ve only had a cursory look at Oil, so forgive me if I have the wrong idea :). I really admire the effort, and I can really relate to what you’ve written about your motivations for creating Oil!

                  Oil is also aimed at people who know say Python or JavaScript, but purposely avoid shell

                  It seems like Oil will be great for people who spend a lot of time maintaining shell scripts and want to work with something a little more sane. However, the Oil docs impression that Oil will be just as confusing as Bash for someone like me who wants to spend as little time as possible writing / reading / maintaining shell scripts and who only finds occasion to do so once every few months. For someone like me, I wish there were some syntax-level construct in Python or Haskell for running shell commands that lets me take advantage of my existing knowledge of the language, rather than having to learn something entirely new.

                  I took another look at the Oil docs, and the control flow actually seems ok, even if minimal. It would be cool to see some more tools to map / flatmap / fold commands, especially when I have a bunch of parameters that are used to generate a sequence of commands.

                  I think the biggest improvement a new shell-scripting language can offer is an alternative syntax for specifying command-line flags. Now that I think of it, this is actually the main reason I wrap all my CLI calls in Python scripts. For instance, here are some commands I had to run at work last week:

                  docker run -itd --name container_name --rm -e CUDA_VISIBLE_DEVICES=0 -e API_KEY="..." -v /home/ben:/ben/ -v /data/v2/:/data image_name
                  
                  docker exec -d container_name sh -c 'mkdir -p /ben/train/train_2021-05-16_144614 && python /ben/project_name/train.py \
                                  --batchSize 16 --loadSize 160 \
                                  --alpha 1.0 \
                                  --annotations /data/project_name/sourceA/v3/really_long_name.txt \
                                  --annotations /data/project_name/sourceB/v1/really_long_name.txt \
                                  --annotations /data/project_name/sourceC/v4/really_long_name.txt \
                                  --data-path /data/project_name/images \
                                  --output-path /ben/train/train_2021-05-16_144614/checkpoints \
                                  --gpu_ids 2 \
                                  2>&1 > /ben/train/train_2021-05-16_144614/log_2021-05-16_144614_outer.txt'
                  

                  I actually had to run the exec command four different times with different arguments. I wrote a Python script to generate the Python command, wrap it in the docker command, and pass in all the necessary parameters with string interpolation. But to me this seems like a pretty silly to glue together different programs, when we already have familiar syntax for function calls! I would love to write instead:

                  var env_variables = [
                    CUDA_VISIBLE_DEVICES=0,
                    API_KEY="..."
                  ];
                  
                  var volumes = [
                    /home/ben/:/ben/,
                    /data/commondata/v2, /data
                  ];
                  
                  docker.run(
                    flags="interactive, tty, remove",
                    env=env_variables,
                    name="container_name",
                    image="image_name",
                    volume=volumes
                  );
                  
                  ...
                  
                  image_annotations=[ /data/project_name/sourceA/v3/really_long_name.txt, ... ]
                  
                  python_command=python(train.py,
                    batchSize=16, loadSize=160, alpha=1.0,
                    annotations=image_annotations,
                    data-path=/data/project_name/images,
                    output_path=...
                    gpu_ids=2,
                  )
                  
                  docker.exec(
                    detached=True,
                    container=container_name,
                    command=python_command
                  )
                  

                  This should work, automatically, for ANY command line interface, not just docker. It should work without writing any custom wrapper code. Without actually running the command, I want some kind of TypeScript-style static type-checking to know if the flag/argument I provided does not match what the CLI expects.

                  1. 1

                    Thanks this is useful feedback!

                    Yes I understand that Oil feels like it’s for the “shell expert” now. Because I do frame many things in terms of an upgrade path from bash, and that requires knowing bash (lots of people don’t!).

                    But it’s absolutely the goal for a Python/JS programmer to be able to pick up Oil with minimal shell knowledge. You have to understand argv, env, stdin/stdout/stderr, and exit code, and that’s it hopefully. Maybe file descriptors for the advanced.

                    I have an upcoming “Tour of Oil” doc that should help with this. It will document Oil without the legacy stuff.

                    It would be cool to see some more tools to map / flatmap / fold commands, especially when I have a bunch of parameters that are used to generate a sequence of commands.

                    Yes, great feedback, there is a pattern for this but it’s probably not apparent to newcomers. In shell map / flatmap are generally done by filtering streams of lines, and I think that pattern will help.

                    https://github.com/oilshell/oil/issues/943

                    There is also splicing of arrays, which are natural for command line args

                    var common = %( --detached=1 --interactive )
                    var args1 = %( --annotations /data/1 )
                    var args2 = %( --annotations /data/2 )
                    
                    # Run 2 variants of the command
                    docker run @common @args1
                    docker run @common @args2
                    
                    # Run 4 variants
                    for i in @(seq 4) {
                      docker run @common --index=$i
                    }
                    

                    Does that make sense as applied to your problem? You can also apply to env vars with “env”:

                    var env_args = %( PYTHONPATH=. FOO=bar )
                    env @env_args python -c 'print('hi')
                    

                    There is no static checking, but Oil’s philosophy is to give errors earlier in general. It will be possible to write your own argv validators if you like.

                    A useful pattern will be to define “subsets” of Docker as “procs”. As in, you can just accept the flags you use and validate them:

                    proc mydocker {
                       argparse @ARGV %OPT {   # not implemented yet
                          -v --verbose "Verbose flag"
                       }
                       docker @myarray
                    }
                    mydocker --oops  # syntax error
                    

                    Thanks for the feedback, let me know if that makes sense, and if you have other questions. And feel free to follow the issue on Github!

                2. 1

                  Golang exec’s module is much better for piping in the classic shell way. You might consider that.

                  1. 1

                    The best solution I’ve found so far is to call subprocess.run from a Python script and use string interpolation to pass arguments to the commands I want to run. But this is far from ideal.

                    I do a similar thing. What sort of improvements can you envision?

                  1. 1

                    why is SQL so successful despite its shortcoming?

                    Human and machine efficiency:

                    1. Human efficiency: If I learn e.g. MongoDB, FaunaDB, or Datomic’s proprietary languages, and they go out of business, I have literally wasted my time. If I learn SQL, I have an industry-standard skill I can re-use, probably for decades. Multiply that across thousands of people working with “data stuff” across the entire world.

                    2. Machine efficiency: We have decades of research papers and industrial experience in optimizing SQL query performance. Some one-off DB with its own query language may or may not be able to take advantage of all that has been learned over the past 40+ years about SQL query optimization. The fact that someone has invented their own query language does not mean they don’t have this knowledge, but it’s a signal that they may not. (Which feeds back into point 1: I need to think your DB is worth spending my time learning, so you need to signal to me in various ways that you know what you are doing. SQL is one of those signals, since it is data’s lingua franca).

                    1. 8

                      Getting LSP integrated pervasively in Emacs in a way that it reliably just works and performs well out of the box

                      I don’t get how this should work, when setting up LSP servers is now the issue. If you want to use rust-analyser, you have to download rust-nightly. If you want to use a Java LSP server, you have to do class-path magic. If you want to use a Python LSP server, you’ll have to let pip download every single python library again to get the right versions. If you want to compile a Haskell LSP server, you’ll need to buy some more RAM.

                      But if this would be solved, and all I have to do is run “sudo package-manager install java-lsp-server”, then it would be a considerable advancement. The best one can do, as I understand is planned for 28.1, is to get Eglot into a working and stable state, so that it’s ready for when the server implementations have matured.

                      Having a wizard showing up in new Emacs installations might be a great low-hanging fruit way of making Emacs more accessible.

                      I have discussed this on the mailing list, but haven’t ever had the time to implement this myself (yet). What Emacs needs is a wizard language, that can be used to teach Emacs itself, but also any package (just like ert is used for tests). Most proposals haven’t been too pretty though.

                      Take a look at the screenshots of these Neovim GUIs:

                      I don’t see any notable difference, tbh. What I would get is better integration into window managers (opening downloaded files in existing emacs sessions), but I have never understood what neovim actually achieves by splitting UI and backend. Sure, it’s a good design decision in principle, but considering the complexity of Emacs display engine, I’m not sure if it is worth it.

                      It is very easy to either freeze Emacs or cause it to run very slowly. Multiple times a day I have to hit C-g incessantly to bring it back from being frozen. When that fails, I am sometimes able to get it back with pkill -SIGUSR2 Emacs. At least once per week I have to pkill -9 Emacs because it turned completely unresponsive.

                      I used to have these problems, but in my experience this comes from fighting Emacs instead of cooperating with it. Using unnecessary packages, weird themes and wanting popups everywhere will simply slow stuff down. The main parallelism in Emacs is done by starting external processes, since Emacs is more of a Unix shell than Lisp machine.

                      It would be great to have more and more of these influencing and being incorporated to the Emacs Lisp standard library and made to be very performant.

                      Please not, dash and s are quite annoying and don’t always lend themselves to idiomatic Elisp (I understand they are influenced by Clojure, which has a difference heritage). seq.el and map.el are nice, but since they are based on generic functions, they are often slower.

                      It would be great to move to a forge style of contribution.

                      As long as it doesn’t require new accounts, and everybody is free to participate in the conversations, ok, but I’ve only seen this implemented by source-hut, and none of the GitHub clones that have spoiled so many people.

                      1. 6

                        Emacs is more of a Unix shell than Lisp machine

                        Strongly agree with this. It’s extremely good as a “visual shell” / UI framework. It’s not usually a good UX when you try to implement too much of your world in Elisp with dozens of packages, spinners, networking, etc. You end up in C-g hell as the author describes. Over 15 years of usage I’ve actually evolved away from the “rewrite everything in Elisp” view and towards “drive scripts from Elisp and slurp results into a buffer for processing”

                        Please not, dash and s … don’t always lend themselves to idiomatic Elisp

                        Also agree. Writing idiomatic Elisp using the built in APIs isn’t that hard, but it’s more “old skool” and can be imperative/verbose, so I guess some people don’t like it

                        On the other hand I think it’s kind of dumb to have to import a “cool” string library to install a library that does something I actually want.

                        Elisp isn’t Clojure and I’m glad for that. I think a lot of the complaints are that it isn’t Clojure/CL/etc, but its not supposed to be. It’s basically a 70s era MacLisp clone.

                        In general I think two (no three) things:

                        1. People can write whatever they want to write, it’s free software, their itch etc
                        2. I wish a lot of the people trying to make Emacs “better” (by their definition) but who are also possibly adding complexity to the code base, changing variables and APIs and default settings etc would just chill and learn more about what it is they’re trying to change before concluding “this needs feature X, let’s hack it in and now everyone has to live with it”
                        3. I know #2 is something of a straw man and not 100% fair but Emacs is where I live all day every day and somebody else’s “cool new feature” is my “unnecessary risk and complexity” (and possibly lost hours troubleshooting)
                        1. 5

                          Take a look at the screenshots of these Neovim GUIs:

                          I don’t see any notable difference, tbh. What I would get is better integration into window managers (opening downloaded files in existing emacs sessions), but I have never understood what neovim actually achieves by splitting UI and backend. Sure, it’s a good design decision in principle, but considering the complexity of Emacs display engine, I’m not sure if it is worth it.

                          The exactly same thing has been said for vim, yet neovim managed to pull if off. Just for the sake of it being a great design decision, it would be worth implementing it. The case in points are so many different forks of emacs, trying to make it run on different platforms (emacs-mac, xemacs, …).

                          1. 4

                            I don’t get how this should work, when setting up LSP servers is now the issue. If you want to use rust-analyser, you have to download rust-nightly.

                            This imo is exactly the problem which should be solved by editor’s extension. In VS Code, the user just installs the rust-analyzer extension and the binary is downloaded automatically. IIRC, coc-nvim provides a similar experience.

                            It is important that LSP support in an editor allows for each language to have a dedicated, easy discoverable plugin which deals with dirty integration bits.

                            1. 4

                              I’ve had pretty hit-or-miss luck with things actually working in VS Code when you do that. Just yesterday I tried using it for C# for the first time, responded “yes” to the prompt when it asked if I wanted to install the recommended C# extensions, and now I have some kind of dependency hell around .NET version mismatches (“Error: The reference assemblies for .NETFramework,Version=v4.7.1 were not found.”). Some of the language extensions install without problems, but I’ve come not to expect it.

                            2. 3

                              If you want to compile a Haskell LSP server, you’ll need to buy some more RAM.

                              haskell language server provides precompiled binaries that transparently download for vsode, and lsp-mode is working on reusing this ability.

                              You can also install prebuild binaries via ghcup

                              1. 3

                                That’s basically the same problem as with rustup, that I have to install software from external sources, instead of having it all done by my (system’s) package manager.

                                1. 3

                                  You can consider using Nix to declaratively create project development environments that includes build tools like LSP servers, which is exactly what I do in my open source Haskell project (run nix-shell --run "code ." and things “just work”, be it on Linux, macOS or WSL).

                              2. 2

                                I used to have these problems, but in my experience this comes from fighting Emacs instead of cooperating with it. Using unnecessary packages, weird themes and wanting popups everywhere will simply slow stuff down.

                                I don’t think it’s productive to put blame for these problems on the user. Extensibility is the main selling point of Emacs for many, people use it because they want those features. Of course it’s always possible to shot oneself in the foot. But if some of the problems are solvable on the level of platform, they should be solved there. When I pick a theme I want to pick it for its look and not think if it will slow down my editor. If an extension delays startup, becomes unresponsive or slows down every key stroke I want to be told by Emacs about what’s to blame but I also want it to be hard to achieve.

                                It shouldn’t be as easy to bring the editor to a crawl. The user doesn’t have to be an expert or to use vanilla Emacs for it to be fast. The machine I’m typing this on has 8 cores, I don’t see why crazy computations to paint parentheses in rainbow colours should slow down my typing or scrolling. I think support for asynchronous execution is one of the main reasons why Neovim succeeded.

                                1. 1

                                  Hard agree.

                                  1. 1

                                    Extensibility is the main selling point of Emacs for many, people use it because they want those features.

                                    What features? My comments are more related to UI stuff like pop-ups or constant redrawing effects – in other words non-functional aspects. They can be nice and fun, but since most of the time they are hacks, it’s nor surprising that they aren’t efficient.

                                    If an extension delays startup, becomes unresponsive or slows down every key stroke I want to be told by Emacs about what’s to blame but I also want it to be hard to achieve.

                                    The problem here is that there is no such thing as an extension. There is no means to distinguish between the core and (so-called) extension, because they are equal.

                                    It shouldn’t be as easy to bring the editor to a crawl.

                                    As an editor, yes, but again Emacs is a fully programmable environment. Evaluate

                                    (defalias 'eval #'identity)
                                    

                                    and slowly everything will start to break. Emacs’ power necessarily includes the ability to break stuff. The barrier can be lowered, but adding more (finite) computing power doesn’t solve the “issue”.

                                  2. 2

                                    If you want to use rust-analyser, you have to download rust-nightly.

                                    rust-analyzer provides pre-built binaries

                                    1. 1

                                      But if this would be solved, and all I have to do is run “sudo package-manager install java-lsp-server”, then it would be a considerable advancement.

                                      I’ve only tried clangd, which I was able to just grab via my distro’s package manager and it just worked (other than that every few hours clangd gets OOM killed…)