Threads for ilyash

  1. 0

    I have decided against method aliases in my programming language, Next Generation Shell, eliminating exactly this type of issues.

    1. 4

      This is the first time I see an attempt to do this. A robust language with all the constructs and unambiguity syntax of general.purpose programming languages,.but attempting a direct syntax for.command invocation.

      The documentation is clearly a out this. The choice of inspiration (lua) is a fair one, althou I think something a bit more minimal in terms of delimiters and separators is what people want ultimately. But I understand it’s not easy to strike a balance. And to be fair, the multi line syntax for commands is well thought through and much more convenient than putting a slash in the end of each line.

      Good luck with the project, but please: put the part about command invocation and redirection at the begining of the documentation. All that walkthrough about language constructs, doesn’t set the language appart from all other languages.

      Make a cookbook of examples illustrating the killer features.

      1. 4

        This is the first time I see an attempt to do this

        That means I have failed miserably in marketing / PR

        https://ngs-lang.org/

        1. 3

          There are many shell languages that have features from general purpose languages like Python:

          https://github.com/oilshell/oil/wiki/Alternative-Shells

          Including Oil: https://www.oilshell.org/release/latest/doc/oil-language-tour.html

          And to pick 3 from the “Python-like” section

          https://github.com/abs-lang/abs

          https://koi-lang.dev/#fn

          https://github.com/alexst07/shell-plus-plus

          Hush appears to be very similar to these, except it’s more based on Lua. (At a quick glance it appears to be missing some of the good parts of shell – i.e. that it composes with external processes and the file system.)

          1. 1

            A robust language with all the constructs and unambiguity syntax of general.purpose programming languages,.but attempting a direct syntax for.command invocation.

            What about powershell?

          1. 6

            I don’t know fish very well but, pardon my asking, why?

            There’s a commonly held belief that one should at least think long and hard about using a more established programming language rather than trying to code in the shell. This feels like an even longer slide down a slippery slope.

            Or am I missing something about fish shell’s magical-ness?

            1. 13

              Yeah, the only reason anyone ever writes scripts in shell is that it’s guaranteed to already be installed, which isn’t true of fish.

              Plus the I in fish stands for “interactive”; like … that’s very clearly what it’s designed for.

              1. 4

                [T]he only reason anyone ever writes scripts in shell is that it’s guaranteed to already be installed

                This right here.

                Fish is very much something I would install on my personal laptop, but on the environments where I run a lot of shell scripts, it’s either bash or a higher level language like Python depending on the complexity.

                1. 2

                  Personally another reason I use shell scripts is cuz there are nice primitives. Every machine I use has Python 3 installed, but subprocess.Popen isn’t the funnest interface.

                  I would love to have a shell script language that is fun to use and isn’t Powershell

                  1. 1

                    Yes, process management in your average language sucks compared to bash. That’s why you can find libraries for $YOUR_LANGUAGE that do it significantly better than the standard library of that language. In your case, google for “python shell scripting library”.

                    If you are like me and believe that if something is too frequent it should get its own syntax and in general prefer a language built ground up for this type of scripting, you are welcome to try Next Generation Shell (I’m the author).

                  2. 1

                    Yeah, the only reason anyone ever writes scripts in shell is that it’s guaranteed to already be installed, which isn’t true of fish.

                    I don’t really understand what the problem is here. OK I do not work with servers in my daily job but I’m pretty sure that most people do install some software first on those machines. Why not add fish to the list of web servers/JS packages/Python packages/etc…? I mean who runs their system bone stock (except on OpenBSD ;) ) so what’s the problem with installing fish?

                    1. 4

                      There is a spectrum, but with four important points:

                      • Installed everywhere (in the default install). This basically means POSIX shell for *NIX systems.
                      • Almost always installed, not necessarily by default, and small. Things like bash are in this list.
                      • Not always installed but a simple package with no dependencies and so low friction to install. Something like Lua is a good example here.
                      • Not always installed and brings in a messy load of stuff with versioning problems. Python is the common example here.

                      Each step in this list adds some potential friction and so should come with some benefit. There are some nice bashisms that make shell scripting easier and bash is pretty easy to install on any system that has a POSIX shell, so it’s often worth using !#/usr/bin/env bash instead of !#/bin/sh for your shell scripts. Using a real programming language such as Lua has some bigger benefits but means that users probably need to install it.

                      Something like fish or zsh is in an interesting place because it’s as much effort as installing a programming language such as Lua or Go (easier than one like Python or Ruby), but generally the uplift in functionality relative to POSIX shell or bash are fairly small.

                      If your scripts are really scripts (i.e. mostly running other programs) then a shell scripting language may be a better fit than a programming language, but now you need to think about how much easier fish is than POSIX shell or bash and whether the benefit outweighs the additional friction for users.

                      Note that the friction also implies to security audits. If someone is deploying your software in a container and it depends at run time on a scripting language, then that language implementation is now part of your attack surface. Things like Shellshock showed that bash isn’t always a great choice here, but it’s probably already in the attack surface for other things and so will be on the list of evaluated risks already. The same is true of something popular like Lua or Python (though that may not be a good thing as it may already be flagged as high risk). Something like fish will likely trigger extra compliance processes.

                      1. 3

                        It’s not that there’s a problem with installing fish; it’s that if you’re going to bother with having prerequisites, you’ve just discarded the one advantage shell programming has over regular programming languages.

                        1. 2

                          Bit of a chicken and egg problem though, you have to install those interpreters and what would you install it with?

                          At some point you’re interacting with the OS primitives. The most elegant thing at that level is sh so you’re already using it.

                          1. 1

                            But is it really the most elegant? Or just lazily more convenient?

                            1. 1

                              Usually those are the same thing.

                              Ultimately there will always be some shell running. The only way to get around that is ansible (and some kind of priming system) or — well arguably it can get a lot more more complicated to do very simple things.

                              Maybe someone should make a Linux distro with fish under the hood?

                            2. 1

                              At some point you’re interacting with the OS primitives. The most elegant thing at that level is sh so you’re already using it.

                              Huh? In POSIX systems, the system libc function invokes a shell, but [v,pd]fork / clone + execve doesn’t go anywhere near the shell and these expose things that the shell does not. Any higher-level language that exposes the process-creation system calls directly will give you more control than the shell and will not depend on the shell for anything.

                              Unlike VMS or DOS, the shell is not in any way special on a *NIX system, it’s just a program that talks to a terminal (or something else) and does some system calls. This is why things like fish can exist: the program that you use for sitting between you and the operating system’s terminal facilities can be anything that you want.

                        2. 12

                          I wouldn’t say fish is magical, it just seems like a small, sane shell language with fewer thermo-nuclear footguns. On paper it should be an ideal candidate for project automation and general scripting.

                          EDIT: I guess my point is that we often treat perl/ruby/python as “a better bash”, while it seems to me that fish should be a candidate for the same role.

                          1. 2

                            From a strictly technical standpoint I can see where you’re coming from, but think about this in terms of commercial viability.

                            You do what you’re suggesting and create a pile of snowflake Fish shell code to run the company.

                            You then leave a year later for greener pa$$tures, and the next person inherits a technically superior codebase which sadly resolves into a perfect black hole of sadness and despair because NO-ONE anywhere uses this tool this way and NewPerson is totally, UTTERLY alone.

                            1. 2

                              Yes. Easier to use correctly. If bash is a quoting hell where the right thing to do – using arrays – is awkward, fish trivializes both: Compare "${array[@]}" (bash) with $array (fish): Every variable is an array and you don’t need to quote it.

                              1. 1

                                Being not POSIX compatible, fish shell scripts and bash scripts are not compatible.

                                This was a deliberate choice to be better for interactive use, which makes fish an immature and non appropriate choice for a shebang.

                            1. 3

                              I’m trying to materially improve the conditions of using literally any language other than C.

                              My problem is that C was elevated to a role of prestige and power, its reign so absolute and eternal that it has completely distorted the way we speak to each other.

                              This is highly related to my recent post, A Sketch of the Biggest Idea in Software Architecture

                              In those terms:

                              • Various C ABIs are haphazardly defined and evolved. But they are implicit narrow waists because so much code is written in C. Practically speaking to solve an interoperability problem between Rust and Python, or Swift and Ruby, you will need to speak some kind of C API or C ABI.
                              • Type systems have limitations when your program is written in multiple languages!
                                • Here’s the kicker: ALL programs that are not written in C depend on the semantics of multiple languages! Because the kernel interface is specified in C on both Unix and Windows as a bunch of header files. (If your program doesn’t use the kernel at all, then you’re exempted from this … it’s also likely not a very interesting program :-) )
                                • If you want things to look like Swift or Rust functions, you will be sorely disappointed when doing runtime composition with C code.
                                • runtime composition with ABIs gives you bad error messages
                                • build time composition is also hard because you need a whole C compiler …

                              So basically we need a precisely specified narrow waist, not a haphazardly evolved one … But we will still need narrow waist. You inherently have an O(N x N) explosion of languages trying to talk to each other.

                              I find that programmers misunderstand this a lot… they think because a Python functions have roughly similar syntax to a Rust function, or to C functions, it should be “easy” to interoperate. But the semantics are completely different … The lowest common denominator ends up being very low, all though you can special case for instances where you’re only passing integers, etc. If you’re trying to pass big heap allocated structures then you have a big problem !

                              i.e. basically the lowest common denominator ends up more like message passing and IPC. It doesn’t look like RPC very much

                              (copy of Reddit comment)

                              1. 3

                                Because the kernel interface is specified in C on both Unix and Windows as a bunch of header files.

                                Linux at least defines system calls in terms of numbers and buffers. I have no doubt that they were heavily influenced by C, but how we make a Linux system call is now independent from C.

                                I’m not sure Windows even has a stable kernel ABI. Instead we’re supposed to link to system dlls.

                                So basically we need a precisely specified narrow waist, not a haphazardly evolved one

                                Oh yes we do. Also it would be so nice for this waist to be narrower still: while it’s nice to have one ABI per platform, it’s a bummer when we have over 170 platforms…

                                1. 2

                                  Yes Linux maintains a stable kernel ABI, unlike the BSDs. Windows does too BTW – you can run 30 year old binaries on modern Windows machines.

                                  I didn’t fully explain it, but that is what I meant by these parts:

                                  http://www.oilshell.org/blog/2022/03/backlog-arch.html#characteristics-of-narrow-waists

                                  POSIX C APIs → Linux x86 ABI

                                  The narrow waist was supposed to be C, but that doesn’t help other languages. And C isn’t defined unless you add in the ABI emitted by the C compiler for a particular architecture, so we got the Linux x86 ABI as our de facto standard / narrow waist for a large class of apps.


                                  Justification:

                                  • Windows emulates Linux with WSL
                                  • FreeBSD and Illumos also emulate Linux:

                                  Another example is when Illumos borrowed FreeBSD’s Linux syscall ABI emulation in order to run user-uploaded Docker containers. This is dynamic, runtime composition with ABIs, not static composition by compiling code against kernel APIs expressed as C header files.

                                  On the other hand, Win32 is becoming a de facto narrow waist for GUI apps on Linux, also mentioned in the post.


                                  So I think I need to coin a term like “sloppy waist” or “messy waist” … Something we didn’t design, but just arose through “Emulation of Waists”.

                                  1. 5

                                    Narrow waist vs free as in beer belly.

                                    1. 2

                                      Ha, a beer belly could be a fitting image … although I think it is better to avoid any body connotations – the narrow waist is for an hourglass, not a person :-)

                                      A lot of the “accidental” waists tend to come from whatever companies happen to be successful, e.g. Microsoft and Intel.

                                      I think a funny thing is that sometimes those companies do have clean slate efforts. I think Intel’s Itanium was supposed to fix a lot of problems with the ISA … but they probably got killed by their more successful older brother.

                                    2. 2

                                      Nitpick: The Windows kernel ABI as in syscalls was historically at least, not stable - but the userland libraries were.

                                      1. 2

                                        Yes Linux maintains a stable kernel ABI, unlike the BSDs. Windows does too BTW – you can run 30 year old binaries on modern Windows machines.

                                        Windows does have a stable ABI for sure, I just heard it didn’t have a stable kernel ABI. That the stable layer was a little bit higher, and was provided by system DLLs instead of the kernel directly. But that was a nitpick anyway. What really matters is we have a stable ABI somewhere.

                                        1. 1

                                          “accidental waist”?

                                    1. 9

                                      Modern Bash Scripting is implementing it instead in Go/Python/<insert language you like> for anything longer than 4 lines.

                                      1. 2

                                        I can shamelessly recommend Next Generation Shell which was designed to be exactly that language.

                                        https://github.com/ngs-lang/ngs

                                      1. 1

                                        Author here. AMA here or in Discord - https://discord.gg/6VqgcpM

                                        1. 2

                                          Hm. Interesting! But what can I do with this that I can’t do with either Bash or my favorite scripting language?

                                          1. 3

                                            It’s not about what you can or can not do. It’s about how it would look like. NGS is best for the intended use cases. The strength of the language is the “focus” on the domain.

                                            Compared to bash, NGS has sane syntax, proper error handling, data structures. The things that are expected from anything modern. I feel like comparison to bash is not very fair to bash. It comes from another era, another set of expectations, other norms. Nobody today would allow $x to represent zero or more arguments depending on the data in $x.

                                            Compared to general purpose programming languages, NGS has syntax and facilities for the “DevOps”y scripting domain: syntax for running external programs, handling of exit codes (surprise: not any non-zero exit code is an error), assert(Program("gsed")), warn("..."), error("...") and other domain specific things. Think jq or awk - they are just better for specific use cases.

                                            Edit: as you don’t mention Java or Assembler because bash and a scripting language are more suited for the task at hand, I claim that for some use cases, NGS is even better fit. See intended use cases - https://github.com/ngs-lang/ngs/wiki/Use-Cases

                                          1. 1

                                            Below is how you “integrate” jc with Next Generation Shell.

                                            data = ``jc PROGRAM ARGS ...``
                                            

                                            The double-backtick syntax runs the external program, jc in our case, and parses the output. It means that the “integration” is not jc specific.

                                            data is now structured data that comes from the parsed JSON.

                                            Example (run from your shell):

                                            ngs -pl '``jc ifconfig``.filter({"name": /docker/}).ipv4_addr'
                                            

                                            Will print IPs of all docker interfaces, one IP per line

                                            1. 5

                                              Yeah. I like this a lot, an extensible crutch until first class JSON support lands in GNU utils.

                                              Frankly passing objects via pipes is one of those things that Powershell gets right, and makes writing powershell feel somewhat elegant compared to bash pipes.

                                              The jq syntax is a bit difficult for me at times still though; but the good thing about json is that I don’t have to use jq.

                                              I think this is super. :)

                                              1. 2

                                                don’t have to use jq.

                                                Here is my (I guess controversial) opinion on jq: https://ilya-sher.org/2018/09/10/jq-is-a-symptom/

                                                TL;DR - handling JSON should be in the shell.

                                                1. 2

                                                  I’d say json is a symptom that you haven’t thought through your data model enough to use sqlite.

                                                  If you want to know what SQL is a symptom of…. read CJ Date. He has decades worth of books worth of rants on the subject.

                                                  1. 2

                                                    I’d say json is a symptom that you haven’t thought through your data model enough to use sqlite.

                                                    I’m not entirely sure what this means. How do you propose to pass data between two processes not on the same machine using sqlite?

                                                    1. 1

                                                      An SQLite DB is just a tight text file. If there’s any real structure in your data, it’ll be smaller than a JSON file.

                                                      JSON is the winner on the web because you have to use other people’s data models, and you can’t expect them to think it through. We don’t want to learn two ways of dealing with data so it’s used when we have full control too, anyways.

                                                      If you’re gonna integrate structured data in the shell (but somehow don’t want to go full powershell objects), SQLite is probably the better choice.

                                                      1. 2

                                                        Can the sqlite API start an in-memory db from a byte stream? Or do you mean to stream the DDL between processes?

                                                        1. 1

                                                          Can the sqlite API start an in-memory db from a byte stream?

                                                          SQLite itself AFAIK not, but you can do this: Reading SQL scripts:

                                                          cat script.sql | relpipe-in-sql | relpipe-out-tabular
                                                          
                                                        2. 2

                                                          It’s less about sql or sqlite itself or json, it’s more about the notion of a relational algrebra that Codd envisioned, and that sql is a stonking Bad implementation of (but the best we have at the moment).

                                                          TL;DR; we got so badly betamaxed by SQL that the better standard never even emerged.

                                                          1. 1

                                                            Sorry, I don’t get it. If I want to send an RPC to a server and read a response. are you suggesting I should construct an entire SQLite database and send that? And then parse the database on response, even if it’s just an object with a key to indicate success/failure and one for the error message?

                                                            Also, in general, I’ve found deeply nested data to be much more pleasant in JSON than in any SQL. No need to mess around with joins when I can just access a property on an object directly.

                                                            1. 1

                                                              No, not really. A DB-first world would have had answers for that but we live in a different world. JSON is better in this world for plenty of reasons.

                                                  1. 1

                                                    Among other things, this release contains work on pattern matching.

                                                    Release notes - https://github.com/ngs-lang/ngs/releases/tag/v0.2.11

                                                    More about pattern matching design in NGS - https://github.com/ngs-lang/ngs/wiki/UPM-Design

                                                    I would appreciate your thoughts about the pattern matching. I’m having hard time to “get it right”. I’ve looked at many other implementations (listed at the “UPM-Design” link above) and all of them feel “not 100%”. Unfortunately, right now, I am failing to come up with my own design that would feel “100%”. Biggest next obstacle is syntax (smaller) and semantics (bigger) of capturing values.

                                                    Problems:

                                                    • NGS does not have a reference type and I’m not fond introducing one just for patterns to be able to return values.
                                                    • A syntax (example @a) for creating a function which would set a variable (F(val) a=val) would introduce additional problem - to which scope to add the a variable. According to language rules, it would be set inside the F..., which is not useful at all. Maybe in this case it should be “one above F”?
                                                    • Capturing all the values and returning them is a bit verbose, “direct” mention of a variable name is more expressive.
                                                    • Does ability to pass a pattern as value contradicts the ability to capture directly into variables?

                                                    I’ve probably missed some information, feel free to ask.

                                                    1. 2

                                                      Thinking (in order to design) about Universal Pattern Matching (for the lack of better name)

                                                      https://github.com/ngs-lang/ngs/wiki/UPM-Design

                                                      1. 3

                                                        I’m looking for feedback on this – is it comprehensible / useful? It will probably take a few moments to figure out, but once you do, I think it should scale to large shell programs, and has many advantages over shell tracing.

                                                        More examples here:

                                                        https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/xtrace_rich.20.3A.20Oil's.20enhanced.20tracing

                                                        (requires login, feel free to chat with me about it) Also note this is NOT released yet, but you can build it from the repo if you want. I can help

                                                        Another good example:

                                                        $ bin/osh -O oil:basic -x -c 'echo foo; sleep 0.1 & sleep 0.2 & echo bar; wait'
                                                        . builtin echo foo
                                                        foo
                                                        | fork 31218
                                                          . 31218 exec sleep 0.1
                                                        | fork 31219
                                                        . builtin echo bar
                                                        bar
                                                          . 31219 exec sleep 0.2
                                                        > wait
                                                          ; process 31218: status 0
                                                          ; process 31219: status 0
                                                        < wait
                                                        
                                                        1. 2

                                                          That is pretty nice, I usually debug shell scripts using echo statements. Tbh I completely forget about the trace setting most of the time.

                                                          1. 1

                                                            Yeah the tracing in bash is “OK”, I use it reluctantly…

                                                            I think this is a lot better and could lead to more tools on top! The format should be parseable so many it can be visualized with timing. Could be useful for big continuous builds.

                                                            It will be out soon! Let me know if you have other feedback

                                                            1. 1

                                                              In NGS this could be debug("blah") or debug("myfacility", "blah") which you could leave in and turn on and off with the DEBUG environment variable. The output of debug() goes to standard error, yet another convenience over bash where you should redirect the output of echo.

                                                              1. 1

                                                                In bash I just do this, then use it like log "x = $x".

                                                                log() {
                                                                  echo "$@" >&2
                                                                }
                                                                

                                                                You can also wrap that in if test -z "$DEBUG", etc.

                                                                This might be built in to Oil, i.e. https://github.com/oilshell/oil/issues/750

                                                                1. 1

                                                                  In bash I just do this

                                                                  On top of many scripts. That’s why I moved this to standard library. It’s just common.

                                                                  You can also wrap that in if test -z “$DEBUG”, etc

                                                                  Again, I find it’s common enough to deserve to be in standard library.

                                                          1. 1

                                                            Looks like this is yet another feature which highlights our difference in approaches. Oil is heavily rooted in the shell world. Next Generation Shell tries first to be a programming language. Oil has set -x while NGS went with DEBUG environment variable, modelled after https://www.npmjs.com/package/debug ( DEBUG has comma separated facilities to trace, * for everything).

                                                            It is interesting to see how both projects trying to solve mostly same problems but in different ways.

                                                            1. 1

                                                              I don’t think debug() is quite the same thing, because you have to insert those statements right? This kind of tracing doesn’t use any annotations.

                                                              Does NGS have a tracer that logs every process start and exit, with PID and status? i.e. every component in a pipeline? That would be the equivalent to this.

                                                              (Whether it looks like set -x or DEBUG=1 seems like a cosmetic issue. The idiomatic to write it in Oil is shopt --set xtrace , or shopt --set oil:basic xtrace, but the shell style is supported too.)

                                                              1. 1

                                                                I don’t think debug() is quite the same thing, because you have to insert those statements right?

                                                                Yes, you have to insert the statements. On the other hand standard library which handles running external processes already has these debug()s.

                                                                Does NGS have a tracer that logs every process start and exit, with PID and status?

                                                                Yes:

                                                                $ DEBUG=process ngs -e '$(ls | wc -l)'
                                                                [DEBUG process 237725 main] Parsed command: [ls]
                                                                [DEBUG process 237725 main] [find_in_path] got ls
                                                                [DEBUG process 237725 main] [find_in_path] will search
                                                                [DEBUG process 237725 main] [find_in_path] ls found at <Path path=/usr/bin/ls>
                                                                [DEBUG process 237725 main] PID after fork: 237733
                                                                [DEBUG process 237733 main] PID after fork: 0
                                                                [DEBUG process 237725 main] Reading all output of the child process
                                                                [DEBUG process 237725 main] Parsed command: [wc,-l]
                                                                [DEBUG process 237733 main] will execve()
                                                                [DEBUG process 237725 main] [find_in_path] got wc
                                                                [DEBUG process 237725 main] [find_in_path] will search
                                                                [DEBUG process 237725 main] [find_in_path] wc found at <Path path=/usr/bin/wc>
                                                                [DEBUG process 237725 main] PID after fork: 237734
                                                                [DEBUG process 237734 main] PID after fork: 0
                                                                [DEBUG process 237725 main] Reading all output of the child process
                                                                [DEBUG process 237725 main] Creating thread process-237734-output-fd-1-reader
                                                                [DEBUG process 237725 process-237734-output-fd-1-reader] Thread set up done, will run thread code
                                                                [DEBUG process 237725 main] [wait] joining reading and writing threads
                                                                [DEBUG process 237725 main] [wait] will waitpid(237733)
                                                                [DEBUG process 237725 main] [wait] waitpid(237733) -> [237733,0]
                                                                [DEBUG process 237725 main] [wait] joining reading and writing threads
                                                                [DEBUG process 237725 main] Joining thread <Thread process-237734-output-fd-1-reader>
                                                                [DEBUG process 237734 main] will execve()
                                                                [DEBUG process 237725 process-237734-output-fd-1-reader] Read all output of the child process for descriptor 1, closing reading/parent end
                                                                [DEBUG process 237725 main] [wait] will waitpid(237734)
                                                                [DEBUG process 237725 main] [wait] waitpid(237734) -> [237734,0]
                                                                

                                                                I think the output of Oil’s set -x is more elegant.

                                                                Whether it looks like set -x or DEBUG=1

                                                                Except that it’s DEBUG=facility1,facility2,... or DEBUG=*, which gives more flexibility.

                                                            1. 8

                                                              What’s described in the article is a particular type of “information loss”. You have valuable information and you throw it away.

                                                              Examples, tutorials, and blog posts should favor Either over Maybe

                                                              Developers should be careful about not overusing Maybe

                                                              … and more generically (and in other languages), not to throw away valuable information. (Empty catch clause is the poster child here.)

                                                              I was thinking about that and posted on my blog: https://ilya-sher.org/2019/05/18/on-information-loss-in-software/

                                                              Parts of “Maybe Considered Harmful” would fit well in the “Information Loss at Runtime” section of my post.

                                                              1. 2

                                                                On Information Loss in Software (My talk at PerlCon 2019, under 25 minutes)

                                                              1. 1

                                                                Adding the =~ (match) and !~ (doesn’t match) operators to Next Generation Shell (programming language for Ops that I’m working on since 2013).

                                                                In parallel I’m thinking about huge related topic: unified pattern matching. There are many mechanisms for pattern matching: regular expressions, match operators or functions, case statements, parsers such as PEG, multiple-dispatch/overloading, etc. Note the less obvious pattern matching functions such as take_while, drop_while, unpack, scanf, etc, which I suspect are rarely though as pattern matching functions.

                                                                I’m thinking about unifying many of these mechanisms into one coherent facility. I assume it will be very nice to have it in Next Generation Shell. That’s because I think that huge chunks of programming are dealing exactly with that: matching patterns and reacting to them.

                                                                Interesting challenges are: make the facility coherent and easy to understand and operate. How in the world you match non-flat data structures such as Map (aka Hash, aka dictionary)?

                                                                1. 6

                                                                  sh is a cute-sy interface for shell scripts in Python. Examples from the docs:

                                                                  sh.git.show("HEAD")
                                                                  sh.wc(sh.ls("-1"), "-l")
                                                                  p = sh.find("-name", "sh.py", _bg=True)
                                                                  # ... do other things ...
                                                                  p.wait()
                                                                  
                                                                  1. 3

                                                                    Plumbum is usually the Python shell DSL I see recommended. Haven’t used either though, as subprocess.run is generally good enough.

                                                                    1. 2

                                                                      I feel like I’ve written 60% of that library in a less useful way several times. I need to remember that for next time. Thanks!

                                                                      Also, it begs for a sibling that wraps paramiko in an ssh interface.

                                                                      1. 1

                                                                        Doesn’t look as convenient as

                                                                        p = $(find -name sh.py &)
                                                                        ...
                                                                        p.wait()
                                                                        

                                                                        [ That was shameless plug, that’s NGS ]

                                                                      1. 12

                                                                        I mostly use shell scripts because it’s almost always available and doesn’t change much; although I don’t really like it, I don’t really know of anything better as most alternatives are:

                                                                        1. Somewhat cumbersome to quickly write scripts in.
                                                                        2. Unlikely to already be installed.

                                                                        Ruby does support running shell commands with backticks:

                                                                        irb(main):005:0> `grep $USER /etc/passwd | cut -d: -f5-`
                                                                        => "Martin Tournoij:/home/martin:/bin/zsh\n"
                                                                        

                                                                        And general string processing etc. in Ruby is fairly nice as well. I don’t really use it for stuff like this, mostly because of point 2.

                                                                        Perl can do this as well I believe (and was originally conceived to replace shell scripts), but I never really got to gripes with the syntax. For many purposes, I consider Ruby to be “Perl done better”, but Perl is much more likely to be installed on most systems (and Ruby still has a rather complicated syntax and a bit of a learning curve).

                                                                        Tcl is nice for interacting with shell commands as well:

                                                                        exec grep $env(USER) /etc/passwd | cut -d: -f5-
                                                                        

                                                                        But I never worked much with it, and it suffers from point 2 as well (although it’s a lot more light-weight than Ruby, and easier to install).

                                                                        There are other shells as well, such as rc and oil, but they’re even less likely to be already installed. Alpine Linux, for example, doesn’t even seem to have packages for either in the standard repos.

                                                                        1. 3

                                                                          rc is available in alpine linux: it’s in the 9base package

                                                                          1. 2

                                                                            Ah right; I just searched for rc.

                                                                            1. 1

                                                                              pacman -F is good for finding files in packages that you have not installed.

                                                                          2. 2

                                                                            tcl is GREAT, and whilst tclsh isn’t as quite likely to be installed as perl, I think the expect (especially autoexpect) is a killer application for operations/composition purposes.

                                                                            1. 1

                                                                              grep $USER /etc/passwd

                                                                              Demonstrates two bombs in one place.

                                                                              (1) $USER can be expanded into two arguments

                                                                              irb(main):005:0> ENV["X"]="a b"
                                                                              irb(main):007:0> `printf "%s|%s|" $X`
                                                                              => "a|b|"
                                                                              

                                                                              (2) When $USER is a substring of another user … or group … or anything.

                                                                              That’s why I prefer data structures. Yes, I know, Unix philosophy, text is the common denominator with ton of tools, composable, etc..

                                                                            1. 1

                                                                              Around 2013, I’ve started working on Next Generation Shell. Frustrated with bash and Python, which I was using extensively, I looked at the niche of what’s on intersection of modern and for Ops. I’ve seen nothing.

                                                                              While bash was definitely for Ops, it did not meet any expectations for “modern”: horrible syntax, poor error handling and missing (OK, very limited) data structures. Using jq from bash felt like a symptom.

                                                                              On the other hand there was a bunch of general purpose programming languages such as Python, Ruby, etc. Well .. they are not specifically for Ops and in practice that means that dealing with files and processes is unnecessary verbose. In NGS for example, if you run a process, it’s an exception if it exits with an error exit code. Note that error exit code is not anything above zero. There are commands such as test where exit code 1 is fine.

                                                                              Over the time I’ve discovered several projects in that modern-for-Ops niche. Somehow the feeling is that they are trying to improve over existing shell while NGS is based on the idea of creating a programming language for ops … with a shell.

                                                                              PowerShell was not on my radar for some time but when I did look at it I’ve seen horrible language design choices such as -eq comparison which is not equality comparison.

                                                                              But if you need to determine whether a variable is $null, you must put $null on the left-hand side of the equality operator. Putting it on the right-hand side does not do what you expect.

                                                                              (Emphasis mine).

                                                                              Back to NGS.

                                                                              Project status: useful programming language which we use at work; starting working on the shell.

                                                                              I hope that other people will find NGS to be a helpful tool.

                                                                              1. 9

                                                                                For me, Julia recently completely replaced bash&Python for shell scripting. There are two reasons for that:

                                                                                Of course, you need to have Julia installed. It also starts a bit slower than Python. This used to be a deal breaker, but was mostly fixed in one of the versions, it is now fast enough for short interactive scripts.

                                                                                Here’s a example from my dot files: a script to remove merged branches:

                                                                                https://github.com/matklad/config/blob/master/bin/gbda

                                                                                1. 5

                                                                                  Hm that’s a good use case. FWIW in Oil this will look like:

                                                                                  git branch --merged | while read --line {
                                                                                    if (_line != "master" and not _line.startswith('*')) {
                                                                                      echo $_line
                                                                                    }
                                                                                  } | readarray :branches
                                                                                  
                                                                                  
                                                                                  if (len(branches) == 0) {
                                                                                    echo "No merged branches"
                                                                                  } else {
                                                                                    git branch -D @branches
                                                                                  }
                                                                                  

                                                                                  (Not tested, but the only thing that might not be there is the startswith method)

                                                                                  Differences from bash:

                                                                                  • curly braces
                                                                                  • Python-like expressions for both if statements
                                                                                  • read --line fills $_line, instead of read -r
                                                                                  • readarray works at the end of a pipe because shopt -s lastpipe is on by default
                                                                                  • :branches has a : sigil to remind you it’s a variable name
                                                                                  • Many semantic problems are fixed with arrays. They’re very broken in bash.
                                                                                  • @branches for splicing instead of "${branches[@]}"

                                                                                  I might make a blog post out of this! I posted some notes on Zulip here https://oilshell.zulipchat.com/#narrow/stream/266575-blog-ideas/topic/Julia.20Shell.20Script (requires login)

                                                                                  1. 4

                                                                                    Yeah, pipeline concrete syntax for function composition reads better to me here. Alas, it is almost available in Julia: https://discourse.julialang.org/t/why-are-there-no-curried-versions-of-map-filter/47478

                                                                                  2. 2

                                                                                    Curious about your workflow for this. I tried to use Julia to replace shell scripting for myself but found that many scripts (especially anything with any dependencies) would take 30s+ to run.

                                                                                    1. 3

                                                                                      I don’t have any workflow, it’s just fast enough for me:

                                                                                      19:16:54|~/projects/rust-analyzer|t⚡*
                                                                                      λ bat -p (which gbda)
                                                                                      #!/usr/bin/env julia
                                                                                      
                                                                                      branches = filter(strip.(readlines(`git branch --merged`))) do branch
                                                                                          branch != "master" && !startswith(branch, '*')
                                                                                      end
                                                                                      
                                                                                      if isempty(branches)
                                                                                          println("no merged branches")
                                                                                      else
                                                                                          run(`git branch -D $branches`)
                                                                                      end
                                                                                      
                                                                                      19:17:29|~/projects/rust-analyzer|t⚡*
                                                                                      λ julia -v
                                                                                      julia version 1.5.3
                                                                                      
                                                                                      19:17:43|~/projects/rust-analyzer|t⚡*
                                                                                      λ t gbda
                                                                                      no merged branches
                                                                                      real 0.12s
                                                                                      user 0.09s
                                                                                      sys  0.02s
                                                                                      rss  171912k
                                                                                      

                                                                                      In absolute terms, 120 ms latency here is unreasonable (Python’s equivalent finishes in 20), in human-relative, this is ok enough for me. Unlike Clojure, Julia owns its runtime, so I also hope that this’ll improve in the future (one low-hanging fruit, iirc, is not initing blas&friends at startup).

                                                                                      30s does seem unreasonable. Either this is some kind of config problem, or dependencies are particularity heavy.

                                                                                      1. 1

                                                                                        I manged to get it down to a few seconds IIRC using some tips from https://discourse.julialang.org/t/can-julia-really-be-used-as-a-scripting-language-performance/40384

                                                                                        1. 1

                                                                                          Thanks for the link! Yeah, looks like I am severely underestimating the cost of deps in Julia :-(

                                                                                          Seems theoretically fixable by creating & caching sysimages on the fly…

                                                                                          1. 1

                                                                                            Did you try to use PackageCompiler.jl or your issue does not fit its usage/worth the time to dive into it?

                                                                                            1. 1

                                                                                              I’ve looked at it, but it is too complicated for my use-case. I want “stick this into #! line” solution, not Project.toml solution. I don’t have Project.toml at all.

                                                                                      2. 1

                                                                                        Sorry I’m late to the party.

                                                                                        I’m author of the Next Generation Shell. It was born out of frustration with bash and Python.

                                                                                        Here is how the branches deletion code would look like in NGS.

                                                                                        #!/usr/bin/env ngs
                                                                                        
                                                                                        F strip(s:Str) s - MaybePfx('  ') # Imagine that strip() (but more generic) will be part of stdlib :)
                                                                                        
                                                                                        branches = `git branch --merged`.lines().map(strip).reject(AnyOf('master', Pfx('*')))
                                                                                        
                                                                                        if not(branches) {
                                                                                            echo("no merged branches")
                                                                                        } else {
                                                                                            $(log: git branch -D $*branches) # bonus: "log:" causes the command to be logged to stderr before execution
                                                                                        }
                                                                                        
                                                                                      1. 2

                                                                                        As always, I compare other shells to my Next Generation Shell.

                                                                                        $ cat p.ngs 
                                                                                        #!/usr/bin/env ngs
                                                                                        script = fetch("package.json").scripts[ARGV[0]]
                                                                                        log: bash -c $script
                                                                                        

                                                                                        Output:

                                                                                        ./p.ngs 'say hello'
                                                                                        [LOG 2020-09-30 08:17:45 IDT] Running command: bash -c echo\ \"Hello\ world\!\"
                                                                                        Hello world!
                                                                                        

                                                                                        Similarities:

                                                                                        • No variable expansion bomb. $script would be "$script" in bash
                                                                                        • Data structures
                                                                                        • JSON support

                                                                                        Differences:

                                                                                        • No bash compatibility in NGS (hence bash -c)
                                                                                        • Programming language less similar to bash

                                                                                        Extras:

                                                                                        • log: prefix before command. It is common to want to print the command before execution so it’s in. The language facility is supporting SOMETHING: syntax before commands. The next level is the log itself.
                                                                                        • fetch() knows it’s JSON by looking at extension and you don’t have to specify