1. 60

Hey guys, I have been increasingly frustrated with bash and sh, so decided to try writing a replacement for myself. It is almost ready to be my daily shell, though certainly has issues.

I am submitting it here to see what people think, and hopefully recruit people to help make the project a a serious contender in the shell space via either direct help, or just indirect support.

Feel free to ask questions and I can explain how things work.

  1.  

  2. 14

    Just some feedback on that README: all those animated GIFs would be a lot easier to read if they were just pain text:

    $ some_command
    output
    
    $ other_thing
    more output
    

    It’s just really hard to follow with all the animation, and you can’t look at anything for more than what feels like half a second before the text disappears.

    1. 7

      I’ll make the change. Thanks for the feedback.

      1. 5

        Yes. Animations and videos are never the way to go for answering your audience’s very first question of, “what the heck is this and why should I care?” If your README.me says “play this video to learn about FooBar”, you’ve already lost me as an interested user.

        1. 2

          Personally, I agree, but there are people who would rather watch a video.

          1. 3

            I am in the video camp, my compromise in the end was a single gif followed quickly by text, hopefully it keeps everyone happy now.

        2. 2

          Not to mention that they’re not accessible to people who use a screenreader.

          1. 2

            That is an excellent point that I had not considered. Hard to get out of your own bubble.

        3. 5

          Another fairly mature project in the same vein:

          http://www.oilshell.org/

          1. 8

            I like reading about oil (and I’m a fish user myself) but I’m not sure if either are like this one. Here we see actual, functional lisp/scheme/clojure like code being executed right in the shell. This is pretty unique.

            1. 4

              If I remember correctly, the Oil language hasn’t actually been implemented yet, though; most of the current progress has been on Bash compatibility and the design of the language.

              1. 3

                I never really understood why the oil project spent so much time implementing bash. I mean, I understand why they say they did it, but in reality, I don’t agree with the approach. It really is a monster challenge that I don’t think is necessary.

                1. 4

                  In shortest words, I believe the idea is to pull an EEE (Embrace, Extend, Extinguish), but meant in a good way & intention, on bash. I do believe taking big challenges is worth respect, and I do believe this one has huge potential value. And knowing how bash is entrenched, I do believe Oil’s author is right in that’s the only way that may work to displace bash.

                  1. 2

                    It is definitely a long term plan. I don’t mean to disrespect the skill or effort of oil shell developers, because it is clearly fantastic.

                    If I took that path, I would have almost certainly started with an Oil to bash translator, and worried about a dedicated VM later. More like how coffee script originally took a large number of javascript developers.

              2. 2

                We’re all keeping an eye on it but to say it is mature is premature :). I would not say it is mature in accomplishing the goal of using a different language in the shell.

              3. 4

                If you are looking for alternative interactive shells, also check out Elvish: https://elv.sh https://github.com/elves/elvish

                1. 7

                  Nice, there’s also closh which is a shell based on Clojure that compiles with Graal.

                  1. 10

                    In addition to Closh, which looks pretty similar to Janetsh (both are really cool!), I’d like to recommend checking out two more related projects. First is my own project, Rash, embedded in Racket. Of the shells embedded in general-purpose languages, I think Rash has the best story for extensibility of the shell language and integration of subprocesses and host-language functions (using byte streams in the Unix tradition or host-language objects more in the vein of Powershell). The second is Xonsh, embedded in Python, which I think has the most interactive polish of these embedded shells at the moment.

                    1. 7

                      I was indeed inspired by rash, thank you! Rash is a fantastic project.

                      The two main factors in making this:

                      My first lisp was clojure and I got used to the syntax, which made me dislike scheme, as petty as that sounds…

                      Second thing was that launch time of small scripts was one of my concerns which racket wasn’t so good at. I admit for interactive shells and some programs it isn’t a big deal. Barring those two things, rash is a far more mature, better tool to use.

                      closh itself is hundreds of megabytes to install and starts extremely slowly (multi second, just like clojure), so wasn’t something I could use.

                      1. 2

                        Yeah, Racket’s startup time is not great. It’s one of my least favorite things about Rash…

                        I see you got job control working, and I’ll probably copy whatever you did. I tried for a day or two to get it working some time ago but didn’t find quite the right incantation of syscalls to get it to work right, so I set it aside and never got back to it.

                        If you ever want to talk shop about embedded lisp shells I’m always interested to see or show cool ideas and features. The Closh author and I have talked a bit, he’s a nice guy. Also more friendly competition in the lisp shell world might motivate me to spend more time improving my own shell…

                        At any rate, I look forward to seeing what you do with Janetsh.

                        1. 3

                          I would not have been able to make it work except for reading:

                          https://www.gnu.org/software/libc/manual/html_node/Implementing-a-Shell.html

                          It really is arcane stuff and I did not understand it before starting, there are probably still bugs and lots of edge cases to consider.

                          1. 1

                            Yeah job control is pretty ugly. There is a quote in APUE by Stevens about it, something to effect that it’s a bunch of hacks involving both the kernel, the shell, and the terminal driver… but you have to have it because it’s in POSIX :-(


                            BTW I have also prototyped a non-invasive solution to the VM startup time problem, e.g. for Clojure and Rash.

                            https://www.oilshell.org/blog/2018/12/05.html#toc_2

                            https://github.com/oilshell/shell-protocols/tree/master/coprocess

                            It would make batch invocation of such shells much faster. Basically the idea is to transparently turn a process into a coprocess (single-threaded server).

                            You have a little driver that takes argv, env, and returns a status. It behaves like the original process. But then it uses some file descriptor tricks to proxy the coprocess. So it can be like a “drop-in” replacement for any tool that starts slowly.

                            If you’re interested in prototyping it in Rash or any program, let me know! It still needs implementations to work out the details, although I believe the prototype proves it’s feasible.

                            1. 1

                              Well, at least signals and waitpid do have the beauty that they let you do a lot things with a single thread.

                              My criticism of coprocess launchers is that we know it is possible to implement fast starting VMs (like janet), and adding things to the tower slows them down in the long term.

                    2. 3

                      I’d definitely like to see a shell that allowed standard functional programming idioms. This looks neat and I’m interested in contributing code or other help if possible.

                      1. 3

                        My suggestion to getting started is to download + install janet , and get a bit familiar with the language and how to build + run it. Once you are familiar with janet and how it works, janetsh is a small C module on top of that. Anyway, I think the official chat for this project will be the gitter chatroom I linked in the readme.

                        Currently the janetsh todo’s include:

                        • Tab completion.
                        • Whatever is in the github issues (though they are disorganized).
                        • Finding bugs in what is there already there.
                        • Working out how to make a nice test suite for all the odd shell conditions we are likely to encounter.
                      2. 2

                        That’s impressively small for how usable it looks already. Nice work!

                        1. 3

                          Thank you - It is my hope it will stay small, yet powerful.

                          Much credit to Calvin Rose for creating Janet which is an amazing project.

                        2. 2

                          Nice! I’m a fan.

                          One comment on the presentation of the README: I have to sit and wait to watch you type to see something that could just be a code block. And then it disappears before I have a chance to take it in.

                          I see that the script to generate them is right there as an example, which is neat, but I’m discouraged from even getting that far. (Turbolinks on GitHub mean that navigating to another page and back again doesn’t reset the gif state either, which is a tad weird.)

                          1. 3

                            Hmm, okay, I need to work on that presentation. My general thought of the gif’s was that it highlights that is an interactive shell - Maybe it is too overwhelming. Thank you for the feedback.

                            1. 1

                              I’d like to re-emphasise how much I appreciate the work you’ve done here! My comment does not mean to detract from that, and I’m going to try janetsh out next time I’m spending time at a shell!

                              1. 2

                                No problem, it is good feedback. It is quite tricky to install currently, in the next week or so I think things will become better. It really depends on how intrepid you are :), I just was eager to show people because I was having lots of fun with it.

                            2. 1

                              Can I ask some help when it’s warm during summer? :trollface: (sorry, I had to do this joke :D)

                              1. 1

                                Sure! :D (I’m reminded of one of the early memes.)

                            3. 1

                              I may have missed it in the readme, or because I didn’t read the Janet docs, but if I can just ask to clarify: where’s the “imperative” part of the syntax documented/specified? I.e. the ls -la | head -n 3, and the echo foo > /dev/null, and the rm ./demos/*.gif, etc.? Are those part of Janet, or are they introduced by Janetsh somewhere?

                              1. 1

                                Running commands is a janet macro (language extension) implemented in janet by the janetsh project. Janet itself supports imperative programming because it can do things like:

                                (var x 1)
                                (++ x)
                                (set x @{:hello "earth"})
                                (put x :hello "mars")
                                

                                Which is editing variables in place.

                                1. 1

                                  Hm, I’m afraid still don’t grasp enough to understand the full context :/ Wouldn’t it be too much if I asked you to try and expand how the ls -la | head -n 3 and the rm ./demos/*.gif are evaluated, step by step? (Hopefully with names of macros/functions that perform the subsequent steps?) Starting from the fact they have no parentheses…

                                  1. 3

                                    In the janetsh interactive repl, a line:

                                    ls -la | head  -n 3
                                    

                                    when not beginning with parens when we are not nested implicitly gets converted to ->

                                    (sh/$ ls -la | head -n 3)
                                    

                                    We can see what it expands to with macex

                                    (macex '(sh/$ ls -la | head -n 3))
                                    -->
                                    (do (do (def j @{:procs @[@{:args @[@["ls"] @["-la"]] :stopped false :redirs @[]} @{:args @[@["head"] @["-n"] "3"] :stopped false :redirs @[]}]}) (<function launch-job> j true) (if true (do (def rc (<function job-exit-code> j)) (if (not= 0 rc) (do (error rc)))) j)))
                                    

                                    This code then launches a job using the sh/launch-job function, which inserts it into the global job table.

                                    If you ignore some of the gross redundancy in the macro output (which I should fix), I hope it is more clear how it works now.

                              2. -1

                                Very cool! Although I admit I wish it were written in Janet or Rust or Go instead of C. Any reason C was chosen instead of one of those?

                                1. 6

                                  It is mostly written in janet, only C at the lowest level where janet cannot be used.

                                  Rust or go are possible, but it doesn’t make sense to add them when all the C is doing is providing the minimal layer of unsafe system calls and interfacing with the janet API. The rust added would have just been unsafe and very unidiomatic anyway. The janet implementation is already in C, so it makes sense to keep the dependency tree small and use janet as the implementation language.

                                  1. 3

                                    That makes sense, thanks! It’s interesting that Github doesn’t understand Janet yet as a source language, so it doesn’t appear in the stats.