1. 11

I have developed a programming language specifically designed for Ops. There is a vision for interactive shell and other features to be added to the language but recently I was told that it’s hard to sell vision and one must sell use cases. Working on that:


  1. 4

    I like that you’re building out a language that puts these things front and center, but I wonder if building your own language (aside from it being incredibly fun!) is the most practical way to reach your goal? Contrast this method to say, creating a distribution of Lua that includes well appointed standard library support for common devops tasks, selected integrated primitives from lume or luafun, etc.

    I admit that I don’t have a great solution for a CLI interface, but would point you to the work that’s happening in rash which uses hints to interpret shell commands or otherwise. In the case of a lua based thing like I’m suggesting it might be as simple as cat @filter(function(x) ... end, ls()) where the @ in this case flattens the returned array, and also indicates the next thing is Lua. The command interpreter then actually does something like loadstring("sh_run("/bin/cat", unpack(filter(function(x) .. end, ls()))")

    1. 1

      most practical way to reach your goal

      • I think yes if you don’t want to compromise on syntax.
      • It would also be difficult/awkward to have the types/multimethods system which I think is the best for the Ops tasks in any other language (except maybe Lisp)

      solution for a CLI interface

      I’ve got design for UI/CLI but it’s not implemented yet.



      cat @filter(function(x) … end, ls())

      Somewhat similar. In NGS you can cat ${CODE HERE}) or cat $*{CODE HERE} (for splat). The syntax is consistent with $ substitution in strings and * splat in other places in NGS. And then using NGS types/multimethods system you can customize how to:

      • Convert your type to command line argument
      • How to run particular command (any customization)
      • How to treat exit codes of your particular command (NGS knows several programs out of the box). If the exit code is not logically OK, NGS throws an exception
      • How to parse the output of a particular command (NGS knows about JSON)

      I can’t think about every other detail now but on intuitive level you can’t be 100% on the spot if you don’t build the language ground up with Ops tasks in mind. You can’t be 100% on the spot even if you do but that’s another story :)

    2. 2

      I don’t want to bash† this project because it seems really, really cool. It’s the kind of thing I hope becomes very popular, but I have one pet peeve. I’m not part of the Rust Evangelism Strike Force, but a modern language like this should be written in a modern systems language, in my opinion.

      It could be Rust, it could Go, it could be Swift, it could be C#/F#, heck, even Java can be AOT compiled these days :)

      I’m guessing that the project is reasonably old for something announced recently (I see commits in the Github interface from 2 years ago and if I’m reading the Github commits graph correctly, the first commits seem to be from 2014), plus the author probably already knew C aaaand C is portable so it helps with Ops adoption.

      Anyway, this is a minor point, I hope this gets mass adoption, it’s badly needed. We need a language designed in the 21st century as the default Ops language. Powershell is a bit clunky plus adoption is a bit low so there’s plenty of room in this space.

      Good luck!

      † Pun intended.

      1. 3

        I see what you did there with bash :)

        Don’t have time to elaborate too much but C was chosen

        • as the language that I will be never be stuck with and won’t be able to solve something.
        • I can look how the problems I will be seeing are already solved in bash, Ruby, Python (all in C)

        I also pushed as much as I can into stdlib which is in NGS, hoping that if I need to change the implementation it will not be harder than it needs to be.


        1. 4

          I can tell you rust isn’t so hot for interpreters due to the ownership model being at odds with GC, and Go/C# ain’t so hot for low level things because of the runtime with many background threads.

          1. 1

            the ownership model being at odds with GC

            Could you elaborate more on this? As far as I understand there are tons of ways to move the borrow checking to runtime.

            1. 1

              Then your interpreter takes a large performance hit, which isn’t the greatest thing. I didn’t say it is impossible, just not as great as people claim.

              1. 1

                Why does writing a GC in Rust mean a larger performance hit than doing it in some other language like C?

                1. 1

                  Because you need to use runtime ownership reference counting on every mutable operation with RefCell (I think) .

                  1. 1

                    You don’t have to. You can just use raw pointers the same way you would in C.

                    One could argue that using unsafe operations like that defeats the purpose of using Rust, but I disagree. You can limit these unsafe operations to your gc/allocation code and vet them for safety, while the rest (bulk) can be in safe Rust.

                    Having written GCs in both C and Rust I prefer doing it in Rust. You don’t get much better safety but you can get better ergonomics.

                    1. 1

                      Having written GCs in both C and Rust I prefer doing it in Rust. You don’t get much better safety but you can get better ergonomics.

                      How do you use GC’d types from rust safely? The GC owns the value so they are usually always immutable values in rust gc’s.

                      1. 1

                        Since we’re talking about a scripting language, you would only directly access the GC’d types from Rust in the actual runtime code itself. However you don’t have to expose those values directly to any “user” code. For example, Lua uses a stack for interop instead of giving direct access (and all the mess that entails).

                        In Rust, you could give the user code in those cases something like a &Handle<RealType> that counts as a borrow on the stack and thus prevents destructive operations from invalidating the reference.

                        This gets a lot hairier if your runtime allows concurrent access, but at the end of the day anything you can do in C you can also do in unsafe Rust.

                        If you mean to implement a GC for Rust itself to be used seamlessly with other code, then yeah you run into the issues you’re talking about but that’s not the case here.

                        1. 1

                          I see what you mean, I guess it may be easier to divide the objects into two areas and not use safe rust to access gc objects ever.

                    2. 1

                      runtime ownership reference counting on every mutable operation with RefCell

                      RefCell does not do reference counting; are you thinking of Rc? Rc is reference counting garbage collection - that’s the whole implementation, just use Rc. No extra cost over a reference counting garbage collector implemented in C. If what you want is a tracing GC, many exist: for instance, this one by withoutboats. https://github.com/withoutboats/shifgrethor

                      1. 1

                        RefCell does do reference counting, but not like you think… From the docs of RefCell:

                        A mutable memory location with dynamically checked borrow rules

                        The way they dynamically check borrow rules is by reference counting them…

                        As for your tracing GC reference, you could read the full readme of that GC to find more of it’s limitations.

                        1. 1

                          RefCell does do reference counting, but not like you think… From the docs of RefCell:

                          If you want to be pedantic, then sure, a RefCell does “count references”, but it does not implement a reference-counting GC pattern (the memory is freed only in accordance with normal ownership rules, as can be seen by the simple fact that it does not implement Drop (see src)) and therefore isn’t really relevant here.

                          If you are unhappy with that GC you could take a look at the lustre GC which is used in a Lua VM at the moment, or at Manishearth’s gc and his blog post on it, or at this WIP gc for use with VMs, or at any of the other interesting GC projects written in Rust.

                          Perhaps you are thinking of rcgc, which does use Rc reference counting to implement a garbage collector.

                          I guess I’m just not sure what it is you’re trying to show here. Implementing a GC has a cost, but my point is that there is no additional cost from using Rust. Because you are implementing the component of the system that enforces memory safety, this is the appropriate place for unsafe code and therefore you will not have any mandatory performance penalty from using previously implemented abstractions.

                          1. 1

                            if you want to be pedantic, then sure, a RefCell does “count references”

                            but my point is that there is no additional cost from using Rust.

                            This is a direct contradiction… Let me put it this way, in rust who owns a GC ref? to use a GC ref in a mutable way you need to use something like RefCell which has a runtime cost. You don’t need that in C. That is all that I am saying…

                            1. 1

                              You don’t have to use RefCell, so it doesn’t matter what RefCell costs. If you were to implement a RefCell-like counter for mutable vs immutable references, you couldn’t do much better than RefCell, but if you don’t need that functionality you can just forgo it and use raw pointers, just like in C.

            2. 2

              What would be your ideas for modern ops language? Here are mine:

              • Steal the sh pipe syntax (Python, Ruby, TCL, Lua fail here)
              • More data types than string (bash, zsh fail here)

              Powershell would qualify. Is there anything really wrong with it except it looks verbose?

              1. 2

                Powershell would mostly qualify but it needs higher adoption and in my opinion it needs to be a truly Open Source project, preferably a foundation. Microsoft might decide to add stuff that is detrimental to it, overall, just to further business interests.

                It also needs to start popping up in distribution repos. You should be able to just apt install pwsh and yum install pwsh, from the default repos of most Linux distributions.

            3. 1

              https://github.com/ngs-lang/ngs is probably what we want the link to be.

              1. 2

                I see. Or maybe the title something like “Rate my language’s use cases page”?

              2. 1

                As a language to design a script looks pretty good. Did not see example of interactive use.

                1. 1

                  UI is not implemented yet. I’ve got quite a few ideas though.


                  1. 1

                    There is no cli design?

                    1. 1


                      Most points in “UI Software Design” apply to both CLI and Web UI.

                      Quoting from Requirements > General:

                      • Ability to support several i/o modules
                        • Console module
                          • LayoutManager to handle widgets layout on the screen (the counterpart of web layout engine in the browser)
                            • Dynamic width support for Table and probably other objects on the screen
                          • Use last line for status() as in apt/apt-pkg/install-progress.cc
                        • Later: web module
                2. 1

                  One thing that hamstrings all progress on the command line / shell front is everything is stringly typed.

                  If you invoke a process, what information does POSIX allow to be passed to the subprocess?

                  And array of strings in ARGV and globals in ENV

                  What is the format of the strings?

                  Undefined. Whatever the subprocess thinks.

                  What does the subprocess expect?

                  At best informally specified via a help…. although a hackish kludgish work around is being attempted in bash through completion functions.

                  I’d love the interface definition language between processes to be at least as rich as the domain (usual JSONish PoD’s and shell relevant files, directories, … etc.)

                  I’d love it if processes could announce their interface.

                  Dbus takes us some small step towards these goals… but leaps sideways into XML and inter user security bounding stuff.

                  Legacy stuff could be wrapped in a something that knows it’s interface and translates down to stringly typed options and args.

                  1. 1

                    Yep. That place where simplicity of concept and coding (argv strings) puts the burden on all future users. Could you maybe add a wiki page on GitHub? There are quite a few pages for “design” topics (to get the feeling). The “requirements” section is a great start.

                  2. 0

                    What does it have that PowerShell does not ?

                    This started 5 years ago, and Powershell was open sourced 3 years ago. I understand that during propriatery era of posh this made a lot of sense but not so sure now. Powershell is using .NET framework so you can literally create anything in it that is possible with C# - that is the feature level that is very hard to approach to by any foss tool that isn’t backed up by huge company.

                    One potential problem with posh is that it is far from fast shell. Not sure how relevant this is in majority of use cases.

                    Looking trough use cases I literally use everything there in posh today:

                    • Testing shell/cli/whatever with Pester (I even test WebApi with it), cross platform
                    • Build system with Invoke-Build, cross platform. Easy and powerful.
                    • Data manipulation is native in posh - ConvertTo/From - CSV/Json/XML/Yaml/HTML is there OTB. There are bunch of other functions too and even ConvertFrom-String which is total academic mumbo jubmo.
                    • Filtering is there OTB with select-object, where-object etc.
                    • Running processes in number of ways - with redirection. Even creating entire process inside of the script, compile it and run it.
                    • Parallel connects to servers and execute scripts….

                    And then there are bunch of things that aren’t in uses cases here: workflows, custom drives (anything as file system), universal package management, desired configuration, etc

                    Its very hard to get all that from 0.

                    Saying all that, I have high hopes on Guile and Guix - its backed up by GNU foundation and since its Scheme impl. it is even more powerful then posh … or it can be eventually. Adoption will probably be lower due to the language having strong math background.

                    1. 1

                      I’m onboard with all your points but my good is powershell far from consice . I know that it comes from having a systematically and well built up object model that is it’s strength. As you write Net in essence.

                      1. 0

                        It also comes with majority of Unix aliases as the original team came with that background.

                        This problem came into existence by force buy the NO ALIAS crew which includes MS and all MVPs. And its total BS. I fought their invalid arguments time and time again without any luck. I use default aliases majority of time so my code is as concise as it could be.

                        They think that Get-ChildItem is more readable then ls or Get-Process more readable then ps. They think that Get-ChildItem | Where-Object {} is more readable then ls | ? {}. Well maybe it is for n00bs but everybody else knows what ls/ps is since kindergarden and to n00bs themself any variant is equally new.

                        The only reason for not using aliases might be the pwsh (i.e. cross platform scripts) since posh 6 changed many of them (for the first time since first version of posh). For example ls now calls native thing instead of oo powershell thing.

                        1. 1

                          I just played around with Powershell a little and I’m not sure about this alias-or-not styling. I doubt that Powershell-ls is not exactly POSIX-ls or GNU-ls, so can’t you get confused there? I get confused between bash-builtin and coreutils sometimes.

                          1. 2

                            Alias confusion is not specific to posh. Yeah, posh-ls is nothing like posix-ls or gnu-ls but that is the whole point - we don’t need multiple ls’s and if we sometimes do, there are better ways to introduce them without breaking compatibility with existing variants (see proxy objects). Also, this is not specific to aliases - what happens if you have the same executable on path on different locations? Posh has good way to specify so called fully quolified cmdlet name which is then guarantied to call what you want on any system. This is better then for example knowing the exact path of executable which might differ from system to system.

                            Anyway, I gave on SO shortest version of your experiment. Its shorter then bash variant and way more readable.

                          2. 1

                            Maybe it’s a generational thing, those who learnet coding with autocomplete and those by reading manuals / reading source code. Then again I wouldn’t say that the powershell in itself doesn’t have that cool an autocomplete.

                            But I think that the argument for shorter names could be made! Less text would make the code quicker to read.

                            While coding myself I tend to prefer shorter code. So mv instead of move pv instead of player_velocity p0 instead of initial_position and so forth. To say that this would be worse is quite silly just take a look at all math or physics that ever has been written. What I’d really would like to see in code however is more usage of UTF-8 / math symbols.

                            1. 1

                              To say that this would be worse is quite silly just take a look at all math or physics that ever has been written

                              Indeed. Plus, its not only reading less, but also scrolling less, visually keeping related concepts closer together, easier way to combine expressions etc.

                              To me, all that looks like millennial/politically-correct shenanigans and from the rest of us it is required to take on a babysitting.

                              1. 1

                                Exactly my thoughts,couldn’t have said it better myself

                        2. 1

                          I know a bit more now about PowerShell so I need to update the comparison section but meanwhile:


                          1. 1

                            Nice comparison. Maybe its better to put in in a wiki so other people can easier contribute and reference.

                          2. -1

                            This appears to be cross-platform, while powershell is not

                            Edit: Apparently powershell runs on more than just windows. Oops.

                            1. -5

                              PowerShell is x-platform (i.e. in what cave did you live past 3 years?). Toast is done in C but that doesn’t magically make it x-platform and, certainly, there are no windows binaries or instructions of any kind (nor it is tested on windoze).

                              1. 4

                                i.e. in what cave did you live past 3 years

                                Apparently a cave in which I don’t pay attention to all the crap microsoft throws over the wall

                                Also, no need to be an ass here. This isn’t ‘hacker’ ‘news’, reddit, or twitter. A simple ‘actually it is cross platform, see here’ would have been sufficient.

                                Toast is done in C but that doesn’t magically make it x-platform

                                The readme in the repo shows build instructions/tools for multiple platforms. It’s cross-platform.

                                1. -9

                                  Apparently a cave in which I don’t pay attention to all the crap microsoft throws over the wall

                                  Still a cave :) And please, do not be melodramatic. It was a test, to see if guys here can tolerate something a 13 year old girls probably would or we can’t be honest here.

                                  Actually, such an attitude is already known and well understood. Honestly, “the cave” expresses the best what I had in mind, if you have trouble with that, probably grow up? Or exit the cave? Or LMGT4U. I do not use bunch of things (Oracle/SAP/OSX/whatever) but still follow the crappy tech, if anything, then to know its crappy.

                                  1. 5

                                    It was a test

                                    Don’t “test” people, that’s just called “being a dick and then not taking responsibility for it”.

                                    1. -1

                                      Don’t be a dick about it, I take a full responsibility! We all have our caves.

                                    2. 4

                                      It was a test, to see if guys here can tolerate something a 13 year old girls probably would or we can’t be honest here.

                                      I don’t think I want to visit the cave where the only way you can be honest with another human is by first insulting them. There are other platforms on the internet which are better suited for this method of ‘discussion’.

                                      1. 0

                                        It was an honest thing to do :). I wish people do it to me more often for my caves.

                                        It’s also more like adjective, not a discussion. In discussion you give arguments (which I did, I wouldn’t put x-platform multiple times in comment above if it wasn’t; I could equally argue that replied comment was insulting because it implies I am sloppy and my words can not be trusted).

                                        Lets end this with typical finalizer: lets agree to disagree ;)