1. 5

    I’m generally suspicious of DSLs because they throw out all the tools I’m familiar with from programming languages. In this case I think the author took inspiration from the right languages and the result seems reasonable. But being a contrarian I tried thinking of an encoding of the examples in Ruby.

    The path specification looks to me like sequencing some elements so that’s what I’d use for the path selectors

    # /foo/goo/blah/x/y
    path('foo', 'goo', 'blah', 'x', 'y')
    # /foo/*/blah/*/y
    path('foo', :*, 'blah', 'y')
    # /foo/$k/blah/$l/y
    path('foo', :k, 'blah', :l, 'y')
    # /foo/goo[instance='a2']/blah/x[a='S 512']/y
    path('foo', field('goo', 'instance', 'a2'), 'blah', field('x', 'a', 'S 512'), 'y')
    # /foo/goo[instance=*]/blah/x[a=$a_val]/y
    path('foo', field('goo', 'instance', :*), 'blah', field('x', 'a', :a_val), 'y')
    

    But Ruby has operator overloading so we could reduce some of the noise in all of the above examples by overloading the division operator

    'foo' / 'goo' / 'blah' / 'x' / 'y'
    'foo' / :* / 'blah' / 'y'
    'foo' / :k / 'blah' / :l / 'y'
    'foo' / field('goo', 'instance', 'a2') / 'blah' / field('x', 'a', 'S 512') / 'y'
    'foo' / field('goo', 'instance', :*) / 'blah' / field('x', 'a', :a_val) / 'y'
    

    Since we are in a programming language we don’t have to worry about closures because we get them for free because we get blocks, procs, and lambdas for free in Ruby.

    The main use case seems to be filtering, grouping, and transforming the matches and it turns out Ruby (and many other languages) have support for these kinds of constructions with things like iterables, e.g. map, filter, reduce, partition, etc.

    So using the iteration constructs of Ruby we’d get pretty simple equivalent translations

    # select(/foo/$x/$goo/z/y/w, { value($1) < 3 })
    path('foo', :x, :goo, 'z', 'y', 'w').select { |x, goo| x.to_i < 3 }
    
    # group_by($xs, { len($1) })
    xs.group_by { |x| x.length }
    
    # let $even = {$1 % 2 == 0} in filter(range(1, 10), $even)
    (1..10).select { |x| (x % 2).zero? }
    
    # let $xs = range(1, 10) in for_each($xs, { yield('value=', $1) })
    (1..10).map { |x| ['value=', x] }
    

    So I probably wouldn’t have built a DSL now that I’ve gone through this exercise. I would have a built a small library on top of Ruby’s enumerable module and called it a day.

    For folks interested in these kinds of embedded DSLs I recommend looking into “higher order abstract syntax” and “object algebras”.

    1. 2

      I think the point of the author is that because this language will be used for configuration supplied by other users, it shouldn’t be a full language where you can do arbitrary things, but instead be limited to the very things you need to do the configuring. I think the author brought this point very badly across, any language could do what they need, but only few can do it in a side-effect free and total way, fit for using it to run arbitrary code submitted by untrusted users.

      1. 3

        Personally I haven’t met a DSL I like.

        it shouldn’t be a full language where you can do arbitrary things, but instead be limited to the very things you need to do the configuring.

        Translation: I don’t trust or respect my users and want to hamstring them.

        Whenever a program author does that to me…. we’re immediately have a hostile relationship.

        If your user owns the iron the program is running on….

        …give them a full featured, debugged and documented extension language like guile.

        Or if you running something like Ruby, then Ruby is your extension/configuration language.

        fit for using it to run arbitrary code submitted by untrusted users.

        If this is a security sensitive interface… a quick DSL isn’t going to cut it.

        All you’re doing is adding a lexer and a parser and an interpreter to your attack surface. ie. A DSL isn’t a quick way to get security.

        It’s a very hard way to get security, be prepared to do some very hard work.

        1. 1

          Translation: I don’t trust or respect my users and want to hamstring them.

          Translation: my users will come complain to me if it doesn’t work, so I want to make my job easier by making theirs simpler

          1. 1

            Unfortunately this usually is implemented as…

            1. Here is a completely new language you have never met before, enjoy climbing the learning cure.
            2. By yourself.
            3. It’s documentation is half assed and there is no support for it.
            4. Nobody has much experience of it,
            5. googling won’t help, and there aren’t many examples out there.
            6. asking on stack overflow isn’t going to help you either.
            7. Tool support? IDE support? Debuggers? Linters? Syntax highlighting? Nah.
            8. Variables? Adding 1 plus 1? Functions? Decent control structures? Arrays?
            1. 1

              I see the issue. It probably isn’t better when DSLsb like e.g. Jinja has so many ways to do things and Jinja for Ansible can’t be relied upon to be in mainline Jinja.

        2. 2

          But even those requirements are easy to satisfy with Ruby without building anything custom. It’s easy to do a pass through the AST and flag any operators and constructs that are not allowed. Coupled with sandboxing the no I/O or side-effect thing is essentially a solved problem. Elixir even has macros so the AST transforms are basically free if the library is built with macros.

          The overhead of maintaining a custom language is usually not worth it. I’ve worked at places that had their own custom configuration languages and those parts of the codebase are consistently the buggiest and least liked by customers and engineers alike. As soon as the original language author is gone that part of the code is essentially on life support. Whereas a DSL embedded in a real language is less likely to become derelict.

          1. 1

            It’s easy to do a pass through the AST and flag any operators and constructs that are not allowed.

            How easy is this actually though? Can you easily transform Ruby AST’s with some existing library? What do you do for imported libraries, do you recurse on those, or just disallow libraries altogether? If you disallow all libraries, how useful is the language still? What if you missed something in the blacklist (should probably be a whitelist instead)? These aren’t unsolvable problems, but it might be harder than you think (I don’t know Ruby, so maybe there really is something that makes all of this easy).

            The alternative I’m describing in my comment is to use a good enough type system to have such guarantees.

            1. 2

              Ruby comes with Ripper which allows analyzing the AST with very little ceremony. Even if you have a good enough type system you’d still need to perform some kind of analysis because even Haskell has undefined, unsafePerformIO, and exceptions that can sneak past the type system.

              1. 2

                See my comment, I’ve mentioned SafeHaskell, which is built into GHC, disallowing undefined, unsafePerformIO, or anything else that might subvert the type system.

                1. 2

                  It’s better than nothing but both Python and Ruby (and other languages) support sandbox modes that do the same. Also,

                  Now, even if safe Haskell disallowed undefined and error, a function could still fail to return, just by looping endlessly. And an endless loop is bottom too. So safe Haskell guarantees type and memory safety but doesn’t try to guarantee that functions terminate. Safe Haskell is, of course, Turing complete, so it’s not possible in general to prove termination.

                  That’s from SO but the link you posted says the same.

                  1. 2

                    Ah yeah you’re right, Haskell can’t guarantee termination with its type system. But I’ll argue that the type safety is so nice that implementing a timeout for user submitted functions wouldn’t be too much of an effort to solve this.

                    However! There is one language I know of very similar to Haskell that can guarantee termination, Idris! Idris has a totality checker built in and you can force functions to be total, see the Totality Checking section in the Idris docs. Unfortunately Idris doesn’t (yet?) have an equivalent to SafeHaskell, so there’s still some escape hatches you can use, like assert_total, mentioned later down on that very page.

                    1. 1

                      I agree type safety is nice.

                2. 1

                  Except Ripper is written from the perspective of whoever is running Ripper trusts the input it is feeding Ripper.

                  ie. It hasn’t been written and tested from the perspective of “My inputs are carefully crafted ‘with malice aforethought’ to hit every high order algorithm to DoS them, sneak out via every conceivable injection attack, drive every leak until we swap, find every buffer overflow.”

                  Ripper is mostly written in bison/C. ie. Around 18K lines of plain old “my inputs are friendly” C code before you add in the bison autogenerated stuff.

                  ie. You make my point for me. Adding in Ripper into security sensitive code adds in a HUGE attack surface.

        1. 5

          I think a language like Haskell will do: Just let the users write an implementation of a function of a suitable type (that doesn’t allow IO). With Haskell’s type system and Safe Haskell, you can be sure that the user doesn’t do anything nasty. Sprinkle some TemplateHaskell in if you want some syntactic sugar for the paths glob things (or don’t, I’d rather code in normal Haskell than a TH DSL). Then users can use a full, established language with a lot of libraries, a good community and editor support. Maybe it could look something like this

          handle :: Message -> Action
          handle Message { msgPath = "foo" // "goo" // "blah" // Seg "x" (annot "a" -> a_val) // "y" } = Reply "foobar"
          

          This is using the OverloadedStrings and ViewPatterns GHC extensions to make it a bit simpler to write.

          1. 4

            Why doesn’t the first stable release not have a version number 1.0?

            1. 6

              It’s probably a joke. 9 is associated with Cirno and the subtext of “strongest” confirms this.

            1. 19

              A lack of privacy tends to lead to a lack of candidness. If there’s one person in the world I should be able to be candid with, it’s my fiancée, but I wouldn’t be able to be candid with her if I thought our conversations were not private.

              This is an argument that really struck me. I wouldn’t be surprised if just the thought of the possibility of some third-party reading conversations makes me behave differently. If we were to compare it to something in the real world, it’s like trying to have a sensitive conversation in a sparsely-populated public park.

              1. 3

                Exactly. I find it very concerning.

              1. 6

                Is this about packages or NixOS itself? If it’s about packages, I noticed the CHICKEN package is still on 4.13. Our current release is 5.0.0, which is now fully reproducible. If someone were to update it and all eggs, that would raise the percentage a tiny bit again :)

                1. 5

                  Everything is a package, but the packages in iso_minimal.x86_64-linux are “minimal”.

                  I think the packages are limited to the packages listed in the base profile and their dependencies.

                  1. 1

                    Thanks for clarifying!

                  2. 4

                    Duncaen got it right. Here is the full list of what was built: https://r13y.com/reproducibility-log-f2a1a4e93be2d76720a6b96532b5b003cc769312

                    1. 2

                      I merged the chicken 5.0.0 update PR for nixpkgs some time ago, so it should be in unstable :)

                      1. 1

                        Wonderful, thank you!

                    1. 2

                      We need more shots at having a cross-(Linux-)distro (if not crossplatform) package manager since flatpak and snap don’t seem to get much traction. Homebrew might have some advantage by starting with a non-zero marketshare.

                      1. 9

                        Nix is on a good path, it supports all Linux distros and macOS, has really good support for cross compilation, and has a lot of up-to-date packages (source: https://repology.org/).

                        1. 2

                          Nix supports macos and fulfills all criteria, but just has no adoption, so I wouldn’t bet on it. It’s really about size of package repo and afaik homebrew blows all other cross-platform options out of the water there.

                          1. 6

                            According to repology (https://repology.org/repositories/statistics), Homebrew has 4,694 packages (78.6% up to date), and nixpkgs unstable has 49,637 (84.3% up to date). If it is about size of package repo, I hope you take a look!

                            1. 2

                              That seems… off. Debian unstable is ~29,000. Does Nix consider different versions to be different packages or something?

                              1. 6

                                No. We package all of the haskell package set, plus all the emacs modes. I we ignore having packaged some of these collections, we have about 23,000 packages.

                                1. 1

                                  Interesting, is the “normal” stuff in there, like Firefox? I looked at Nix years ago but I found it too complicated to set up compared to just using APT, given that I don’t really care about rolling things back (now that I’m thinking about it, the only time I can ever remember having something broken by a package manager was when I used Arch for awhile).

                                  1. 6

                                    Interesting, is the “normal” stuff in there, like Firefox?

                                    You bet! On Linux, you’ll get firefox no problem: nix-shell -p firefox.

                                    For macOS, Nix is very good for getting dev tools, but I’d still get .app’s from Homebrew or the app store.

                                    1. 3

                                      Could you sum up why I should try using Nix instead of Homebrew on MacOS? I’m genuinly interested in switching over if it offers benefits over Homebrew.

                                      1. 7
                                        • Allows you to use older versions without problems, hell I can easily get an emacs build from 2009! Very useful for older projects or across different machines.
                                        • There’s no problem dealing with many different versions of the same package at once. I could have 10 different packages all having a different version of gcc or clang and I won’t even notice.
                                        • Customize builds easily. Useful for using forks of projects, using alternate dependencies, or changing configure flags.
                                        • Reproducible builds. A nixpkgs revision can be enough to reproduce a build, no more Builds on My Machine (tm).
                                        • Pin exact versions of packages, always creating the same environment. Really useful for development with other people.
                                        • Easily and atomically roll back updates without any traces left behind.
                                        1. 2

                                          Hmm, that sounds good. There isn’t any tradeoff whatsoever?

                                          1. 2
                                            • It is very different from what you’re probably used to, making it harder to learn.
                                            • The documentation could be better.
                                            • The majority of users are on Linux, so the amount of packages that work on macOS isn’t as high, but it’s slowly getting there with more automated building happening.
                        2. 3

                          Why do we need cross-Linux package managers? The package manager is arguably the main thing that differentiates the various Linuces (it certainly isn’t the kernel).

                          1. 1

                            I used snap a little and found it’s nice. Intellij points to it for command line installation in their docs, that’s nice and easy to use, but I don’t like the extra parameter that is usually required for all apps. I wish this was specified in the package installation file instead so end user wouldn’t have to bother.

                          1. 3

                            Stuff I wrote goes in ~/devel. Stuff other people wrote goes in ~/software.

                            1. 2

                              I do the same: ~/projects and ~/src , chosen (with hindsight) to be less and more Unixy respectively.

                              1. 1

                                Pretty much the same for me, except I use “prj” instead of “projects”

                            1. 3

                              There is already https://github.com/defunkt/gist btw

                              1. 4

                                I’m kind of disappointed declarative user profiles didn’t make it into the release. That was the big thing I was looking forward to.

                                1. 2

                                  That would solve a lot of problems and replace ad-hoc tools. I’m glad to see that this is on the roadmap for Nix!

                                  1. 2

                                    A lot of people, including me, are already using home-manager successfully. It has its own set of user modules (out of necessity) and works like NixOS’ configuration.nix. It also has a NixOS module you can use to put all your home-manager configuration directly into configuration.nix and have it apply in a single nixos-rebuild switch. Its state is moderately active, getting a decent amount of PR’s which are merged quickly. I and many others can definitely recommend it to anybody using NixOS.

                                    In comparison, the PR you linked seems more dead than anything. The creators of home-manager (rycee) and NixUP actually intended to work together at some point, but from what I heard from rycee, there wasn’t much communication for a while now.

                                    1. 1

                                      How is it dead? It’s just been awaiting review for a while.

                                      Home manager does seem like a good option, and I’m sure it works, but I’d like to see something that’s built directly into NixOS.

                                      1. 2

                                        No update in 6 months and very sparse updates in general, there have been a ton of merge conflicts for a while now. The author almost hasn’t been active at all the last 12 months. There have been multiple instances of him picking it up again, but then silence again (see the comments on the PR). It’s now the third oldest open PR in nixpkgs. It’s probably one of the biggest changes to nixpkgs too.

                                        The main reason for the success of home-manager is probably just that the author set up a nice repository with a readme and spreading the word. And it’s completely usable on any Linux and even macOS, all in user space, the NixOS integration module is the cherry on the top.

                                        1. 1

                                          I see, that’s a little disappointing. I’ve been following the PR for a long time - while it will get put to good use either way, I even made an additional donation to Nix purely because I was hoping to see some work going towards it. I was hoping to see it merged soon. I guess I’ll have to start taking a better look at home-manager.

                                          Do you know if there is any reason home-manager hasn’t been merged into NixOS? Is it not suitable to be in NixOS, or is the plan to get it in at some point?

                                          1. 2
                                            • Always having to do PR’s to the already PR-overloaded NixOS will probably reduce home-manager’s development speed.
                                            • home-manager modules can’t be shared with NixOS modules as of now, which means there would be lots of duplicated functionality in a single repo, which nixpkgs maintainers wouldn’t like.

                                            These are the reasons I can think of right now.

                                  1. 2

                                    This might be the best git tutorial I have ever seen, well done! While most of these concepts were known to me, this holding hands through the commands really got me more comfortable with them.

                                    I noticed a mistake in a command though, there’s a > too much in echo cloned change rebase >> anewfile.

                                    1. 1

                                      Well I tried this out, and just after a couple minutes with 5 people or so playing it stopped working, reconnecting doesn’t help. Cool concept but it seems a bit buggy!

                                      1. 16

                                        I’m not a devops guy, but I use Nix for this sort of thing in my personal projects.

                                        A less invasive approach might be Dhall, which I’ve not used but AFAIK you can write your config with functions, etc. and run a command to “compile” it into e.g. a JSON file.

                                        1. 7

                                          I can vote for Nix as well, it can do exactly what OP needs. Nix is a lazy, functional language pretty much made for writing configuration. Specifically Nix can solve their problem because:

                                          • It can build output paths, which can for example contain a from Nix generated config, along with the parameters used.
                                          • Fetch Nix expressions from a repository and evaluate them, this let’s you parameterize your repos.
                                          • Nix has lists and attrsets (records), which represent structured data.
                                          • Makes it very easy to deal with environment variables

                                          Edit: As an interesting example, check out my ssh keys NixOS configuration which sets up an nginx virtual host for all my public keys (declared in another Nix file) including an automatically renewing Let’s Encrypt certificate. The function to convert an attrset to a folder structure I wrote is here (should probably make a PR to add it to nixpkgs).

                                          Nix has so much more to it though, there’s NixOS built on Nix with an insanely powerful module system, there’s NixOps for deploying NixOS machines. And most importantly nixpkgs and the Nix package manager which builds packages in a reproducible fashion and enables a lot of stuff you wouldn’t even have thought of. For the interested I can recommend dropping in #nixos on Freenode.

                                          1. 4

                                            thumbs up, dhall looks esoteric, but actually seems like an amazing solution for making statically typed config generators.

                                            1. 2

                                              Both nix and deals look neat, but for ops the last thing we want is more esoteric things. Life is hard enough using normal tools!

                                              1. 3

                                                I don’t think you understand the point of it, sometimes you need to step outside of the normal things to make life simple again.

                                                1. 3

                                                  I’d recommend considering that the reason life is so hard with normal tools is because they’re all trying to solve the same problem in the same fundamental way. Trying new tools that work the same way is going to ultimately end up in the same painful position all the other ones have lead us to. This is why I recommend trying Nix.

                                                  1. 1

                                                    I totally agree…

                                                    I recently shifted my desktop to nix, it is fundamentally different, and solves all these problems better. I was experimenting with broken configs, and could revert the whole OS atomically via grub. That is power I haven’t seen anywhere else, and yet I didn’t even need to know how it works, it was just an option presented to me when I booted.

                                                    My next set of servers is going to be nixops, or something directly inspired by it.

                                                  2. 1

                                                    I think it’s relative: I think if you’re using Chef/Puppet/Ansible/Docker/etc. then Nix is just another alternative, which IMHO is cleaner. It’s got a decent community, commercial users, etc. so if you’re thinking about adopting one of these technologies, Nix is definitely worth a shot and I wouldn’t consider it esoteric at all.

                                                    The problem with Nix for a situation like this is that it’s quite heavyweight; just like switching a server over to Puppet or something is also a heavyweight solution to the problem of “put different values in these config files”. I would say Dhall is certainly esoteric, but I think it’s worth a look for solving this problem since it’s so lightweight: it’s just a standalone command which will spit out JSON. In that sense, I think it’s a more reliable alternative to hand-rolled scripts, which wouldn’t require any extensive changes to implement (call it from bash, like anything else; use the JSON, like anything else).

                                              1. 1

                                                That table mode seems excessively verbose. My own home-brewed example uses tab-delimited fields and a few special markers:

                                                #This is a caption line
                                                *header1 header2 header3
                                                data1 data2 data3
                                                

                                                (only in this example, replace spaces with tabs). It’s easier to create (but admittedly a bit harder to read in the raw format) and easier to parse (since the output in my case will always be HTML).

                                                1. 3

                                                  The thing is, org-mode does all that verbose stuff for you, have a look at this great demo that shows what you can do with tables: https://youtu.be/fTJVLJd_gz0?t=2m45s

                                                1. 2

                                                  While I agree that you can learn a lot of technologies one works with, there is a thing you probably won’t learn like this: New ideas. Which is completely understandable, production environments can’t afford that. Only the very few technologies that survived for a decade or so do eventually get adopted for work (bar some exceptions). I’m talking about things like Idris, Elixir, Nix, Rust, and more, all of which have some great thinking behind them.

                                                  1. 3

                                                    Rust was developed at an organization that wants to use it (Mozilla) and has been adopted by companies that want to use it (e.g. Dropbox). I know companies that have used Nix. Elixir is also used by real companies.

                                                    1. 5

                                                      I’ll “me too” on this one. Where I currently work we have quite a bit on Rust in production. There’s a little bit on the cloud side and a significant amount on our hardware product (embedded linux box).

                                                      The last place where I worked had Elixir in production too. Just a little bit in a mostly erlang codebase.

                                                      1. 0

                                                        While Rust has seen some adoption recently, on a global scale it’s pretty insignificant

                                                    2. 3

                                                      Why couldn’t you learn those things at work?