1. 14
  1. 7

    An alternative is to use a general-purpose/language-agnostic package manager to (also) package shell libraries/executables. One big advantage of this is that your shell packages can depend on packages written in other languages (or, you can have packages where some tools are written in shell). We’ve used this approach in build2 and are quite happy with it: https://build2.org/build2/doc/build2-build-system-manual.xhtml#module-bash

    1. 2

      An alternative is to use a general-purpose/language-agnostic package manager to (also) package shell libraries/executables

      Language-specific package managers are one of my pet hates. They’re predicated on the assumption that a program is written in a single language. For something like Java or .NET, this may be close to the truth (programs may use libraries that aren’t native to the VM but doing so is generally discouraged) but in the general case it’s not true.

      For the shell, this is by definition not true. The entire purpose of a shell is to script programs that exist outside of the shell. Any shell script that is not a pure function will have dependencies (possibly indirect dependencies) on programs that are not part of the shell and therefore can’t be part of a shell-only packaging system.

      The thing that you really want for this use is not so much a packaging system as a portable driver for the platform’s native packaging system. There are two difficult parts here. The first is one of the core difficult problems in computer science: naming things. If my script depends on GNU grep, what is the platform’s package that contains it? The second is what to do if the platform doesn’t provide a package? Do you compile it from source and install it in a private prefix? Do you just give up? Option 2 is much easier and is probably the right place to start, but you might also want to be able to do things like provide third-party repos for your packages and have these coexist with the platform’s native ones. There are some annoying corner cases here. What do you do if the program is packaged, but not with the options that you need (e.g. you need a Python or Lua optional module but that isn’t provided)?

      1. 1

        The thing that you really want for this use is not so much a packaging system as a portable driver for the platform’s native packaging system. There are two difficult parts here. The first is one of the core difficult problems in computer science: naming things. If my script depends on GNU grep, what is the platform’s package that contains it? The second is what to do if the platform doesn’t provide a package? Do you compile it from source and install it in a private prefix?

        You can solve both of these problems if you go the other way around: from-source first and from system package manager as an optimization. This way you can capture the package name mapping for various distributions in the source package’s manifest. You can even have a notion of a “stub package” which doesn’t actually have any source and can only be obtained from the system package manager.

        1. 1

          For the shell, this is by definition not true. The entire purpose of a shell is to script programs that exist outside of the shell. Any shell script that is not a pure function will have dependencies (possibly indirect dependencies) on programs that are not part of the shell and therefore can’t be part of a shell-only packaging system.

          Indeed! This is a major motivation. I wanted to be able to split up some things I’ve worked on into smaller layers that I and others can re-use. Without the user having to manually discover and fetch nth-nested dependencies on a version bump.

          … If my script depends on GNU grep, what is the platform’s package that contains it?

          Yes. The step up to a toolchain that will recognize that no package has supplied a grep identifier is a big step up, but it does still lean heavily on the ~packager/user to know where to look. It’s a far-cry from the cross-platform/PM vision, but in Nix’s own corner, I’ve imagined the next few nibbles I might be able to take out of this problem…

          1. At some point the resholve + nix ~packaging process could be integrated with nix-index so that it can suggest packages that can supply missing identifiers. I’m trying to resist folding Nix-only affordances into resholve itself, though, so I’d probably end up with some sort of --missing-identifier-callback <executable>?

          2. It is still very early/crude, but one of resholve’s big improvements this past year is driven by the use of a “lore” directory, and the default/reference lore-provider: binlore.

            Currently there are just two types of lore:

            • a low-resolution indicator of whether an executable might exec user input
            • an exec-wrapper -> wrapped mapping

            (If it’s maintainable/sustainable…) I think there will be some more types eventually. I’ve been chewing on what it would take to be able to notice when a flag and the provided executable are incompatible, for example, which might lean on a lore type that collects arg/flag/opt information.

        2. 1

          Thanks for the link. I don’t have time to read over it just now, but I’ve got it open.

          I think I’ve under-communicated, maybe by choosing terms that aren’t landing? (I got a similar response on the NixOS discourse…)

          I just pushed an attempt to clarify; curious what you think.

          The last paragraph of the intro read:

          By comprehensive package manager for shell, I mean roughly: a package manager that can supply Shell scripts and all of the dependencies (other Shell libraries, executables, and so on) needed to just run them. It should not fall prey to the dependency-management problems I mentioned in part 1: it should automatically install the script’s dependencies without polluting the user/system environment (and causing related dependency conflicts).

          I do intend “comprehensive” to mean it’s a general-purpose package manager (but, only ones fit for the purpose of packaging Shell). I don’t think a Shell-/Bash-only PM can cut the mustard.

          Do you think changing “a package manager that can…” to “a general-purpose package manager that can…” is enough?

          1. 1

            I personally think it’s strange to approach a general-purpose package manager as shell-first/primarily. But I also think it’s a matter of how you view the solution: as a binary package manager or as a source package manager. If you view it as a binary package manager, then you can probably piggy-back on some existing package manager(s) (e.g., Nix) to provide “comprehensiveness” for you. If you view it a source package manager, then shell-first is definitely feels strange: you would want to start at the base (C/C++) with shell building naturally on that.

            1. 1

              I personally think it’s strange to approach a general-purpose package manager as shell-first/primarily.

              Not sure I follow where “shell-first/primarily” comes in from. Is this just inference from “for Shell”?

              If so, I’m curious how you’d phrase the thing. Maybe a more recent example. How would you describe the distinction (or change) between a (general-purpose) package manager that cannot handle Zig, to one that can?

          2. 1

            Thanks again for the link. I wasn’t familiar with build2, but there’s some synchronicity between how you’re handling bash and what we’re doing with Nix + resholve. I’ll keep the language in mind, too.

          3. 5

            This problem is why we all started using Perl back in the 1990’s. The fact that we’re still having the same discussion, when Perl has come and gone, is kind of mind boggling.

            1. 3

              If you’ve written/commented about this before, I’m curious what you think about the whys, here. (Don’t feel obliged to do so now if you haven’t, though.)

              1. 7

                I haven’t commented on it. There’s a lot to unpack here…

                First, I think the shell was a mistake. It descends directly from JCL on OS/360, which Fred Brooks himself says in retrospect was a mistake. What you want isn’t a separate scheduling language. What you want is scheduling primitives in your programming languages. The Burroughs large machines did this by putting scheduling primitives for the OS into ALGOL W. Cocoa/NeXT have long had APIs for controlling programs via AppleScript, and there was a neat project called F-Script that exposed those to a Smalltalk instead.

                .NET figured out all the interoperability stuff between languages…but then made the same mistake as shell and wrapped a lot of functionality into cmdlets that aren’t available as .NET functions. Shell did the same thing: imagine having sed, find, grep, awk, etc. all available as functions/libraries in your programming languages on Unix. But instead all that useful stuff was tied into shell, and shell was different everywhere, and it was utterly nightmarish. Larry Wall created Perl basically to create a portable version of what people were doing with shell. And Perl was a big improvement, but then we got things like Python and everyone said, “Screw Perl, I never want to write that mess again.” But then everyone scattered off to their own language ecosystems…or back down to shell since all the world had become Linux in the meantime. But Perl and the subsequent language ecosystems still don’t have a lot of stuff that’s in shell (again, sed, find, grep…). And when you write a useful program and ask, “Okay, how do I expose this for people?” if you’re not in .NET or a single language ecosystem, then your only option is basically to make it a shell command.

                Meanwhile if you try to provide a programming language as the primary interface, you quickly hit the same thing that makes notebooks like Jupyter painful to use: as you run commands one after another they change the state of the system, and the only good way to keep track of that is basically not being able to go back. Emulating a teletype turns out to be a pretty good way of avoiding getting tangled up. The only way I see out of that is to switch to a logic based system where you write out a partial state description of a system and then either query for whether it’s true or assert that it should be true. For example, writing (excuse the JSON, or just imagine it’s Cue or something):

                {
                    filesystem: {
                        "/etc/telegraf/telegraf.d/blarg.conf": {
                            content: "# This is a file",
                            owner: "telegraf"
                        }
                    },
                    systemd: {
                        "telegraf.service": {
                            state: "active"
                        }
                    }
                }
                

                I can either query that and be told where it doesn’t match (or put in logic variables to be unified!) or assert that and have the system state change to match it. This is more or less taking ansible or salt to their logical conclusion. You can imagine some kind of dataflow system where you can have live queries that get updated as other ones are asserted.

                So what I would like to see is a component system like .NET (the closest thing outside of Microsoft’s control is OSGi) that lets us all expose code across languages without having to go via stdin/stdout; a unification based approach to querying and setting system state in a more text-editor like environment instead of a teletype emulator; and all those wonderful tools pulled out of shell.

                1. 6

                  Gosh, there’s a lot to unpack in these thoughts. I appreciate you sharing them. I must say though I haven’t got very far through them - I got almost immediately stuck on where you say “shell […] descends directly from JCL on OS/360” :-) Do you feel that is really the case? In many ways the two worlds couldn’t be more far apart, at least from where I’m standing. I wrote JCL day in day out in the late 1980s, and now write Bash shell scripts and work on the command line. I don’t think it’s controversial to say that they are very different, serve different purposes, and come from different ancestral lines. So I’d be genuinely interested if there’s anything that you can point us to that expands on this.

                  FWIW, I’d posit that there is something today that echoes JCL - and that’s the YAML workflow files in GitHub Actions.

                  1. 4

                    Yes, JCL is unlike bash, and neither of them are like VMS’s command shell…but the notion of a separate command language begins with OS/360 as far as I know. The patterns that OS/360 laid down were clearly already well entrenched in the minds of the Multics designers in 1964 when they started.

                    The various ways people extend YAML and JSON into control languages certainly recapitulates the tragedy of shells. Probably as farce.

                  2. 2

                    Thanks–I appreciate this. I’m pulling out a few bits that reminded me of some things, but it’s not a 1=1 with what I found helpful to chew on.

                    instead all that useful stuff was tied into shell, and shell was different everywhere, and it was utterly nightmarish. … or back down to shell since all the world had become Linux in the meantime. But Perl and the subsequent language ecosystems still don’t have a lot of stuff that’s in shell (again, sed, find, grep…).

                    This is very much the kind of thing I had in mind when I wrote the comment that I quoted at the top of the post. I think/hope the (comprehensive) package manager is a bit of a snowball here…

                    1. It lights a path to developers being able to assert the dependencies they need to run.
                    2. Once the developers of a given project can assert the interpreter and dependencies, it’s easier to choose to get off a lot of their ~portability/compatibility treadmills.
                    3. If they can get to the point where their project executes in a specified target shell with the specified dependencies, I think it gets a lot easier to daydream about (quickly or incrementally) upgrading to more maintainable Shells/languages such as OSH/Oil.
                    4. People who write Shell get to read blog posts about how a Bash/Shell project with name recognition was able to get out of the rat race….

                    Meanwhile if you try to provide a programming language as the primary interface, you quickly hit the same thing that makes notebooks like Jupyter painful to use: as you run commands one after another they change the state of the system, and the only good way to keep track of that is basically not being able to go back. Emulating a teletype turns out to be a pretty good way of avoiding getting tangled up. The only way I see out of that is to switch to a logic based system where you write out a partial state description of a system and then either query for whether it’s true or assert that it should be true. For example, writing (excuse the JSON, or just imagine it’s Cue or something):

                    … I can either query that and be told where it doesn’t match (or put in logic variables to be unified!) or assert that and have the system state change to match it. This is more or less taking ansible or salt to their logical conclusion. You can imagine some kind of dataflow system where you can have live queries that get updated as other ones are asserted.

                    These are a mix of oblique and on-target (sorry–I’m a leaper), but–a couple things I’ve been ruminating for a while:

                    • I wish there was a single-executable ~declarative filesystem DSL. I don’t mean it for quite the same thing you do here–I mostly have in mind unifying filesystem ~CRUD operations for cases like installing, packaging/building, ci, etc. I suspect it could help reclaim a fair bit of execution time in some contexts, and a unified/humane API may help avoid a good number of bugs, errors, and security problems.
                    • Once in a blue moon, I’ll see in the ~Nix ecosystem that someone has gone to all of the trouble to package software with care for reproducibility, just for it to break on some systems because of files in ~/. I’ve thought, with a cross-platform approach, we could be giving the software a ~filesystem container, perhaps UUID-identified, if we came up with a good API for managing it. (I don’t entirely know what I mean, yet–just the sensation of having something fast by the tail.) This might mean some things like:
                      • We could checkpoint it a little more ~semantically (when it’s run, versions change, etc.), and maybe associate the state changes.
                      • We could declaratively manage things like whether each version gets its own state or shares it.
                      • We could declare which ones we care to persist and not.
                      • We can have a toolchain for working with those snapshots.
                    1. 1

                      Yeah, unifying filesystem CRUD operations at a lower level than Ansible or Puppet would be a nice thing. I think writing such an API would be a fascinating exercise, mostly because you have to figure out what kind of structure lets you handle the full weirdness of POSIX file semantics. I would expect to see the same kind of barrier statements and the like you do in CUDA.

                      macOS has been working with per-app sandboxes for a while now and has some experience with that. They basically create a filesystem namespace, mount in what the app should have plus tmps for scratch space it would otherwise share, and turns it loose. Maybe the right answer in Nix is a way of parameterizing apps of some sort over user config that you can “join” after the fact. Set up the system by creating a partially applied function, and then users provide the final argument to get a version of the system that is lots of namespaced chunks with the state added in. Each one just sees a ~ directory of its own.

              2. 5

                I’ve written a lot of bash shell in the last 10+ years, so much so in the last ~2-3 years that my SLOC output is probably not far off from 33% Bash. Any time I’ve felt like I needed a package manager – that copypasting a small utility library was an unacceptable risk – I knew it was time to rewrite in something else. The current object of my shell ire is a CI plugin that’s probably approaching 2,000 SLOC between production and testing code, metastasizing from about 300 SLOC MVP when I picked it up from the previous owner. Major shout to bats for enabling this!

                1. 2

                  This is pretty relatable. My goal isn’t really, “write more Bash!” But, as you note with bats, it is one risk/outcome of better tooling.

                  I’ve been a little shy about potentially breaching etiquette by cross-linking these posts, but the first post in this series addresses this tension more directly:

                2. 3

                  I call it “guix”

                  1. 2

                    Guix isn’t magic, here.

                    Because the problems are intrinsic, there’s no trapdoor that lets Guix and Nix skip grappling with them.

                    Having good tooling for wrapping executable scripts does give them both a leg up over the chaos of installing every dependency on the global system/user PATH–but this approach doesn’t work with Shell libraries (and it doesn’t help you find dependencies you haven’t noticed).

                    1. 2

                      Why does it not work for shell libraries? I would set up the script to source the library from the store and the library gets installed automatically when the script is installed. Yeah?

                      1. 1

                        Imagine you have an executable script, foo, which uses openvpn, and a shell library, bar, which also uses openvpn.

                        When you go to invoke foo, one of a few things usually has to be true:

                        1. openvpn must be pre-installed to some location on the user/system PATH before you invoke foo.
                        2. foo invokes openvpn by absolute path
                        3. foo is actually an exec wrapper that sets a different PATH containing openvpn before invoking the true .foo-unwrapped

                        The latter of these is the “easy” way for package managers like Nix and Guix supply the dependencies without having to leak all of them into the global system/user environment.

                        The same practice is a footgun with the bar library. You can technically write a source wrapper around it, and set the PATH in that wrapper, but it’s going to leak the manipulated PATH into the sourcing script (and all of the other modules that it sources). This can:

                        1. Leave the script vulnerable to all of the ~global-state conflicts/fragility that we use Nix/Guix to avoid in the first place.
                        2. Leave unspecified-dependency time-bombs in your script, since it may only be running correctly due to sub-dependencies added to the PATH by a source wrapper.
                        1. 2

                          The right way to do this with guix is option 2, absolute paths.

                          1. 2

                            Yes, absolute paths are the right way to do this (particularly for Nix/Guix).

                            How are you going to put them there?

                            1. 2

                              Here’s an example of me doing it:

                              https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/dns.scm#n114

                              You’ll note this package also uses a PATH wrapper, but only for finding grep and curl, which was enough easier at the time that I decided not to go for absolute paths to those binaries, but I could have of course using a similar patching strategy.

                              Of course if helps if the scripts are written with this in mind, and I have several internal scripts also packaged into internal guix packages that are and have everything as an absolute path. But you can always do it to stuff in the wild too, just a bit more work.

                              1. 4

                                Exactly so! This has historically been the main option (alongside wrapping) in nixpkgs as well.

                                It’s ~tolerable for short scripts, or for scripts you wrote yourself, but it comes up short when you need to package large/complex projects you didn’t write. (You have to know the dependencies are there before you can substitute them, ensure you aren’t also overwriting aliases or functions, etc.)

                                I stripped a lot of detail about what I’m doing with resholve + Nix out of the “Towards comprehensively-packaged Shell” section of the linked post to avoid distracting from the underlying concepts–but if you’re working on this in Guix you might find a deeper dive here interesting:

                                1. 1

                                  Nice! I may compare some of my current efforts with and without resholve. I’m currently working on a similar “linker” for ruby for similar reasons.

                                  1. 1

                                    I’m currently working on a similar “linker” for ruby for similar reasons.

                                    Is this for ~exec within it, nailing down require (both?)?

                                    1. 1

                                      require for now

                  2. 2

                    So what’s different here from resholve? I think resholve is a really great and interesting project, but when I hear “package manager for shell” I get worried as many other commenters here are. Is this just how you conceive of the project you started with resholve?

                    Maybe it’s better to frame this as a comprehensive build system for shell? That seems more like what you’re doing: optimizing away PATH lookups and sources at build time, resolving library lookups statically, etc. And I feel much less worried and much more excited about a build system/compiler for shell than a package manager. That fits in the Nix distinction between build system and package manager, too.

                    You might emphasize also the static, build-time nature of what you’re doing - replacing things which can fail at runtime with statically-resolved stuff. I think the static-ness will very much not be obvious to people at first glance, certainly not if they aren’t already familiar with Nix. So maybe “a comprehensive static linker for shell”? Except I guess that’s just what resholve is… Well, then, if resholve is the linker, then your broader system is the build system, so maybe “build system” is the right phrase…

                    All that being said, what you’re saying here does seem quite interesting, if I can get past the “package manager” part which distracts me.

                    1. 1

                      Thanks for helping pull on this thread. Terminology has been tough, here, from day 1.

                      So what’s different here from resholve? … Maybe it’s better to frame this as a comprehensive build system for shell? That seems more like what you’re doing: optimizing away PATH lookups and sources at build time, resolving library lookups statically, etc. And I feel much less worried and much more excited about a build system/compiler for shell than a package manager … So maybe “a comprehensive static linker for shell”? Except I guess that’s just what resholve is… Well, then, if resholve is the linker, then your broader system is the build system, so maybe “build system” is the right phrase…

                      Not 100% sure I’m interpreting this as you intend, but I think it’s fine to describe resholve as a linker or compiler.

                      But that isn’t really the point. The point is more like:

                      1. General-purpose package managers have ~always been able to “package” Shell, in the sense that they can download and “install” it. But, they are generally not fit for purpose:
                        • Manually identifying all of the dependencies of complex Shell and continuing to do so as it evolves is very hard. Very few Shell packages actually specify all of the dependencies. Anything they miss will sit around waiting to break (perhaps long after deployment).
                        • The PM or user will have to leak identified dependencies onto the global user/system PATH or supply them with wrappers (which ~works for executables but is fraught for libraries).
                      2. A package management ecosystem can use resholve to produce Shell packages that are actually fit for purpose.
                      1. 2

                        I see… so you’re specifically talking about the sense of something being “missing” here.

                        But when I thinking of “the missing package manager” for X language, my primary thought is not “people don’t list the dependencies of X programs”, it’s “there’s no way to install those dependencies of X programs, even if they were listed”. Being able to list dependencies in an unambiguous way is, in some sense, a secondary purpose of package managers - they need to provide that capability for the package manager to work, but what they’re really for is being able to install/deploy software.

                        “The missing Nix/functional package manager for shell scripts” is appealing (to me at least), but “The missing (unspecified, generic) package manager” is less appealing - since most package managers are not like Nix - so my first thought (and the thought of most commentators it seems) is that you’re making another language-specific package manager.

                        And

                        The PM or user will have to leak identified dependencies onto the global user/system PATH or supply them with wrappers (which ~works for executables but is fraught for libraries).

                        is much more of a Nix-y concern - lots of people uninitiated into Nix think this is fine or even desirable.

                        Maybe “the missing build step for shell”? That’s certainly very evocative and interesting to me - it makes me think “what could you possibly do in a build step for shell?” rather than causing me to just retrieve the cached thought “oh no, not another language-specific package manager!”

                        1. 1

                          I see… so you’re specifically talking about the sense of something being “missing” here…. Being able to list dependencies in an unambiguous way is, in some sense, a secondary purpose of package managers - they need to provide that capability for the package manager to work, but what they’re really for is being able to install/deploy software.

                          I’m not quite sure how to negotiate this.

                          I agree, but I think the language plays a role in masking the problem. If bigOlPackageGetter install pytest went and installed pytest without any of its runtime dependencies, it would be misleading to say bigOlPackageGetter can package Python applications. It’s not much better if humans packaging for bigOlPackageGetter “could” get this correct but rarely do.

                          If they’re bad enough at this that the software is usually missing dependencies, I think it helps to recognize that they either aren’t actually installing the software or that they’re externalizing dependency resolution and management on the end users. If they were this bad with all software, we’d just say they are bad package managers.

                          They are roughly this bad with Shell, but I’m being diplomatic (because I don’t see this as their failing) and saying they aren’t comprehensively packaging it.

                          is much more of a Nix-y concern - lots of people uninitiated into Nix think this is fine or even desirable.

                          The last half of this seems about right, but I’m not sure the first is. This is the same fundamental sharp-edge behind the proliferation of *venv/env tools. I don’t think it would be fine or desirable for a package manager to leak implementation-detail-dependencies of a Shell library onto the system/user PATH–let’s say it’s busybox when the user is expecting coreutils.

                          Maybe “the missing build step for shell”? That’s certainly very evocative and interesting to me - it makes me think “what could you possibly do in a build step for shell?” rather than causing me to just retrieve the cached thought “oh no, not another language-specific package manager!”

                          I (feel like) this communicates at a lower level than I’m aiming for–I’m trying to address something (eco-)systemic. Does “the missing build step for Shell” communicate that it’s possible to transition from a status quo in which users/wrappers of Shell executables and libraries may be responsible for identifying and supplying the full recursive dependency graph of everything the shell executes/sources, to one where we can just specify the immediate package/dependency? (The latter is more or less my own cached expectation of a package manager.)

                    2. 2

                      Fonts of site quite unreadable. My eyes!