1. 16
  1.  

  2. 5

    A few more things on the subject:

    When secrets are passed as argument they are discoverable by any process on the machine. Instead, it should be good form to only accept paths to secrets as arguments and rely on the POSIX filesystem to enforce ACL.

    The flag parser should convert empty values to defaults if there is one. For example if the default user is root and $USER is empty, -user "$USER" would default as root. This is necessary to avoid having all the users of the CLI find out about the default and re-expand it in their script with "${USER:-root}".

    For more complex cases where a configuration file starts being needed, I have seen a few using the command-line args as the configuration format. This has the advantage to enforce that all configuration options will also be overridable with the command-line.

    1. 1

      When secrets are passed as argument they are discoverable by any process on the machine.

      I think for the most part this is a theoretical problem. It may have been true when shared machines were more common (especially on badly configured ones which allow ps to show processes from other users), but even with VPSs you typically have one user running one program. With containers even more so. If someone has sufficient access to access this kind of information then chances are they can access things like a path with secrets or /proc/self/env, too.

      At any rate, last time I had this conversation I wasn’t able to find any examples of actual breaches happening because of secrets being read from the process information, and neither was my co-worker, so I don’t think this is something to overly worry about.

      The flag parser should convert empty values to defaults if there is one. For example if the default user is root and $USER is empty, -user “$USER” would default as root. This is necessary to avoid having all the users of the CLI find out about the default and re-expand it in their script with “${USER:-root}”.

      I guess that depends; sometimes you want to pass an empty string. In addition, if you don’t want to pass a value then, well, don’t pass a value. Preëmptively adding -option "$OPTION" to every flag kind of defeats the point of explicit overrides from the environment. While there are probably some use cases for -flag “${FLAG:-def}”`, I think that in general is should probably be used sparingly.

      1. 4

        I agree with you that the first one is partially a /proc problem, PID hiding is the standard on any hardened configuration and always a pain point in lots of Linux hosts. If you look at some of Grsecurity’s configuration options this can be accounted for.

        That being said, I totally disagree with your evaluation of breaches and the use of /proc. I have actively used /proc abuses to compromise AWS keys about 4 times in the last year, and actively use race conditions abusing /proc to find calls from sudo/su to hijack files that are user controlled for privilege escalation. Here is an example of one way to do that, it uses ps internally to read /proc but achieves the same thing.

        1. 1

          But you already had access to the system, no? That Python script certainly requires access to run that script. If you have that kind of access then there are many things you can do.

          That script of yours seems to replace the content of myscript when someone runs sudo ./myscript? If you have that kind of access then you’re pwned anyway. Your script seems like a fancy wrapper around find . -type f -a -executable and/or grep sudo .history and then injecting that exploit in the scripts you find. Hell, you could make a sudo alias in their shell config? Either I’m not fully understanding how that script works, or it’s a bit less dramatic than it may seem at first glance.

          If you look at some of Grsecurity’s configuration options this can be accounted for.

          you don’t need Grsecurity: there’s the hidepid mount option. And on BSD systems there’s a sysctl to do the same.

          1. 2

            The Python script was just an example of how you could use it, there are literally infinite ways to abuse that. But when talking about the “risk” being that /proc will store secrets it’s generally assumed that your threat model is a local user file read access no?

            Just to clarify that, you can just use some sort of read primitive for reading from /proc, but you don’t actually have shell. In the cases of AWS key hijacking I actually just needed a SSRF with support for file:// which would allow me to read the contents of /proc/$PID/cmdline (In this case I had to bruteforce the pid).

            You also have to remember that often times payloads may not be running in the context of a user with login and full shell access, ie the www user from a running web service.

            1. 1

              I actually just needed a SSRF with support for file:// which would allow me to read the contents of /proc/$PID/cmdline

              Right; that is probably a better example of your point than that Python script.

              How would you protect against this, beyond hidepid=1? The most common method is to use environment variables, but you can read /proc/self/environ as well, so this doesn’t seem to offer a lot of additional protection against some sort of exploit that allows filesystem access from the app.

              The sugegstion from the original commenter who started the thread (“good form to only accept paths to secrets as arguments and rely on the POSIX filesystem to enforce ACL”) doesn’t strike me as helping here, either.

              1. 1

                Yeah I wasn’t super clear about that so sorry for the confusion, I really should draft up some quick fire examples for these.

                The difficulty of protection on mainline Linux is part of the reason I think a lot of people advocate for not doing flag based secrets, but like you point out there are also environment flags! As far as I know there is no “baked” way of properly protecting proc from the users own processes. The last time we discussed some work arounds for this we actually set hidepid=2 which is more restrictive and then launched the process in a privilege seperation model (ie like OpenSSH) that way the config was applied at the supervisor or through a broker.

                Frankly, I think that’s crap. I think the better way to deal with this is RBAC through SELinux, Grsecurity, and friends. But, that can be a bit too much of an ask for most people as the actual support for SELinux is only in a few distros and Grsecurity is no longer easy to obtain.

                1. 1

                  A simple chroot() should be enough; you don’t need OpenSSH-style privsep as for most applications as there’s no part that needs privileged access after starting. That may not be easy for all applications though, especially things like Ruby on Rails and such. It’s probably easier to add for things like a statically compiled Go binary. You can perhaps also unmount procfs in a container, I never tried.

                  I think stuff like SELinux is very powerful, but also very complex and hard to get right. In practice, most people don’t seem to bother, and even if you do it’s easy to make I mistake. Every time I’ve used it I always found it very opaque and was never quite sure if I had covered everything. I think something like unveil() from OpenBSD is a much better approach (…but it doesn’t exist in Linux, yet anyway).

                  1. 1

                    Generally I don’t like to consider chroot(8) and namespaces(7) security protections, but in this specific case I think that they would work pretty well to prevent access and really is what I should have been thinking of.

                    The reason I pointed out RBAC systems was because I have managed both a pretty large scale SELinux and Grsecurity based RBAC deployment, and you are 100% right about SELinux. It is the biggest pain. Grsecurity RBAC is actually one that I hope more people go back and play with for inspiration, it is trivial to set up and use and even has a learning system that can watch what a system is doing. I used to build Grsecurity profiles as part of testing by running all tests in monitor mode and automatically applying the profiles at deploy time and if possible applying gran to use model checking, and very very rarely ran into issues. But, yes they are not “simple” off the bat.

                    I was sort of staying in Linux land, but I think a better way to handle things is unveil() in most cases, I just don’t know of a way to replicate that in Linux without some sort of RBAC.

                    1. 1

                      So the more I think about it, the more it seems that this is a much harder problem then I thought.

                      For now I added a note about hidepid=1 and that securing secrets is complex. I can’t really find a good/comprehensive article on the topic.

                      One method that seems reasonably secure without too much complexity is to read the values you want and then modify the cmdline and/or environment to remove sensitive information. That’s not 100% secure, but at least it prevents leaking secrets with stuff like pwned.php?file=../../../../../proc/self/environ and is probably “secure enough” to protect against most webapp vulnerability escalations.

                      There are also some dedicated apps/solutions for this (like Hasicorp’s Vault). Not sure what to think of that.

                      1. 1

                        Imagine if you launch a container, as many web apps do these days, then you’d read a secret file into the application and then promptly delete the file. If anyone found an exploit in the application in regards to the file system, the secret would be gone already.

                        If the container is restarted, the file would be accessible again as it is recreated from the container image. You’d probably want to not listen for any new connections before the secret initialization has completed.

                        Would this be a good solution or would it introduce other problems?

      2. 1

        Secrets are the one thing I prefer to pass via Environment Variables or Config files. Everything else I prefer to use flags for pretty much the same reason I don’t want to use them for environment variables.

      3. 3

        I’m a big fan of this style but with a couple of additions. In particular, I like to have variables that, when set, pass their values to the flags, and when not set, just don’t exist in the command line arguments. An example:

        #!/bin/bash
        # File: foo.sh
        
        type=${type:-f}
        dir=${1?:need a directory}
        
        find "$dir" ${type:+-type "$type"} "$@"
        
        # Usage:
        #   ./foo.sh mydir  # find files in mydir
        #   type=d ./foo.sh mydir # find directories in mydir
        #   ./foo.sh mydir -exec wc -l +  # count lines in files in mydir
        #   ./foo.sh  # error: need a directory
        

        The trick is using ${variable:+--flag "$variable"} which, if $variable is unset, expands to nothing, but if set, it expands --flag "$variable". Another example for docker is:

        #!/bin/bash
        # File: docker.sh
        
        data=/mnt/data
        
        docker run -it --rm ${data:+-v "$data:$data"} "$@"
        
        # Usage: ./docker.sh hello-world
        

        This will run whatever container interactively and with the /mnt/data directory (but if you edit the file to remove the data= then it will not mount it).

        (I talked about the Docker approach more at https://mediocreatbest.xyz/1/ if you’re interested)

        1. 3

          This is a neat trick; I didn’t know about :+ substitution. It’s even a POSIX feature! I updated the article to include it. Thanks!

        2. 3

          I totally agree, but mostly for a reason not listed in the article: if flags are the primary way to configure your program, then program -h is an authoritative and easy-to-use way to review the complete configuration “surface area” available to users. Config files and env vars both lack this automatic discoverability.

          After many years of futzing about and trying to find the best approach, I’ve written a configuration parsing package for Go which IMO captures the best of all worlds. peterbourgon/ff (ff for flags-first) hooks onto a stdlib flags.FlagSet, but allows you to optionally parse each flag from a config file or an env var, too, in that priority order. (Explicit flags always take precedence.) So you keep the discoverability, without giving up the usability of other options in situations where they make sense.

          1. 2

            Yeah, documentation is another good reason. I thought I had mentioned that but seems it got lost in rewrite. I’ll add it, thanks!

            I actually wrote another post about using flags in Go specifically: https://arp242.net/weblog/flags-config-go.html, which is a much simpler approach than your package (this post got split from that, as I figured much of it wasn’t Go-specific).

            I’ve worked on “commandline trumps config file trumps environment” apps, and I’m not sure if I’m a fan. It can be hard to see where a variable is set (is it config? Is it environment?). And what if I set ADRESS instead of ADDRESS, do I get an error or that or will it silently use the wrong value? (In my native Dutch address is spelled as “adres”; typos like that can be hard to spot as my brain has a tendency to say “yup, looks okay”).

            Certainly at my last job there was a lot of confusion with a similar scheme as soon as people had to configure an application on their dev environment (many of them were not gophers).

            1. 1

              I actually wrote another post about using flags in Go specifically: https://arp242.net/weblog/flags-config-go.html, which is a much simpler approach than your package

              It’s… different, I don’t think it’s simpler. It also violates a couple of Go idioms, by defining flags outside of func main, by using package global state, and by calling a package “cfg”.

          2. 2

            That’s a fun subject to think about, and I enjoyed reading your thoughts. When thinking about configuration, I find it useful to consider how often it varies: (a) per invocation, (b) per use case, or (c) per user? (There are other levels such as ‘per site’ or ‘at runtime’, but these will do for now.)

            To set config per invocation (a), parsing flags and arguments is fine, and I agree with the author that they’re somewhat nicer than environment variables.

            To set config per user (c), I really like config files. I can slowly grow my pool of personal tweaks in one place, add comments, keep them in my dotfiles, and share them with others. Some config files are set-and-forget, like for elinks or Readline. The config files I interact with most are Vim, Mercurial, and my shell, all programs that allow extending their vocabulary and defining new commands.

            To set config per use case (b), retyping the same flags over and over can get tedious even if they’re short, such as the -Tpng to make graphviz emit a PNG file. Config files are obviously out because then you have to repeatedly specify which config file. Here, the ideal is a program that supports flags, and a wrapper script for each use case. For example, I have a graphviz wrapper script dot2png that ultimately comes down to dot my_input -Tpng -o my_output.png. I’d hate for that script to need to edit a config file on the fly.

            1. 2

              I agree with you in spirit but in practice I find this logic to be questionable.

              Why? Because humans make mistakes with command line invocations, and every decision point that can be removed from the tired human running ./save_or_destroy_the_world.py at 3 AM when their pager goes off is priceless.

              1. 4

                I think this can be solved by creating save-world.sh and destroy-world.sh scripts which runs the program with the desired arguments?

                1. 2

                  That’s a very good point. Make each CLI program do one thing and one thing only and make the fact that you’re doing something destructive super obvious and easy to avoid.

                  1. 2

                    I didn’t state this explicitly, and I should probably add it to the post, but I would always create wrapper scripts. For example for the program I was working on this afternoon has etc/run with:

                    #!/bin/sh
                    exec ./example -prod \
                        -domain "example.net" \
                        -domainstatic "static.example.net" \
                        -smtp "localhost:25"
                    

                    There are other ways, too (runit script, systemd, etc.) Even for every-day stuff like this I certainly wouldn’t rely on remembering to type -prod or -smtp. Forgetting it would mean showing errors to users, or not sending out emails!

                    1. 3

                      Those are handy. However it strikes me reading this that it’s configuration in sheep’s clothing :)

                      1. 4

                        That is correct. I am not against the concept of a configuration file. I just think that you often don’t need complex config parsing libraries (like YAML) when a few simple flags with a script will do just as well.

                        As I wrote in the article:

                        I see plenty of fairly simple programs that have complex configuration schemes just to load a few settings. The project I worked on this week has 2.8k lines of code and 8 settings and uses a config library with 2.2k lines of code plus 9.5k from the YAML library. I don’t think that really makes things easier.

                        By comparison, Go’s flag package is 1k lines, in the standard library, and this app was already loading that for a -config flag to give the location of the configuration file.

              2. 1

                Flags a weird that in some sense they take the place of metadata, while they are just as much text as anything else. I guess getopt helps to hide this mysticism. This causes quite a lot of inconsistencies, -v, -v [necessary arg], –verbose, –verbose [necessary arg], –verbose=[not-necessary arg], etc. could all be used, but aren’t interchangable, even though one could imagine a shell that understands all these notations and generally configures/passes the values to a program. Maybe even do type-checking automatically?

                For my own blog, I use a system like flags to pass arguments to the blog system, and I’ve taken the convention to declare boolean properties with a prepended : (eg. :syntax) while properties with values start with = (eg. :next part32.html). Seems more clean.