1. 74
  1.  

  2. 16

    The most interesting part is this tool:

    https://github.com/GoogleContainerTools/distroless

    Pretty amazing small images.

    1. 20

      The devops people are slowly rediscovering static linking

      1. 10

        The main advantage of distroless seems to be dynamically linking glibc properly in a minimal container.

        1. 16

          what if the container was a file, and the file was an executable. that’d be pretty minimal. they’ll get there some day…

          1. 14

            And then the cycle will begin again. “Hey I have this file but it doesn’t run because it needs to be in the presence of this other file”.

        2. 4

          I am not sure what you mean. Static linking was always an option at different levels. Single JAR file is a statically linked package for Java apps for example. You can do the same thing with AWS Lambda packages (single zip).

          1. 11

            At least last I checked, a JAR file is a self-contained package but not actually statically linked. All the class path lookup stuff is still done, dynamically, when the program is run.

      2. 13

        I distribute my Go-based commercial software in an alpine-based Docker container which weighs in at 15MB. 5MB is my software, 3-4MB is ca-certificates. Alpine and musl are tiny and wonderful.

        1. 10

          Good to know I could still fit my livelihood on a few 1.44MB floppies. 🤣

          1. 5

            . Single JAR file is a statically linked package for Java apps for example. You can do the same thing with AWS Lambda packages (single zip).

            Are you using some custom ca? AFAIK, those certs are still around <1MB.

            du -sh /usr/share/ca-certificates/

            616.0K /usr/share/ca-certificates/

            1. 4

              Why not use a scratch container? What does all that overhead get you?

            2. 7

              I don’t follow the issue here. If glibc always versioned symbols and musl never versioned symbols, then every symbolic reference would be completely unambiguous. Alpine binaries would always use musl, and newly compiled binaries might use glibc, but their dependencies would use musl (so both in one process), but every reference is unambiguous.

              This requires basic ABI cleanliness - things like a library cannot malloc() and expect its caller to free(), since those might be calls to different C libraries. Each module needs to not leak its C library implementation, but that’s achievable (although not necessarily achieved.)

              I think the issue is more that glibc doesn’t always version symbols, and ELF doesn’t have a two level namespace, unlike OS X. OS X uses library + symbol to resolve symbols, whereas ELF traditionally only looked for a symbol (without library) which is very problematic once two C libraries are loaded into one process and are trying to resolve the same symbol name. I’d be interested to know if Linux has changed in this regard; obviously each binary encodes which shared libraries it needs, so the symbol lookup not checking for a library name was always a bit odd. It’s also the basis for how things like LD_PRELOAD work, because they can resolve a symbol from a completely different library.

              If I’m right, the problem isn’t symbol versioning, it’s the lack of symbol versioning. With no symbol versioning, alpine-glibc wouldn’t have got off the ground.

              1. 7

                This requires basic ABI cleanliness - things like a library cannot malloc() and expect its caller to free(), since those might be calls to different C libraries.

                I think you have partially answered your own question. It’s really common for C libraries to have functions that take ownership of their arguments and require that those are heap allocated values, assuming they can free() them when they’re done, or conversely they’ll malloc() something and return it to the caller, and part of the specified API is that the caller is responsible for free()-ing it. So it’s actually really really common for one library to malloc() and another to free(), and if they’re calling out to different implementations of libc, Bad Things will happen.

                Additionally any library that expects to “own” part of the process state (which definitely includes libc) is going to run into problems with multiple copies/versions of it linked into the same process. E.g. from How To Corrupt An SQLite Database File:

                As pointed out in the previous paragraph, SQLite takes steps to work around the quirks of POSIX advisory locking. Part of that work-around involves keeping a global list (mutex protected) of open SQLite database files. But, if multiple copies of SQLite are linked into the same application, then there will be multiple instances of this global list. Database connections opened using one copy of the SQLite library will be unaware of database connections opened using the other copy, and will be unable to work around the POSIX advisory locking quirks. A close() operation on one connection might unknowingly clear the locks on a different database connection, leading to database corruption.

                The scenario above sounds far-fetched. But the SQLite developers are aware of at least one commercial product that was released with exactly this bug. The vendor came to the SQLite developers seeking help in tracking down some infrequent database corruption issues they were seeing on Linux and Mac. The problem was eventually traced to the fact that the application was linking against two separate copies of SQLite. The solution was to change the application build procedures to link against just one copy of SQLite instead of two.

                1. 5

                  So it’s actually really really common for one library to malloc() and another to free()

                  Well, where I’m from, that’s a very clear no-no. There’s probably cultural differences at play here, but note that on Windows, the version of the C runtime library is determined by whatever compiler the application developer chooses, and any library exposing a stable ABI for application developers can’t know that in advance. Binary compatibility, when combined with continual C runtime library changes, implies a need to eliminate this pattern.

                  This leads to a handful of fairly simple patterns that always need to be followed:

                  1. Expose a function that can indicate the size of an allocation, have the caller allocate, and call a second time (or possibly different function) with an appropriate buffer;
                  2. Have a library allocate an object and expose a handle (opaque pointer) where all operations on the pointer, including its destruction, are owned by the module that allocated it;
                  3. In rare cases, specify the allocator used to interchange allocations across a module boundary. This is probably the least common and least clean, but lingers in some places, eg. clipboard.
                  1. 7

                    On Linux, from-source builds are the norm; binary compatibility is indeed fiddlier than I expect it is on proprietary platforms, and shipping cross-distro dynamically linked binaries can be done but is definitely a second-class citizen. Typically, a distro will have one version of libc installed, and if you want to drop a binary onto that distro you have a couple scenarios:

                    1. The binary was dynamically linked, and built against a compatible version of libc.
                    2. The binary was dynamically linked, and built against an incompatible version of libc. you’re probably SOL if you can’t build from source.
                    3. The binary is statically linked, in which case it doesn’t use any system libraries and talks directly to the kernel ABI (which is very stable).

                    For proprietary software vendors static linking isn’t a great option because LGPL licensed libraries require that the user can swap out a compatible version of the library – dynamic linking is usually how this is achieved if you’re not distributing source. So these vendors tend to go to the trouble of making sure (1) is the case, which can be done if the target system uses some version of glibc and you compile against a version that is at least as old as anything your users might be using.


                    In any case, the fact that this pattern does work on basically any Linux system means that applications will inevitably rely on it, and building a system where you might have two versions of malloc()/free() in the same process is indeed a recipie for disaster; the article is right to suggest that this is a terrible idea.

                    (Vague tangent; I read somewhere that internally at Google, if a service is exceeding its SLO, the maintainers will intentionally introduce artificial downtime, to ensure that users aren’t relying on it being more reliable than it’s supposed to be).

                  2. 4

                    So it’s actually really really common for one library to malloc() and another to free()

                    IME that’s not very common, and it’s actually very poor library design to return pointers that need to be cleaned up by the user directly with free()/delete.

                    The libraries I’ve used almost always provide explicit cleanup functions to handle the pointers they return. GDAL, to pick an example I’ve used recently, has a whole bunch of Create* functions that return pointers, and bunch of corresponding Destroy* functions that clean up after them.

                    Not only does it avoid the problem of C library differences, but it also lets the library swap in different mallocs (such as jemalloc) without impacting developers using the library, and it simplifies using the library from languages other than C (if I use your library from Python or Haskell, how do I ‘free’ a pointer?).

                    I’m not saying it doesn’t happen, but IMO it’s a “code smell”, and a good indicator I should look for a different library.

                    1. 2

                      I’m not saying it doesn’t happen, but IMO it’s a “code smell”, and a good indicator I should look for a different library.

                      Fair enough. “really common” isn’t exactly a precise number and I don’t have hard numbers for you; I would agree that it is the minority relative to encapsulating allocation in wrapper functions, but it is common enough that, in the context of the article, it doesn’t really matter whether it’s good or bad, if your system breaks code that does this you’re going to have a bad time.

                    2. 1

                      It’s really common for C libraries to have functions that take ownership of their arguments and require that those are heap allocated values, assuming they can free() them when they’re done, or conversely they’ll malloc() something and return it to the caller, and part of the specified API is that the caller is responsible for free()-ing it. So it’s actually really really common for one library to malloc() and another to free(), and if they’re calling out to different implementations of libc, Bad Things will happen.

                      I’m sorry, you’re posting with the voice of authority but I think you’re just wrong. That’s bad API design and it’s just begging for crashes. APIs that start off like that might be common, but sooner rather than later (assuming user adoption) someone is going to point this out and after the requisite denial period it’ll get fixed or the project will go nowhere because no serious user will integrate such a library into their code.

                      The biggest culprit is poorly designed C++ libraries that pass objects across API boundaries, which are then implicitly freed, but C APIs are much more resilient by nature since they typically pass pointers instead. Example: https://github.com/yue/yue/issues/82

                    3. 3

                      The problem is:

                      However, [alpine-glibc] is conceptually flawed, because it uses system libraries where available, which have been compiled against the musl C library.

                      So you’ve got an unholy, leaky mix of versioned and unversioned symbols from two different implementations calling into each other

                      1. 3

                        …with their own mallocs

                    4. 4

                      I feel like what we really need is a distro that is small but still uses glibc instead of musl.

                      1. 3

                        No, what we need is to end the glibc monoculture and make software less brittle across the board.

                        1. 2

                          Void is fairly small.