1. 10

    I wonder why the kernel community seems to have structural issues when it comes to filesystem - btrfs is a bit of a superfund site, ext4 is the best most people have, and ReiserFS’s trajectory was cut short for uh, Reasons. Everything else people would want to use (i.e. ZFS, but also XFS, JFS, AdvFS, etc.) are hand-me-downs from commercial Unix vendors.

    1. 11

      On all of the servers I deploy, I use whatever the OS defaults to for a root filesystem (generally ext4) but if I need a data partition, I reach for XFS and have yet to be disappointed with it.

      Ext4 is pretty darned stable now and no longer has some of the limitations that pushed to me XFS for large volumes. But XFS is hard to beat. It’s not some cast-away at all, it’s extremely well designed, perhaps as well or better than the rest. It continues to evolve and is usually one of the first filesystems to support newer features like reflinks.

      I don’t see why XFS couldn’t replace ext4 as a default filesystem in general-purpose Linux distributions, my best guess as to why it hasn’t is some blend of “not-invented-here” and the fact that ext4 is good enough in 99% of cases.

      1. 3

        It would be great if the recent uplift of xfs also added data+metadata checksums. It would be perfect for a lot of situations where people want zfs/btrfs currently.

        It’s a great replacement for ext4, but not other situations really.

        1. 1

          Yes, I would love to see some of ZFS’ data integrity features in XFS.

          I’d love to tinker with ZFS more but I work in an environment where buying a big expensive box of SAN is preferable to spending time building our own storage arrays.

          1. 1

            I’m not sure if it’s what’s you meant, but XFS now has support for checksums for at-rest protection against bitrot. https://www.kernel.org/doc/html/latest/filesystems/xfs-self-describing-metadata.html

            1. 2

              This only applies to the metadata though, not to the actual data stored. (Unless I missed some newer changes?)

              1. 1

                No, you’re right. I can’t find it but I know I read somewhere in the past six months that XFS was getting this. The problem is that XFS doesn’t do block device management which means at best it can detect bitrot but it can’t do anything about it on its own because (necessarily) the RAIDing would take place in another, independent layer.

          2. 3

            I don’t see why XFS couldn’t replace ext4 as a default filesystem in general-purpose Linux distributions

            It is the default in RHEL 8 for what it’s worth
            https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_file_systems/assembly_getting-started-with-xfs-managing-file-systems

          3. 2

            Yep. I use xfs for 20 years now when I need a single drive FS and I use zfs when I need multiple drive FS. The ext4 and brtfs issues did not increase my confidence.

          1. 2

            I still use fish. It does the autocomplete part very well which is 90% what I need from a shell. I do not have time to go through gazillion lines of shell configuration to get the same in zsh. I do not want to maintain this much configuration. My fish config is 40 lines. Most of it is git related.

            1. 2

              I really like fish(1) interactive mode and its really a decent shell. I would say that currently zsh(1) and fish(1) are the best currently available interactive shells.

              I tried to use it for a while (not forcing myself - just trying it) and I REALLY miss that I can not use these one liners/typical loops or pipes that I have been using for more then a decade in the fish(1) shell - like those:

              # for I in *.mp4; do ff.mp3.320.sh "${I}"; done
              # find . -type f -iname \*.jpg | while read I; do echo -n "${I} "; identify -format %Q; echo; done
              

              I really can not understand why the fish(1) developers would NOT implement POSIX /bin/sh syntax and invent their own one … its really a NO GO for me.

            1. 1

              There are many things wrong with AWS permissions. The article example is kind of weird. Generally speaking the new services does not play too well with the older services (config service with iam in this scenario).

              1. -5

                Assuming I know what tac is. Assuming I have a problem of grepping through log files that are huge(!) in reverse order. Assuming the task is time critical. Assuming that the performance is limited by the tool and not IO.

                Finally we arrived where a simd tool would be good. Too bad you lost me 3 assumptions sooner.

                1. 10

                  It’s okay to simply not comment if an article isn’t relevant to you. The author made this tool for themselves, evidently satisfying all 3 of these assumptions, and was kind enough to share it. Moreover, even someone without a need for any version of tac would still find this article useful if they wanted runtime SIMD detection in Rust.

                  1. 5

                    This is rude and dismissive. Please do not post comments like this in future.

                  1. 16

                    The most interesting part is this tool:

                    https://github.com/GoogleContainerTools/distroless

                    Pretty amazing small images.

                    1. 20

                      The devops people are slowly rediscovering static linking

                      1. 10

                        The main advantage of distroless seems to be dynamically linking glibc properly in a minimal container.

                        1. 16

                          what if the container was a file, and the file was an executable. that’d be pretty minimal. they’ll get there some day…

                          1. 14

                            And then the cycle will begin again. “Hey I have this file but it doesn’t run because it needs to be in the presence of this other file”.

                        2. 4

                          I am not sure what you mean. Static linking was always an option at different levels. Single JAR file is a statically linked package for Java apps for example. You can do the same thing with AWS Lambda packages (single zip).

                          1. 11

                            At least last I checked, a JAR file is a self-contained package but not actually statically linked. All the class path lookup stuff is still done, dynamically, when the program is run.

                        1. 4

                          Funny, I was expecting to read about all the subject he listed instead of password storage.

                          1. 6

                            Debian has become my favorite Linux distro recently. It is surprisingly good.

                            1. 10

                              Why surprisingly? It’s the foundation of a lot lot lot of distros

                              1. 5

                                That’s interesting, because I can’t stand it. Things are changed at random so programs are different, and package management seems to like to break at random. What makes it your favourite?

                                1. 4

                                  I’ve been using stable since 2008. I can remember one instance where pulling in a package update broke a program: it was when Chromium switched to requiring a GPU and then had a horrid CVE that no one could backport to stable; suddenly I couldn’t use it over ssh -X any more. Hard to place much blame on Debian for that though.

                                  I’d imagine you’d see more breakage running unstable but like … that’s literally what it says on the tin.

                                  1. 2

                                    What in the world? There is hardly a less “things break at random” system than Debian.

                                    1. 2

                                      Apache’s configs being entirely different for no reason has bitten me more than once. Still not sure why that is, it’s pretty horribly annoying.

                                      1. 1

                                        Why would you think it’s for no reason? Actually that’s an example of exceptional care being taken to avoid things breaking at random. It means that you can have packages that provide Apache modules, and packages that provide entire websites (say, Wordpress or netdata), and expressible dependencies between them, and they don’t step on each other, local config is never blown away by upgrading packages, and generally speaking the sysadmin doesn’t get unpleasant surprises.

                                        Admittedly, it doesn’t really feel “different” to me, because I got used to it 20 years ago and have only rarely had to deal with Apache on systems that didn’t do it that way, but what I will say is that it works, and that it’s not arbitrary — basically all of the packaging guidelines (which amounts to a book worth of material) come from a place of “how to be predictable, reproducible, and avoid breakage”. But it is assumed that admins know their system; being different from some other distro or upstream defaults isn’t breakage in and of itself, when those differences are justified and documented (which they certainly are in the case of Apache).

                                        1. 2

                                          I’d rather have the config files match with examples online (and the official docs!) than have any of those features, though. What do other distros do with Apache modules? I’ve never had breakage in them and they don’t do anything like debian.

                                      2. 1

                                        This might have been around 2004, when I encountered a legacy Debian Stable that had missed out on a stable release. So it was like oldoldstable.

                                        Upgraded directly to the contemporary stable and everything went perfectly, despite skipping over a major release altogether.

                                        It was beautiful.

                                    2. 3

                                      It’s been mine since around 1996. I try running other operating systems but I’m so used to the availability of software, the stability of updates and the general attention to detail that I have a hard time using anything else.

                                    1. 4

                                      7 developers contributed to this change in 7 days.

                                      1. 14

                                        What’s going on here? How did this get to the top of lobste.rs with 26 upvotes? I’m happy for the OP that they could get their system to work, but as far as I can tell, the story here is “package manager used to manage packages.” We have been doing that for decades. Is there any way the community can get a lever to push back on thin stories like this one?

                                        1. 25

                                          Would it change your opinion if the article mentioned that the nix shell being used here is entirely disposable and this process leaves no mark in your OS setup? Also that even if this required some obscure versions of common system dependencies you could drop into such a shell without worrying about version conflicts or messing up your conventional package manager?

                                          I agree that the article is thin in content, but I don’t think you can write this story off as “package manager used to manage packages.” , I think nix shell is very magical in the package management world.

                                          1. 6

                                            I could do that with docker too and it would not leave a trace either

                                            1. 17

                                              Yes, but then you’d be inside a container, so you’d have to deal with the complexities of that, like mounting drives, routing network traffic etc. With nix shell, you’re not really isolated, you’re just inside a shell session that has the necessary environment variables that provide just the packages you’ve asked for.

                                              Aside from the isolation, the nix shell is also much more composable. It can drop you into a shell that simultaneously has a strange Java, python and Erlang environment all compiled with your personal fork of GCC, and you’d just have to specify your GCC as an override for that to happen.

                                              1. 4

                                                I get that, but I have to go through the learning curve of nix-shell, while I already know docker, since I need it for my job anyway. I am saying that there are more ways to achieve what the article is talking about. It is fine that the author is happy with their choice of tools, but it is very unremarkable for the title and given how many upvotes that article got.

                                                1. 5

                                                  Why not learn nix and then use it at work as well :) Nix knows how to package up a nix-defined environment into a docker container and produce very small images, and you don’t even need docker itself to do that. That’s what we do at work. I’m happy because as far as I’m concerned Nix is all there is and the DevOps folks are also happy because they get their docker images.

                                                  1. 3

                                                    I work in a humongous company where the tools and things are less free to choose from atm, so even if I learned nix, it would be a very tough sell..

                                              2. 3

                                                As someone who hasn’t used Docker, it would be nice to see what that looks like. I’m curious how the two approaches compare.

                                                1. 6

                                                  I think that the key takeaway is that with Docker, you’re actually running a container will a full-blown OS inside. I have a bias against it, which is basically just my opinion, so take it with a grain of salt.

                                                  I think that once the way to solve the problem of I need to run some specific version of X becomes let’s just virtualize a whole computer and OS because dependency handling is broken anyway, we, as a category simply gave up. It is side-stepping the problem.

                                                  Now, the approach with Nix is much more elegant. You have fully reproducible dependency graphs, and with nix-shell you can drop yourself in an environment that is suitable for whatever you need to run regardless of dependency conflicts. It is quite neat, and those shells are disposable. You’re not running in a container, you’re not virtualizing the OS, you’re just loading a different dependency graph in your context.

                                                  See, I don’t use Nix at all because I don’t have these needs, but I played with it and was impressed. I dislike our current approach of just run a container, it feels clunky to me. I think Docker has it’s place, specially in DevOps and stuff, but using it to solve the I need to run Python 2.x and stuff conflicts with my Python 3.x install is not the way I’d like to see our ecosystem going.


                                                  In the end, from a very high-level, almost stratospheric, point-of-view: both docker and nix-shell workflow will be the developer typing some commands on the terminal, and having what they need running. So from a mechanical standpoint of needing to run something, they’ll both solve the problem. I just don’t like how solving things by doing the evergreen is now the preferred solution.

                                                  Just be aware that this is an opinion from someone heavily biased against containers. You should play with both of them and decide for yourself.

                                                  1. 3

                                                    This comment is a very good description of why I’ve never tried Docker (and – full disclosure – use Nix for things like this).

                                                    But what I’m really asking – although I didn’t make this explicit – is a comparison of the ergonomics. The original post shows the shell.nix file that does this (although as I point out in another comment, there’s a shell one-liner that gets you the same thing). Is there an equivalent Dockerfile?

                                                    I was surprised to see Docker brought up at all because my (uninformed) assumption is that making a Docker image would be prohibitively slow or difficult for a one-off like this. I assumed it would be clunky to start a VM just to run a single script with a couple dependencies. But the fact that that was offered as an alternative to nix-shell makes me think that I’m wrong, and that Docker might be appropriate for more ad-hoc things than I expected, which makes me curious what that looks like. It points out a gap in my understanding that I’d like to fill… with as little exertion of effort as possible. :)

                                                    1. 4

                                                      But the fact that that was offered as an alternative to nix-shell makes me think that I’m wrong, and that Docker might be appropriate for more ad-hoc things than I expected, which makes me curious what that looks like. It points out a gap in my understanding that I’d like to fill… with as little exertion of effort as possible. :)

                                                      I think containers is a perfectly capable solution to this. The closest thing you can use would probably be toolbox.

                                                      https://github.com/containers/toolbox

                                                      It would allow you to even provide a standardized environment which would be decoupled from the deployment itself (if that makes sense). It also mount $HOME as well.

                                                      1. 3

                                                        I use Nix, but also have experience with Toolbox.

                                                        I would recommend most people to use Toolbox over nix-shell. With toolbox you can create one-off containers in literally seconds (it’s two commands). After entering the container you can just dnf install whatever you need. Your home directory gets mounted, so you do not have to juggle with volumes, etc. If you need to create the same environment more often, you can create a Dockerfile and build your toolbox containers with podman. The upstream containers that Fedora provides are also just built using Dockerfiles.

                                                        The post shows a simple use case, but if you want to do something less trivial, it often entails learning Nix the language and nixpkgs (and all its functions, idioms, etc.). And the Nix learning curve is steep (though it is much simpler if you are familiar with functional programming). This makes the toolbox approach orders of magnitude easier for most people - you basically need to know toolbox create and toolbox enter and you can use all the knowledge that you already have.

                                                        However, a very large shortcoming of toolbox/Dockerfiles/etc. is reproducibility. Sure, you can pass around an image and someone else will have the same environment. But Nix allows you to pin all dependencies plus the derivations (e.g. as a git SHA). You can give someone your Nix flake and they will have exactly the same dependency graph and build environment guaranteed.

                                                        Another difference is that once you know Nix, it is immensely powerful for defining packages. Nix is a turing-complete functional language, so nixpkgs can provide a lot of powerful abstractions. I dread every time I have to create/modify and RPM spec file, because it is so primitive compared to making a Nix derivation.

                                                        tl;dr: most people will want to use something like Toolbox, it is familiar and provides many of the same benefits as e.g. nix-shell (isolated, throw-away environments, with your home directory available). However, if you want strong reproduciblity across systems and a more powerful packaging/configuration language, learning Nix is worth it.

                                                      2. 3

                                                        A cool aspect of Docker is that it has a gazillion images already built and available for it. So depending on what you need, you’ll find a ready-made image you can put to good use with a single command. If there are no images that fill your exact need, then you’ll probably find an image that is close enough and can be customised. You don’t need to create images from scratch. You can remix what is already available. In terms of ergonomics, it is friendly and easy to use (for these simple cases).

                                                        So, NixPkgs have a steeper learning curve in comparison to dockerfiles. It might be simpler to just run Docker. What I don’t like is what is happening inside Docker, and how the solution for what looks like simple problems involves running a whole OS.

                                                        I’m aware that you can have containers without an OS like described in this thread, but that is not something I often see people using in the wild.

                                                      3. 1

                                                        Nit-pick: AFAIK one doesn’t really need Alpine or any other distro inside the container. It’s “merely” for convenience. AFAICT it’s entirely possible to e.g. run a Go application in a container without any distro. See e.g. https://www.cloudbees.com/blog/building-minimal-docker-containers-for-go-applications

                                                  2. 3

                                                    Let’s assume nix shell is actual magic — like sourcerer level, wave my hand and airplanes become dragons (or vice versa) magic — well this article just demonstrated that immense power by pulling a coin out of a deeply uncomfortable kid’s ear while pulling on her nose.

                                                    I can’t speak for the previous comment’s author, but those extra details, or indeed any meat on the bones, would definitely help justify this article’s otherwise nonsensical ranking.

                                                    1. 2

                                                      Yeah, I agree with your assessment. This article could just as well have the title “MacOS is so fragile, I consider this simple thing to be an issue”. The trouble with demonstrating nix shell’s power is that for all the common cases, you have a variety of ad-hoc solutions. And the truly complex cases appear contrived out of context (see my other comment, which you may or may not consider to be turning airplanes into dragons).

                                                  3. 19

                                                    nix is not the first thing most devs would think of when faced with that particular problem, so it’s interesting to see reasons to add it to your toolbox.

                                                    1. 9

                                                      Good, as it is not supposed to be the first thing. Learning a fringe system with a new syntax just to do something trivial is not supposed to be the first thing at all.

                                                    2. 4

                                                      I find it also baffling that this story has more upvotes than the excellent and original code visualization article currently also very high. Probably some nix up vote ring pushing this

                                                      1. 12

                                                        Or folks just like Nix I guess? 🤷

                                                        1. 11

                                                          Nix is cool and people like it.

                                                          1. 5

                                                            I didn’t think this article was amazing, but I found it more interesting than the code visualization one, which lost me at the first, “From this picture, you can immediately see that X,” and I had to search around the picture for longer than it would have taken me to construct a find command to find the X it was talking about.

                                                            This article, at least, caused me to say, “Oh, that’s kind of neat, wouldn’t have thought of using that.”

                                                          2. 6

                                                            This article is useless. It is way simpler (and the python way) to just create a 2.7 virtualenv and run “pip install psycopg2 graphwiz”. No need to write a nix file, and then write a blog post to convince yourself you didn’t waste your time!

                                                            Considering all nix posts get upvoted regardless of content, it’s about time we have a “nix” tag added to the site.

                                                            1. 14

                                                              This article is not useless just because you don’t see its value.

                                                              I work mainly with Ruby and have to deal with old projects. There are multiple instances where the Ruby way (using a Ruby version manager) did not work because it was unable to install an old Ruby version or gem on my new development machine. Using a nix-shell did the job every time.

                                                              just create a 2.7 virtualenv and run “pip install psycopg2 graphwiz”

                                                              What do you do if this fails due to some obscure dependency problem?

                                                              1. 4

                                                                What do you do if this fails due to some obscure dependency problem?

                                                                Arguably you solve it by pinning dependency versions in the pip install invocation or requirements.txt, as any Python developer not already using Nix would do.

                                                                This article is not useless just because you don’t see its value.

                                                                No, but it is fairly useless because it doesn’t do anything to establish that value, except to the choir.

                                                                1. 2

                                                                  In my experience there will be a point where your dependencies will fail due to mismatching OpenSSL, glibc versions and so on. No amount of pinning dependencies will protect you against that. The only way out is to update dependencies and the version of your language. But that would just detract from your goal of getting an old project to run or is straight up impossible.

                                                                  Enter Nix: You pin the entire environment in which your program will run. In addition you don’t pollute your development machine with different versions of libraries.

                                                                  1. 3

                                                                    Arguably that’s just shifting the burden of effort based on a value judgement. If your goal is to get an old project to run while emphasizing the value of incurring zero effort in updating it, then obviously Nix is a solution for you and you’ll instead put the effort into pinning its entire runtime environment. If, however, your value to emphasize is getting the project to run then it may well be a more fruitful choice to put the effort into updating the project.

                                                                    The article doesn’t talk about any of the hairier details you’re speaking to, it just shows someone taking a slightly out of date Python project and not wanting to put any personal effort into updating it… but updating it by writing a (in this case relatively trivial) Python 3 version and making that publicly available to others would arguably be the “better” solution, at least in terms of the value of contributing back to the community whose work you’re using.

                                                                    But ultimately my argument isn’t with the idea that Nix is a good solution to a specific problem, it’s that this particular article doesn’t really make that point and certainly doesn’t convincingly demonstrate the value of adding another complex bit of tooling to the toolkit. All the points you’ve raised would certainly help make that argument, but they’re not sadly not present in this particular article.

                                                                2. 1

                                                                  Just out of curiosity, I’m also dealing with ancient ruby versions and use nix at work but I couldn’t figure out how to get old enough versions, is there something that helps with that?

                                                                    1. 1

                                                                      Thank you, very helpful!

                                                                      1. 1

                                                                        Do note this method will get you a ruby linked to dependencies from the same checkout. In many cases this is what you want.

                                                                        If instead you want an older ruby but linked to newer libraries (eg, OpenSSL) there’s a few extra steps, but this is a great jumping off point to finding derivations to fork.

                                                                        1. 1

                                                                          Do note this method will get you a ruby linked to dependencies from the same checkout. In many cases this is what you want.

                                                                          Plus glibc, OpenSSL and other dependencies with many known vulnerabilities. This is fine for local stuff, but definitely not something you’d want to do for anything that is publicly visible.

                                                                          Also, note that mixing different nixpkgs versions does not work when an application uses OpenGL, Vulkan, or any GPU-related drivers/libraries. The graphics stack is global state in Nix/NixOS and mixing software with different glibc versions quickly goes awry.

                                                                    2. 2

                                                                      This comment mentions having done something similar with older versions by checking out an older version of the nixpkgs repo that had the version of the language that they needed.

                                                                      1. 2

                                                                        Like others already said you can just pin nixpkgs. Sometimes there is more work involved. For example this is the current shell.nix for a Ruby on Rails project that wasn’t touched for 5 years. I’m in the process of setting up a reproducible development environment to get development going again. As you can see I have to jump through hoops to get Nokogiri play nicely.

                                                                        There is also a German blog post with shell.nix examples in case you need inspiration.

                                                                    3. 4

                                                                      this example, perhaps. I recently contributed to a python 2 code base and running it locally was very difficult due to c library dependencies. The best I could do at the time was a Dockerfile (which I contributed with my changes) to encapsulate the environment. However, even with the container standpoint, fetching dependencies is still just as nebulous as “just apt install xyz.” Changes to the base image, an ambiently available dependency or simply turning off the distro package manager services for unsupported versions will break the container build. In the nix case, it is sort of forced on the user to spell it out completely what the code needs, combine with flakes and I have a lockfile not only for my python dependencies, but effectively the entire shell environment.

                                                                      More concretely, at work, the powers to be wanted to deploy python to an old armv7 SoC running on a device. Some of the python code requires c dependencies like openssl, protobuf runtime and other things and it was hard to cross compile this for the target. Yes, for development it works as you describe, you just use a venv, pip install (pipenv, poetry, or whatever as well) and everything is peachy. then comes to deployment:

                                                                      1. First you need to make a cross-compiled python interpreter, which involves first building the interpreter for your host triple then rebuilding the same source for the target host triple making sure to tell the build process where the host triple build is. This also ignores that some important python interpreter things may not build, like ctypes.
                                                                      2. Learn every environment variable you need to expose to the setup.py or the n-teenth build / packaging solution for the python project you want to deploy, hope it generates a wheel. We will conveniently ignore how every C depending package may use cmake, or make, or meson, etc, etc…
                                                                      3. make the wheels available to the image you actually ship.

                                                                      I was able to crap out a proof-of-concept in a small nix expression that made a shell that ran the python interpreter I wanted with the python dependencies needed on both the host and the target and didn’t even have to think. Nixpkgs even gives you cross compiling capabilities.

                                                                      1. 1

                                                                        Your suggested plan is two years out of date, because CPython 2.7 is officially past its end of life and Python 2 packages are generally no longer supported by upstream developers. This is the power of Nix: Old software continues to be available, as if bitrot were extremely delayed.

                                                                        1. 3

                                                                          CPython 2.7 is available in debian stable (even testing and sid!), centos and rhel. Even on MacOS it is still the default python, that ships witht he system. I don’t know why you think it is no longer available in any distro other than nix.

                                                                    1. 1

                                                                      Yeah implementation details matter.

                                                                      1. 7

                                                                        Yes you need a job orchestrator to abstract away your machines.

                                                                        You should be running Hashicorp’s Nomad unless you are a Really Massive Shop With People To Spare On Operating Kubernetes.

                                                                        In nomad I can run arbitrary jobs as well as run and orchestrate docker containers. This is something Kubernetes can’t do.

                                                                        In nomad I can upgrade without many gymnastics. That feels good.

                                                                        1. 13

                                                                          Operating a simple kubernetes cluster isn’t that bad, especially with distributions such as k3s and microk8s.

                                                                          You should be running Hashicorp’s Nomad unless you are a Really Massive Shop With People To Spare On Operating Kubernetes.

                                                                          You should do what works for your team/company/whatever. There’s more than just Nomad and Kubernetes and people should be equipped to make decisions based on their unique situation better than someone on the internet saying they should be using their particular favorite orchestration tool.

                                                                          1. 6

                                                                            There’s more than just Nomad and Kubernetes

                                                                            For serious deployment of containerised software, is there really? I did quite a bit of digging and the landscape is pretty much Nomad/Kubernetes, or small attempts at some abstraction like Rover, or Dokku for a single node. Or distros like OpenShift which are just another layer on Kubernetes.

                                                                            I’m still sad about rkt going away…

                                                                            1. 1

                                                                              LXD is still around and developed!

                                                                              1. 3

                                                                                LXD is for system containers, not application containers. Pets, not cattle.

                                                                                I really enjoy using LXD for that purpose, though. Feels like Solaris Zones on Linux (but to be honest, way less integrated than it should be because Linux is grown and illumosen are designed).

                                                                            2. 2

                                                                              Well said ngp. Whether you want to talk about Kubernetes or the class of orchestrators like it, it’s clear that there are enough tangible benefits to the developer and devops workflow that at least some of the paradigms or lessons are here to stay.

                                                                            3. 2

                                                                              No, I do not.

                                                                              1. 2

                                                                                I’ve only heard “you should run Nomad instead” in online comments, every time I hear about Nomad in person (eg from infra at Lob) it’s “we’re migrating from Nomad to X (usually Kubernetes) because Y”

                                                                                1. 1

                                                                                  I haven’t tried Nomad yet, even though I heard nice things about it, and it seems they like Nix. What would be Ys that others list, what are the corner cases where one should avoid Nomad?

                                                                                  1. 1

                                                                                    I think when a lot of work would be saved by using existing helm charts.

                                                                                    Or when you need to integrate with specific tooling.

                                                                                    Or when you need to go to a managed service and it’s easier to find k8s.

                                                                                    And finally I think when you need to hire experienced admins and they all have k8s and not nomad.

                                                                              1. 51

                                                                                I don’t believe Kubernetes and its primitives are a good fit for most companies out there. The article mentions Kubernetes, but what they are really saying is, you should make your applications containerizable and compatible with orchestrators.

                                                                                You should save yourself the pain and burden of using Kubernetes, and try out orchestrators like Nomad (from HashiCorp).

                                                                                1. 15

                                                                                  I could not agree more. Any POC with k8s we did failed. I am not sure what it is for. The workloads we have perfectly fine on VMs with auto-scaling and there is no need for a 1M+ LOC solution that is looking for a problem.

                                                                                  1. 7

                                                                                    This is the tune I’ve been singing for a while, but I think I’m on the verge of changing it. Not because I suddenly think Kubernetes is a fantastic fit for every possible application on the planet, but because the “Kubernetes and containerization and execution environment are just different words for the same exact thing” viewpoint is becoming such an overwhelming consensus in the industry that I suspect it’s shortly going to get hard to hire people who know how to deploy code to anything else.

                                                                                    It is really feeling to me like I’m dying on the wrong hill by insisting that it can be fine to use Nomad or Docker Compose or systemd to run a service at small scale. Kubernetes seems to just be winning everywhere, k8s-only tools are starting to show up more and more often, and at some point it’s going to get more expensive than it’s worth to be the odd man out.

                                                                                    Makes me sad, but it’s not the first time the industry has converged on a technology that isn’t my first choice.

                                                                                    1. 12

                                                                                      I am still on the “keep it simple” team and I am working on improving integration of Erlang projects with systemd. It makes deployment and configuration much simpler IMHO. I do not need to think about ingress, pods, etc. whatever there is in k8s. I just define my application to listen on port, and it will be started only when needed. I do not need anything more than that.

                                                                                      1. 3

                                                                                        Yup yup.

                                                                                        Leaning heavily on systemd units allows for a heck of a lot of ‘separation’ if you need to run different environments/versions on the same OS instance.

                                                                                        A (now) reasonably long term client originally contacted me asking about ‘setting up Dockers’ a few years ago, and the end result is that we avoided Docker/Containers (the business is essentially one website), but modernised their setup for all infra, to use:

                                                                                        • (a) 100% reproducible setup (using versioned shell scripts mostly, and a bit of Make)
                                                                                        • (b) modernised the ‘runtime’ to use systemd extensively, using a combination of instance templates, and generated unit files
                                                                                        • (c) moved local dev to a Vagrant workflow, with a custom base image built using the same scripts prod/staging/test are built with.

                                                                                        Sure, containers could have been applied to the problem, but that doesn’t mean it’s actually any better at solving the problem. You can bang a nail into a piece of wood with the back of an impact driver if you really want to. That doesn’t mean a simple hammer won’t do the job just as well.

                                                                                  1. 1

                                                                                    Not even mention the only real use case for CDNs? Being close to users so page load is faster.

                                                                                    1. 7

                                                                                      I like the article, but it seems mired in yesteryear. Today we’ve got Percona XtraDB Cluster, Timescaledb, Citus for distributed PostgreSQL, CockroachDB – just to name a few.

                                                                                      I’d be interested in hearing the war stories of folks running any of today’s cluster extensions for MySQL/PostgreSQL/etc.

                                                                                      1. 24

                                                                                        Percona Cluster and Citus require substantial operational expertise and hand-holding from the application code to work properly. Vitess is probably the least tricky of the database sharding layers, and even it requires lots of in-depth knowledge. You can’t just put these databases in Kubernetes / AWS Auto Scaling Groups / whatever and let ’em rip. Even NoSQL databases like Cassandra—designed from the ground up to be active-active—require dedicated provisioning and special care by ops teams.

                                                                                        As a database person, I think the author is basically right. But I would instead say that good ideas have to face reality at some point. Reality usually includes concepts like “users don’t like data loss” and “distributed consensus is hard.”

                                                                                        1. 4

                                                                                          could someone take a moment to elaborate briefly on what kind of experience and special care these sharded sql systems need? i have seen this work happen from a distance don’t quite know what the pitfalls and pain points are. how much of this is avoided when using “sharded database in a box” services, and how much can’t be insulated from the application layer?

                                                                                          1. 3

                                                                                            I’m also curious about real-world stories. I looked around a bit about Vitess. Slack and Square Cash shared info about their migrations. Alas, incentives are probably against sharing all the bad parts, but e.g. Square talked about deadlocks in their migration.

                                                                                            There are nearly-inherent tradeoffs to sharding. Your choice of shard key for each table (user ID, product ID, etc.) has performance implications; queries spanning shards or using cross-shard indexing will be slower (more hops), and cross-shard transactions are much slower if supported, because of the coordination. You tend to need management infrastructure, e.g. metadata servers and something like etcd or zookeeper. Having more stateful boxes means you need smooth automated spin-up, monitoring, backups, and so on.

                                                                                            Seems like there’s work/complexity/resource overhead even for well-packaged sharding, but 🤷‍♀️. When you need it, you need it, and running a larger monolithic database isn’t worry free either. The growth of mature, relational distributed database tools (and some experience at the edge of when a single DB makes sense) has moved me a bit from “stay monolithic until you can’t do otherwise” (that’s a long time!) to thinking you might look at options when you’re feeling real growing pains even if you could stretch it out longer.

                                                                                            1. 10

                                                                                              I work at Cash and have built multiple applications on top of Vitess. Everything you said matches my experience.

                                                                                              • Sharding is hard. Query pattern lock-in.
                                                                                              • Sharding is hard. Cross-shard anything is strongly NOT recommended.
                                                                                              • Sharding is hard. The application will have a lot of special code for it.

                                                                                              For Vitess in particular:

                                                                                              • Running Vitess and MySQL more than doubles operational complexity.
                                                                                              • Vitess is not popular. Good luck 🤞🏾
                                                                                              • I found the admin tooling for Vitess very poor.

                                                                                              We’ve deprecated Vitess for all new services. Most teams have moved off it … though a couple critical services linger. We’ve got dedicated teams working on them.

                                                                                              We’re increasingly an Aurora and DynamoDB shop now.

                                                                                              1. 5

                                                                                                Find an alternative to DynamoDB before you outgrow it. Once you hit that wall it is insanely expensive in engineering to work around it.

                                                                                                I helped run one of the largest DynamoDB tables in all of AWS and we ran into such fun problems as “our backup restore was cancelled because they didn’t have cap” and “autoscaling is failing because it’s taking so long to scale it’s timing out.”

                                                                                                The cloud is fun until it runs out.

                                                                                                We just use S3 today it works much better.

                                                                                                1. 2

                                                                                                  How do you use S3 as a replacement for DynamoDB?

                                                                                                  1. 1

                                                                                                    You treat the entire thing as a KV store. Your bucket key path is whatever you were using for dynamo. Cache the heck out of data in a redis cluster and you should be good to go.

                                                                                                    I know this sounds simple and it is because I don’t have all the details, another team did it. It took them a few months but the outcome was really good.

                                                                                                  2. 1

                                                                                                    To share a bit of “hidden knowledge”, this is true for everything AWS (and I suspect most cloud providers) has, not just DynamoDB. We’re one of AWS’s largest customers at $WORK and we frequently run into all sorts of scaling issues with their services. But we also do ensure that data that sits in DynamoDB is small and doesn’t run into hot shard issues. Larger data usually sits in dbs that we manage which are plagued by all the problems everyone in this thread is discussing.

                                                                                                  3. 1

                                                                                                    How’s Aurora working out for you? We’re considering using the PostgreSQL version.

                                                                                                    1. 2

                                                                                                      We use the MySQL version and not the serverless offering. It’s given me no new pain when contrasted against MySQL instances hosted in our data centres. It’s mostly taken away vertical scaling pain.

                                                                                                      I’m waiting for the other shoe to drop.

                                                                                                    2. 1

                                                                                                      Ah, wow, that changed my outlook on Vitess substantially, even having known parts of it in theory. I wish info like this–not the launch post type stuff but what people learn from day-to-day use–was public more often, though I get the reasons it’s hard for that to happen.

                                                                                                2. 1

                                                                                                  One of my clients uses a relatively small PXC (Percona XtraDb Cluster, for those unaware of the “official” name) setup, with three nodes.

                                                                                                  It has some operational overhead in terms of the service itself obviously, but I’m struggling to think many instances where application code has to allow for the clustered nature (or side effects thereof) specifically.

                                                                                                  Could you elaborate on what you meant here?

                                                                                                3. 2

                                                                                                  You did not mention but TiDB gets my vote.

                                                                                                  1. 2

                                                                                                    We’re experimenting with TiDB. What’s your experience?

                                                                                                    1. 1

                                                                                                      Overall positive. Easy to use for sure. I need more experience with stability and reliability.

                                                                                                1. 47

                                                                                                  The things the article complains you can’t do in the database are only possible in stateless components, which is actually the special case. I mean, systems without state are of limited utility—you have to put it somewhere. So the database is the only “normal” part of the system. It just doesn’t look that way because there are all these stateless boxes on the diagram. It’s actually a big improvement that all the state has been pushed down into one component where you can keep an eye on it.

                                                                                                  1. 34

                                                                                                    The database IS the good idea that lets the team partition risk and coordination into a system that handles the freakishly hard exponentially-compounding problems of performant durable concurrent consistency.

                                                                                                    1. 1

                                                                                                      Having state doesn’t mean you can’t make the database a distributed component. You just need good mechanisms for sharding, replication, consensus, etc. This is a solved problem and the way most large databases work nowadays.

                                                                                                      1. 12

                                                                                                        I think calling it a “solved problem” downplays a lot of the consistency problems we have as an industry still.

                                                                                                        There are solutions, but they are largely incredible feats of engineering and toil.

                                                                                                        One need only look at the Jepsen tests to understand just how dire the situation can be, especially for tools that look to tackle hard problems in replicated database spaces.

                                                                                                        1. 11

                                                                                                          FoundationDB showed how building distributed systems with a simulator can push race discovery much earlier into the dev process, practically solving the issues that Jepsen finds in low-quality (yet ubiquitous) distributed systems that are produced today. A few other industries that create safety-critical software tend to lean heavily into things like discrete event simulation to find their bugs with fewer deaths/headaches/etc… And the hyperscalers are employing these techniques to a greater and greater extent over time because they are able to better understand the benefits of doing so. Open source databases will generally be crappier and crappier compared to the systems offered only as services through the hyperscalers over time. As flashy as any of the NewSQL systems seem, they are really incredibly naive, unreliable, and low-performance compared to the functionality of their closed-source inspirations running internally at cloud providers today.

                                                                                                          Choosing how to shard without hitting perf bottlenecks will also eventually become increasingly automated and driven by basic control theory, but this will happen more from the systems offered as services through hyperscalers rather than the simpler systems that are available in open source.

                                                                                                          Data gravity will continue to cause state to be consolidated in hyperscaler datacenters. Cloud APIs will continue to give preferential interoperability to data that sits in the more expensive hosted databases. The profits of the cloud providers will continue to be focused on hosted storage, rather than its commodified compute complement.

                                                                                                          So, yeah, the database product industry will keep cranking out garbage, not because it’s impossible to avoid, but because their management doesn’t prioritize correctness generally. Open source database products generally don’t have to care about correctness because they are already the system of last resort for people who don’t want to pay for cloud providers or expensive licenses. Meanwhile the database service industry will keep eating the customer base that is fed up with being burned and is willing to pay a premium to think about other problems.

                                                                                                          1. 4

                                                                                                            You’re right, I was being a bit flippant. My point was that there are real-world products that do this, not just research systems.

                                                                                                        2. 1

                                                                                                          I think just by dissect the database (data storage, query engine, transactor) you can gain a lot of knowledge about how to have optimal systems. For example, things that do not require transactional mutations or complex SQL queries can live in a simple k:v store.

                                                                                                        1. 0

                                                                                                          JSON? Really?

                                                                                                          1. 1

                                                                                                            What do you mean?

                                                                                                          1. 12

                                                                                                            This doesn’t even scratch the surface of Haskell comprehensions. They’re a full SQL-like sublanguage and they can operate over non-list Monads.

                                                                                                            Some of their SQL-like abilities https://ghc.gitlab.haskell.org/ghc/doc/users_guide/exts/generalised_list_comprehensions.html#generalised-list-comprehensions

                                                                                                            output = [ (the dept, sum salary)
                                                                                                                     | (name, dept, salary) <- employees
                                                                                                                     , then group by dept using groupWith
                                                                                                                     , then sortWith by (sum salary)
                                                                                                                     , then take 5 ]
                                                                                                            

                                                                                                            All of the examples above are filters & maps, but with parallel comprehensions they can behave like zips: [ (x+y) | x <- [1..10] | y <- [11..20] ]

                                                                                                            Monad comprehensions are even more powerful. This syntax works for anything, not just a list. It works for optional values, non-deterministic code, stateful code, generating actual database queries, etc.

                                                                                                            1. 4

                                                                                                              See also Wadler 1990, “Comprehending Monads”. I read this when I was first trying to grok monads, and it clarified a lot of things for me. After using Python, seeing that monads are “just” list comprehensions really helped!

                                                                                                              1. 2

                                                                                                                Hah. Yes, the list of things that monads “just” are is endless from “just list comprehensions” to “just burritos”. It’s better to focus on specific monads that are interesting and get a feel for them rather than looking to analogies. No one analogy is really enough. For example, if you think of them as list comprehensions it’s harder to see what transformers or comonads would be about.

                                                                                                              2. 1

                                                                                                                I mean, if you turn on more extensions they get more features, but naked Haskell they don’t do quite so many things :)

                                                                                                                Conversely, in naked Haskell you have do which works for any Monad, including list, and has almost-identical syntax to the comprehensions

                                                                                                                1. 4

                                                                                                                  Just because your compiler might, without any flags, default to “naked C++” meaning C++98 doesn’t mean that this is what C++ is today :)

                                                                                                                  Extensions are how Haskell evolves and they represent the language as it is today. This is like writing an article comparing C++ to Java but silently using C++98 (pretty much exactly because “naked Haskell” is Haskell 98). Missing out on 20 years of development will make anything seem less interesting.

                                                                                                                  It’s really unfortunate that we call them extensions. Maybe language feature flags or something would have been better.

                                                                                                                  Actually, you’re more right than you think. Even more modern comprehensions are simple to convert to do notation.

                                                                                                                  1. 1

                                                                                                                    Naked Haskell is at least Haskell2010, not 98! The fair comparison for new C++ versions is new Haskell versions. The fair comparison for Haskell extensions in C++ land would be things like GNU C++. Haskell does a much better job of course by naming all extensions and making them source-level opt-in.

                                                                                                                    And yes, that comprehensions can be written with do was my whole point :)

                                                                                                                    1. 3

                                                                                                                      Haskell2010 is not an actual language standard. It continues the terrible messaging of the community. Haskell98 describes the language in 1998, it’s a language standard. The community tried to write a new language standard, to describe new Haskell (Haskell Prime). And it was a disaster (repeatedly). Haskell 2010 is a minor update to 98 that doesn’t even describe the language as it was in 2000 never mind 2010. This is all described and explained in the report.

                                                                                                                      Even if there is an updated language spec one day, and there may never be, it would not change the situation with extensions. The language will always work by having a ~98 core and then having you enable extensions on top of it. All because this has been such an incredibly successful way of evolving the language.

                                                                                                                      Extensions aren’t 2nd tier in any way. Haskell in 2021 is GHC’s implementation plus a bunch of extensions enabled.

                                                                                                                2. 1

                                                                                                                  Most of the time I use list comprehensions as a replacement for loops. I was not aware that Haskell has these capabilities. I try to use the subset that works in Python, Erlang and other languages.

                                                                                                                1. 4

                                                                                                                  I think on the security side this boils down to: Don’t use unauthenticated database setups and don’t use software that is either badly designed or not well understood.

                                                                                                                  The latter seems to be a common pattern when it comes to setups by people that evidently have put work in. Choosing solutions one can comprehend can both help on the stability and security side of things.

                                                                                                                  1. 15

                                                                                                                    Don’t use unauthenticated database setups and don’t use software that is either badly designed or not well understood.

                                                                                                                    You just removed 70% of software from the pool.

                                                                                                                    1. 22

                                                                                                                      Only 70%?! You’re an optimist, I see.

                                                                                                                      1. 2

                                                                                                                        That’s exactly my point though. Why does it have to be like that in first place and how can we even start talking about security and stability when that’s the case? Maybe it would make sense to thin things out. It appears that for various projects (OpenBSD) that process of reducing the “pool” seems to work, yet the thought seems to rarely occur when thinking about what a company’s stack should consist of.

                                                                                                                        At least in my experience it’s common to pull in giant pieces of software (services, libraries, etc.) without even giving it more than a minute of thought, and if there actually is that thought the reasons are on the level of “others do it”, “it’s by someone who worked at Google”, “it has a lot of stars on GitHub”, so not technical reasons and certainly not reasons telling you whether any of fitting the criteria.

                                                                                                                        1. 4

                                                                                                                          I think the reason this happens is because of how business pressures are: just shovel something out the door as fast as you can and deal with issues later. 9 times out of 10 there are no issues (or only very deeply hidden ones that surface after a long time) so you can just invoice the customer and move on to the next thing.

                                                                                                                          This kind of thinking and behaviour carries over to personal projects too. Once you’re used to working in a certain way that’s going to influence everything you do.

                                                                                                                          EDIT: Also see the article on the front page today about the GitHub copilot - shoveling code in bulk at an even faster pace!