1. 13

    This talks about the tooling investment required for a monorepo, but doesn’t address the tooling investment required for multi-repos, which I would argue is far more. Versioning dependencies between services is a hard problem, and a place like Amazon which uses multi-repos internally still hasn’t figured it out.

    1. 4

      Having worked on toolings for monorepo, i would say its fair to say that there are painpoints and tradeoffs at 2 sides of the spectrum.

      Today, the best is middle ground wgere you have a set of ‘macro repos’ separated by toolings/platform/infra needs.

      1. 3

        There are certainly pain-points for a monorepo, but they have known, good solutions (though they may require a high technical investment). I haven’t seen a good solution to the multi-repo versioning dependency problem yet. I agree that both sides have pain-points but I would argue multi-repos has a huge pain-point with no good solution, which in my opinion makes it a non-starter.

        I would be really interested to see further developments in the multi-repo world and be proven wrong, but as far as I can tell most solutions revolve around building a “meta” VCS around the various repositories to try to replicate the atomic commit behavior. At a certain point, is the only downside to a monorepo the performance? (And this is semi-limited to Git, as from my understanding Mercurial has better monorepo characteristics which Meta is investing in)

      2. 1

        Author here. Absolutely. I was working on Kubernetes during the early phases where the API client and types were being transitioned out of the monorepo to their own projects. It was a convoluted staging and sync process that made some of dependency hell problems untenable.

        There’s probably some sort of relation to Conway’s law – where you are shipping your org chart, but the underlying services are severely codependent.

      1. 3

        Thanks for the post!

        Stripe’s API has had 99.999% uptime, or three 9’s

        I believe 99.999% is typically be referred to as “five nines” in most circles. Was this a typo or a different way to frame it?

        1. 1

          Sorry typo! Too many 9’s for me. Fixed

        1. 14

          The (almost) same idea is also known under the name of “Configuration complexity clock” (by Mike Hadlow): http://mikehadlow.blogspot.com/2012/05/configuration-complexity-clock.html

          1. 4

            I thought the same thing! Thanks for reposting this article.

            Both these articles have a flavor of satire/this is insanity. One thing I mused on recently in my last two jobs is maybe we should lean in to this pattern, because a lot of the choices are sensible, and set out to establish formal or informal best practices as to when to move on from one stage to the next, and how best to manage, document, and plan the transitions.

            1. 2

              I don’t think this is insanity, although its comical that even principal engineers get stuck in this cycle without realizing it.

              To your point, I do think that we should absolutely lean into this and carve out some best practices. I might write a follow-up soon!

              1. 1

                Thanks! I wasn’t sure how to read your tongue in cheekness!

            2. 1

              Yep! Mike’s post was top of mind when I wrote this. I thought I’d update it a decade later with how its changed (albeit only slightly) today.

            1. 1

              I’m afraid I do not understand the usecase. Does it configure anything? Is it a library? Is it for drawing graphs?

              1. 1

                It’s an alternative to YAML or JSON. You can use it for your own configuration files, or as a meta-config builder that spits out YAML or JSON.

                Today, to represent non-hierarchical data in configuration files, developers mostly use templating, which can be confusing and error-prone.

                1. 1

                  Is it at the point in its development where you might show an example of translation between the two?

              1. 1

                How is this different from CUE, which also bills itself as a graph configuration language?

                1. 4

                  Good question. Cue is a lattice-based configuration language. Lattices are more strict than DAGS (and undirected graphs). No cycles are allowed in a lattice, and lattices must have a greatest lower bound and least upper bound (single root and single leaf).

                  Virgo also favors simplicity and readability over built-in functionality. If you want to do transforms with a virgo configuration file, you can operate on the graph in code.

                1. 1

                  I’m a sad kubernetes maximalist. I think Matt is basically right, I just don’t enjoy it.

                  1. 1

                    I agree. The terminology is overloaded and the developer tools use different abstractions than the production systems. I wrote an internal doc at Google called “Towards Pod-native Tooling” before podman came around, because I saw that most of our developer tools were container-centric rather than pod-centric. It never got anywhere, but I incorporated some of the ideas into the tools I was working on at the time.

                  1. 43

                    Tell me your job is operating kubernetes and you want job security without telling me that your job is operating kubernetes and you want job security.

                    1. 8

                      I find it disappointing that the top comment, with ~30 upvotes and unchallenged for several hours, is cynically questioning the author’s motives. Isn’t there already enough cynicism in the world? We should be better than that.

                      1. 6

                        It’s meant to be taken humorously. The author’s main argument is an appeal to expertise with the statement that he mixed the kool-aid. The rest of the article is based on personal opinion so there isn’t much else to say. If you have a similar experience to the author then you will agree, otherwise not.

                        1. 2

                          I don’t know, every article about kubernetes is followed by some comments about how there are some conspiracies, and how anyone pro-Kubernetes must be some shill or insecure software engineer looking to hype the industry so they can have jobs. To me this sounds more like low quality troll comment than humor. There’s nothing technical or insightful in @cadey comment.

                          1. 10

                            My comments were meant to be taken humorously. This exact comment is in the vein of a rich kind of twitter shitposting of the model “tell me x without telling me x” as a way to point out ironic or otherwise veiled points under the surface of the original poster’s arguments. I am not trying to say anything about the author as a person (English is bad at encoding this kind of intent tersely) or anything about their skill level. I guess the true insight here is something along the lines of this Upton Sinclair quote:

                            It is difficult to get a man to understand something, when his salary depends on his not understanding it.

                            I also burned out on kubernetes clusterfucks so hard I almost quit tech, so there is a level of “oh god please do not do this to yourself” chorded into my sarcastic take.

                      2. 2

                        fwiw - I am currently unemployed and working on a startup in an unrelated space. I haven’t worked on Kubernetes in 2 years.

                      1. 7

                        Yes you need a job orchestrator to abstract away your machines.

                        You should be running Hashicorp’s Nomad unless you are a Really Massive Shop With People To Spare On Operating Kubernetes.

                        In nomad I can run arbitrary jobs as well as run and orchestrate docker containers. This is something Kubernetes can’t do.

                        In nomad I can upgrade without many gymnastics. That feels good.

                        1. 13

                          Operating a simple kubernetes cluster isn’t that bad, especially with distributions such as k3s and microk8s.

                          You should be running Hashicorp’s Nomad unless you are a Really Massive Shop With People To Spare On Operating Kubernetes.

                          You should do what works for your team/company/whatever. There’s more than just Nomad and Kubernetes and people should be equipped to make decisions based on their unique situation better than someone on the internet saying they should be using their particular favorite orchestration tool.

                          1. 6

                            There’s more than just Nomad and Kubernetes

                            For serious deployment of containerised software, is there really? I did quite a bit of digging and the landscape is pretty much Nomad/Kubernetes, or small attempts at some abstraction like Rover, or Dokku for a single node. Or distros like OpenShift which are just another layer on Kubernetes.

                            I’m still sad about rkt going away…

                            1. 1

                              LXD is still around and developed!

                              1. 3

                                LXD is for system containers, not application containers. Pets, not cattle.

                                I really enjoy using LXD for that purpose, though. Feels like Solaris Zones on Linux (but to be honest, way less integrated than it should be because Linux is grown and illumosen are designed).

                            2. 2

                              Well said ngp. Whether you want to talk about Kubernetes or the class of orchestrators like it, it’s clear that there are enough tangible benefits to the developer and devops workflow that at least some of the paradigms or lessons are here to stay.

                            3. 2

                              No, I do not.

                              1. 2

                                I’ve only heard “you should run Nomad instead” in online comments, every time I hear about Nomad in person (eg from infra at Lob) it’s “we’re migrating from Nomad to X (usually Kubernetes) because Y”

                                1. 1

                                  I haven’t tried Nomad yet, even though I heard nice things about it, and it seems they like Nix. What would be Ys that others list, what are the corner cases where one should avoid Nomad?

                                  1. 1

                                    I think when a lot of work would be saved by using existing helm charts.

                                    Or when you need to integrate with specific tooling.

                                    Or when you need to go to a managed service and it’s easier to find k8s.

                                    And finally I think when you need to hire experienced admins and they all have k8s and not nomad.

                              1. 15

                                I think the key insight here is that container images (the article confuses images and containers, a common mistake that pedants like me will rush to point out) are very similar to statically linked binaries. So why Docker/container images and why not ELF or other statically linked formats?

                                I think the main answer is that container images have a native notion of a filesystem, so it’s “trivial” (relatively speaking) to put the whole user space into a single image, which means that we can package virtually the entire universe of Linux user space software with a single static format whereas that is much harder (impossible?) with ELF.

                                1. 4

                                  And we were able to do that with virtualization for at least 5 - 10 years prior Docker. Or you think that packaging also the kernel is too much?

                                  Anyways, I do not think that a container having the notion of a filesystem is the killer feature of Docker. I think that moving the deployment code (installing a library for example) close to compilation of the code helped many people and organizations who did not have the right tooling prior that. For larger companies who had systems engineers cgroups gave the security part mostly because packaging was solved decades prior to Docker.

                                  1. 1

                                    IMO it’s not the kernel but all of the supporting software that needs to be configured for VMs but which comes for ~free with container orchestration (process management, log exfiltration, monitoring, sshd, infrastructure-as-code, etc).

                                    Anyways, I do not think that a container having the notion of a filesystem is the killer feature of Docker. I think that moving the deployment code (installing a library for example) close to compilation of the code helped many people and organizations who did not have the right tooling prior that.

                                    How do you get that property without filesystem semantics? You can do that with toolchains that produce statically linked binaries, but many toolchains don’t support that and of those that do, many important projects don’t take advantage.

                                    Filesystem semantics enable almost any application to be packaged relatively easily in the same format which means orchestration tools like Kubernetes become more tenable for one’s entire stack.

                                  2. 4

                                    I can fit a jvm in a container! And then not worry about installing the right jvm in prod.

                                    I used to be a skeptic. I’ve been sold.

                                    1. 2

                                      Slightly off topic - but JVM inside a container becomes really interesting with resource limits. Who should be in charge of limits, JVM runtime or container runtime?

                                      1. 7

                                        Gotta be the container runtime (or the kernel or hypervisor above it) because the JVM heap size limit is best-effort. Bugs in memory accounting could cause the process to use memory beyond the heap limit. Absent that, native APIs (JNI) can directly call malloc and allocate off-heap.

                                        Would still make sense for the container runtime to tell the JVM & application what the limits on it currently are so it can tailor its own behaviour to try to fit inside them.

                                        1. 4

                                          It’s easy: the enclosing layer gets the limits. Who should set the resource limits? ext4 or the iron platter it’s on?

                                          1. 2

                                            What’s the enclosing layer? What happens when you have heterogenous infrastructure? Legacy applications moving to cloud? Maybe in theory it’s easy, but in practice much tougher.

                                          2. 2

                                            Increasingly the JVM is setting its own constraints to match the operating environment when “inside a container”.

                                        2. 4

                                          Yes, layers as filesystem snapshots enable a more expressive packaging solution than statically linked alternatives. But its not just filesystems, but also runtime configuration (variables through ENV, invocation through CMD) that makes the format even more expressive.

                                          p.s. I have also updated the post to say “container images”

                                          1. 3

                                            I think the abstraction on images is a bit leaky. With docker you’re basically forced to give it a name into a system registry, so that you can then run the image as a container.

                                            I would love to be able to say like… “build this image as this file, then spin up a container using this image” without the intermediate steps of tagging (why? because it allows for building workflows that don’t care about your current Docker state). I know you can just kinda namespace stuff but it really bugs me!

                                            1. 3

                                              Good practice is addressing images by their digest instead of a tag using the @ syntax. But I agree - registry has always been a weird part of the workflow.

                                              1. 1

                                                addressing images by their digest instead of a tag using the @ syntax.

                                                Be careful about that. The digest of images can change as you push/pull them between different registries. The problem may have settled out, but we were bitten by changes across different releases of software in Docker’s registry image and across the Docker registry and Artifactory’s.

                                                I’m not sure if there’s a formal standard for how that digest is calculated, but certainly used to be (~2 years back) be very unreliable.

                                                1. 1

                                                  Oh I wasn’t aware of that! That could let me at least get most of the way to what I want to do, thanks for the pointer!

                                              2. 3

                                                I noticed Go now has support for, in its essentially static binary, including a virtual filesystem instantiated from a filesystem tree specified during compilation. In that scenario, it further occurs to me that containerization isn’t perhaps necessary, thereby exposing read only shared memory pages to the OS across multiple processes running the same binary.

                                                I don’t know in the containerization model if the underlying/orchestrating OS can identify identical read only memory pages and exploit sharing.

                                                1. 2

                                                  I think in the long term containers won’t be necessary, but today there’s a whole lot of software and language ecosystems that don’t support static binaries (and especially not virtual filesystems) at all and there’s a lot of value in having a common package type that all kinds of tooling can work with.

                                                  1. 2

                                                    As a packaging mechanism, in theory embedded files in Go works ok (follows single process pattern). In practice, most Go binary container images are empty (FROM scratch + certs) anyways. Lots of files that are environment dependent that you would want at runtime (secrets, environment variables, networking) that are much easier to declaratively add to a container image vs. recompile.

                                                  2. 2

                                                    So why Docker/container images and why not ELF or other statically linked formats?

                                                    There are things like gVisor and binctr that work this way, as do somethings like Emscripten (for JS/WASM)

                                                    1. 2

                                                      I really hope for WASI to pick up here. I used to be a big fan of CloudABI, which now links to WASI.

                                                      It would be nice if we could get rid of all the container (well actually mostly Docker) cruft.

                                                  1. 4

                                                    More like “docker is a packager”

                                                    1. 3

                                                      Now that docker (buildkit) does instruction caching, garbage collection, and concurrent dependency resolution, its looking a lot less like just a packager!

                                                    1. 8

                                                      I work with Kubernetes all day so I’m a little biased towards containers. But for new projects, I’ve been using something similar to Tim Hockin’s go-build-template [0] for dockerized golang builds. Its great for managing dependencies and having reproducible builds, but comes in really handy when you’re trying to cross compile binaries that need CGO and architecture specific C headers for linux-amd64 and darwin-amd64 [1].

                                                      I also use the nvidia-docker plugin[2] for GPU passthrough and to bring up a TensorFlow environment locally.

                                                      [0] https://github.com/thockin/go-build-template

                                                      [1] https://github.com/karalabe/xgo

                                                      [2] https://github.com/NVIDIA/nvidia-docker

                                                      1. 2

                                                        As mentioned last week, I took part in Ludum Dare this weekend. You can see/play my daft entry here: Bathroom Break Manager

                                                        Had an absolute blast, and did not completely kill myself. Got a full 8hrs of sleep on of the nights. Definitely the most complete game I’ve submitted out of my three attempts. Spent a reasonable amount of time laughing childishly at variable and function names relating to toilet activity.

                                                        This week I will be attempting to get enough work done that I can happily relax for the holiday period. This mostly means tidying up a kubernetes setup and helping team members get set up and running with it.

                                                        1. 3

                                                          If you have some spare time or particular pain points, please feel free to PM me with any feedback around getting devs set up with kubernetes. I work on local kubernetes developer velocity.

                                                          [0] https://lobste.rs/s/d4ben6/what_are_you_working_on_this_week/comments/qpcive#c_qpcive

                                                          1. 3

                                                            Nice! Thank you, I will definitely ping you if I have questions / come across a problem etc. I’ve been using minikube extensively to get setup and build out our developer workflow, and it’s been a treat to work with, so thanks for all your work

                                                        1. 1

                                                          Congrats to the team on shipping v1.5.0. tldr;

                                                          • StatefulSets and PodDisruptionBudget are moving to beta for stateful apps

                                                          • Support for windows containers!

                                                          • kubefed alpha command for managing federated clusters

                                                          1. 1

                                                            I’m working on moving minikube [0] to use a custom buildroot [1] ISO. Minikube runs a local kubernetes cluster on your laptop. We use a lightweight VM to run docker on windows or mac, but also run all the kubernetes components as goroutines. Since its cross platform, we need to make sure that all our integration tests pass with all the VM drivers (xhyve, kvm, hyperv). Right now there’s only one remaining blocking issue with the KVM driver I’m trying to debug.

                                                            [0] https://github.com/kubernetes/minikube [1] https://buildroot.org/