1. 57
  1.  

  2. 10

    Always love to see tools in this area. Can you give a more detailed overview of the goals of the project, and how it differs from Bazel or Nix, or how it will differ when finished?

    1. 15

      Goals are basically:

      • Take lessons from Rust and Go tooling and bring them to build systems. Automatic documentation generation, tests in docs, good ergonomics and documentation, incredible error messages, etc..
      • Packages are a community effort like language libraries, no core package tree. This creates security concerns, but I think assuming packages are hostile is better for the long-term security of the system.
      • Limit as much as possible. No network access (where possible), no arbitrary build inputs, no complex command line flags, be very cautious with feature additions, get to API stability quickly. Strong limitations have helped with some of the projects features and I also want to take the “runs on my computer” problem very seriously.

      Not very different from Bazel. Should enable similar workflows (once mature tooling is built for all languages :| ), just with more sharing, better cache availability. Bazel is hard to use because you have to do a lot of explicit dependency definition, I’m excited about some ways to make that easier with Bramble, but we’ll see if they pan out.

      The builds work like Nix, but it’s not an OS, and it’s not many of the other things Nix is. I hope to solve some of the pain points of Nix, specifically around approachability and documentation.

      Nix also expects libraries to do a lot of code generation to support language dependencies. Like go2nix, cargo2nix, etc… I think first-class support for this workflow is very important, and don’t want to leave it to third party libs to generate Bramble code. Will be trying to support Go and Go modules first to see if I can arrive at patterns that are a little more user friendly. Basically I want the user interface to be this:

      def build():
          return go_build(files["./"])
      

      And then you go.mod and go.sum are parsed under the hood, their resolution is persisted without code generation, all while maintaining reproducibility guarantees. There is work to be done here, but I think I have a rough plan. You’re welcome to check out these notes for more on that strategy, although my notes are usually quite messy and hard to follow.

      1. 3

        Thanks! Sounds like a cool project!

        How do you expect to see better sharing and better cache availability? I don’t properly understand how Bazel structures this, but I would expect cache sharing to be quite high, given the stress on the system inside Google.

        1. 2

          Bazel workspaces are pretty porous and tend to source a bunch of dependencies from the system. It’s actually quite hard to get a good coverage across machines.

          1. 2

            A few ways:

            1. There will be a public build cache and public build architecture. Bramble is designed to be safe in ways that I think will make running a publicly available build server much easier.
            2. Very limited build inputs, if you’re building something public there’s a higher chance that you’re building it in the same way as the cached build.
            3. The sharing piece is just being hopeful about the package ecosystem. In my experience with Bazel you’re mostly working with the built-in builders, with Bramble I would hope a given project would be composing together publicly maintained build tools across hundreds of repositories. I just hope that the ecosystem is more diverse/vibrant, but that’ll take some doing.
            1. 2

              Ah, that sounds fantastic! If you can solve the security issues around the public build cache I can see that being a huge advancement. Honestly having the shared build cache is why Bazel works so well at Google, but naturally that doesn’t help anyone else.

      2. 3

        Looks interesting, thanks!

        Very limited build inputs. No env-var, arguments, or other inputs allowed for build configuration. Almost all configuration must be done in-code.

        “Almost”, what’s some of the remaining configuration which can (or must be) done out side of code? command line flags?

        1. 3

          The only planned command line flag at the moment is --target which will allow someone to specify the target architecture/os to build for. Bramble gets a lot of benefit from being that locked down, like it’s easy to just “build everything in the project” since there are no arbitrary configuration values to consider, but I wanted to be a little cautious in the post because Bramble is immature and I’m not sure what legitimate use-cases I’m preventing by having such little build input.

        2. 3

          This is super cool! I spent a few months building something similar (in Go, using Starlark). I would get hung up on bootstrapping various low-level dependencies and eventually gave up. I’m excited to check this out and maybe contribute. :)

          1. 3

            No way! Yeah, navigating bootstrapping dependencies was (and continues to be) very tough. I would stop working on the project at times solely because of how intimidated I was about getting some random build working. Imagine it will get easier, but I feel you. And yes please check it out and contribute :)

          2. 3

            Very nice! I’m a NixOS and Guix user/contributor, but recognize they both have approachability/learning curve issues.

            I think Skylark is a great choice: its similarity to Python will make it much more approachable.

            Would using Bazel directly have been possible? That is, have a Bazel rule that returns a derivation. This would have benefitted from the existing Bazel features:

            • remote caching
            • external dependencies
            • sandboxing
            • Bazel’s build rules, e.g. https://github.com/bazelbuild/rules_go . I hear Nix plans to support source-file level building, and this would allow you to jump to that in one go.

            Perhaps there’s some downsides I’m missing?

            1. 1

              Interesting, possibly. I did want this use case “bramble run github.com/maxmcd/busybox ash”. Running arbitrary programs from remote sources. Not sure if I could have gotten that with Bazel, or how I would have made it work.

              I will say that to my understanding (which could be wrong) in Bazel:

              • Builders can touch anything in the operating system.
              • Dynamic dependencies are not supported (more here and elsewhere).

              I wanted Bramble to be able to support custom builders without having to trust them to not mess with things on the host machine. I don’t know a ton about Bazel custom builders, but I think they require some level of trust. When similar language-specific builders are added to Bramble they will be able to define custom build logic without having access to the host machine. This was part of the goal, lower barrier to entry to contribute custom builders, both in terms of complexity and trust. Would be interested in your thoughts on this one, might have it wrong.

              I’d also really like to support dynamic dependencies with Bramble, some references to that here. This work is ongoing, but I think there’s a path.

              On a more honest note, I wanted to build a thing, and didn’t spend as much time auditing nix/guix/bazel, so could have missed a more harmonious opportunity. I do also wonder if building a Starlark frontend for Nix was something else I should have explored.

              1. 2

                I will say that to my understanding (which could be wrong) in Bazel:

                • Builders can touch anything in the operating system.
                • Dynamic dependencies are not supported (more here and elsewhere).

                https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf compares build systems and finds (see Table 1) that Bazel internally supports dynamic dependencies:

                ``As of writing, it is not possible to express dynamic dependencies in user-defined build rules; however some of the pre-defined build rules require dynamic dependencies and the internal build engine can cope with them''
                

                Though as you point out, the Bazel developers don’t think that’s possible: https://github.com/bazelbuild/bazel/issues/2362#issuecomment-273508287.

                I wanted Bramble to be able to support custom builders without having to trust them to not mess with things on the host machine. I don’t know a ton about Bazel custom builders, but I think they require some level of trust. When similar language-specific builders are added to Bramble they will be able to define custom build logic without having access to the host machine. This was part of the goal, lower barrier to entry to contribute custom builders, both in terms of complexity and trust. Would be interested in your thoughts on this one, might have it wrong.

                I don’t know much about Bazel’s sandboxing internals.

                On a more honest note, I wanted to build a thing, and didn’t spend as much time auditing nix/guix/bazel, so could have missed a more harmonious opportunity. I do also wonder if building a Starlark frontend for Nix was something else I should have explored.

                That’s fair enough. It’s great that you’ve created this and shared it. The ecosystem benefits from different approaches.

            2. 2

              How does this compare with shake?

              1. 2

                I think starlark is very good pragmatic choice which might drive adoption.

                My dream would be a statically typed functional programming language for these kinds of things, though. Similarly in simplicity to Elm with nice auto-completiom. Maybe lazy.

                1. 2

                  Thank you for contributing isolated package management technology.

                  Could direnv invoke Bramble? I heavily rely on direnv’s ability to use nix and incant nix-shell automatically.

                  1. 1

                    It’s a little awkward to run programs at the moment. I think you could put a “bramble.toml” in your home directory to define dependencies for all sub-folders and then alias various invocations of “bramble run” to do what you’re saying. However the “bramble run” functionality is very new and there are some limitations.

                    This is definitely a workflow I hope to support, but a few things need to be worked out.

                    I think that would eventually work very well though, since you’d be able to define the sanbox rules for every program and only give things access to what they need to run.

                  2. 2

                    I come from a more traditional understanding of the term “build system” (i.e., a system that direct compilation and linking of software) and I can’t understand from the article what exactly is Bramble building. For example, there is this sentence:

                    Here Bramble is building the necessary dependencies to run bash.

                    Does this mean that Bramble compiles and links bash and all its dependencies from C source?

                    1. 2

                      For the moment, and for convenience, that demo is just downloading an archive and using patchelf to link it to a specific libc. In the near future it should do exactly what you’re saying, all of the derivations/executables in github.com/maxmcd/bramble will build from source. The intention is get rough feature parity with something like Bazel or Nix and have good tooling to compile various programs in various languages from source.

                      That being said, since Bramble is just a system and users can define whatever build rules they want, there’s nothing stopping other users from running builds that are really just archive downloads.

                    2. 1

                      Any plans to support any of the BSDs? Specifically, FreeBSD?

                      1. 2

                        Yep! That’s next after macOS, although not sure how/if I can get rootless to work on FreeBSD/BSDs.

                      2. 1

                        Hi maxm, I got pointed to this from the Racket community. I’d love to exchange research with you because I’m working on a project that has a lot of overlap with Bramble. It’s called Denxi. It’s already capable of replicating functionality in Guix, such that the end-user is entirely responsible for the manufacture of entire functional package managers, dumb download clients, or arbitrary data workflows on their systems.

                        I’m now working on a bootstrap path through GNU Mes to the Racket CS VM in particular. You can find the current code on GitHub under zyrolasting/denxi

                        This isn’t meant to be a self-plug, but as an invitation to collaborate. If you have any questions about solving certain data distribution problems while you work on Bramble, please reach out. I want to see more tools like this in the space, because I believe they are better for end-user freedoms. Hopefully I can adapt some of my code to save you some time if our licenses are compatible.

                        1. 1

                          Very interesting, thanks for reaching out. Emailing you now.

                        2. 1

                          Anyone have any ideas why this essentially-static site is making Firefox run at about 15 fps? I poked around in the inspector a little but the site doesn’t actually seem to be doing anything insane. Firefox 94.0 for reference.

                          1. 1

                            SVG animation most likely. The image subtly sparkles. Might be more of a CPU hit than I expected :/

                            1. 1

                              Hmmm, blocking the header image gets it to 20 fps but performance is still weirdly slow. It’s most noticeable when swapping tabs. ¯\(ツ)