Threads for stravid

  1. 4

    I’m reluctant to use anything apart from the original, since I cannot predict in what hot waters I’m getting myself into in the future. And I’m saying that as a Go beginner that failed to get a static build with SQLite and the ICU extension. After a 14-hour struggle, I gave up and switched the deployment to a dynamic executable.

    1. 2

      Putting all your queries behind one interface will bite you once you leave “very small app” territory. You will have to stub the whole implementation for each test, and if you change some query/add some functionality, you will always have to touch code in two places. I’d argue that’s already the case here.

      1. 2

        Somebody wrote a blog post about the same issue you are pointing out: https://jrock.us/posts/go-interfaces/ As someone learning Go right now I found that very educational.

      1. 2

        The real question here is, I feel, why did the writer this not do it on their own and never bothered anyone?

        Like, not calling her lazy or anything, but why wasn’t there an incentive for him to just get it done on his own? Since it’s obviously possible.

        Indeed, getting anything “simple” to the level of “management handled issue” will by default add a standard amount of overhead (e.g. 10 mins in a startup, 1hr in round B, few weeks in a megacorp). But that’s just the nature of consensus making in large orgs… however a lot of things can be resolved exactly by not having to get them to management, by just doing them yourself (or yourself + a group of friends that don’t mind asking for help)

        1. 10

          First, I believe the “guy” writing at the “Rachel by the Bay” blog is a woman.

          Second, they did write an initial dashboard by themselves, but then a “dashboard team” came along and insisted they’ll provide said dashboard—and then promptly decided they had more important things to do for 5 months. Here’s paragraph 3 and 4 of the post:

          January 1: we put up a terrible hack: a shell script that runs a Python script that talks to the service to get the status and then dumps out raw HTML to a file in someone’s public_html path.

          January 29, early: there’s this team that nominally owns dashboards, and they got wind of us wanting a dashboard. They want to be the ones to do it, so we meet with them to convey the request. We make a mockup of the list and the eventual big red button to give them some idea of what it should look like.

          1. 3

            First, I believe the “guy” writing at the “Rachel by the Bay” blog is a woman.

            Ups, edited

            Second, they did write an initial dashboard by themselves, but then a “dashboard team” came along and insisted they’ll provide said dashboard—and then promptly decided they had more important things to do for 5 months. Here’s paragraph 3 and 4 of the post:

            What I was saying here was that, i.e. “WHY IS THERE a dashboard team!?” that seems like the fundamental question to tackle here. Why does the company not provide incentives to just leave the stupid python script up, or get the devs to spend the few days they did writing the dashboard without implicating a whole other department.

            I think the author might also think the same thing here, I was trying to paraphrase my takeaway, not criticize.

            1. 12

              “WHY IS THERE a dashboard team!?” that seems like the fundamental question to tackle here.

              I can answer this question as if it was the BigCorp for which I’m currently working. Please take my answer with a grain of salt, as this is most likely biased by my “grunt” point of view.

              1. Up high, the CEO most likely decided they wanted to reduce cost on the technology.
              2. They gave this task to a Senior VP. This senior VP discussed with a VP and a few directors, and they came up with one of the conclusions that the technology is too fragmented across the organisation, and too many people solve the same problems.
              3. The task got split up into smaller chunk “consolidate server technology”, “consolidate dashboard technology”, etc. At some point, some director got assigned the objective to “consolidate the dashboard technology”
              4. This director within their budget created a “dashboard team”
              5. Now you have a team which sole purpose is consolidate the dashboards, their performance is measured every quarter in terms of: how many dashboard were consolidated, and how many new dashboards are created by this consolidation team.
              6. This team now hunts down anybody who is using a dashboard or creating a new one, the team existence depends on it. (If they don’t perform well they will get reassigned, and members will loose the opportunity to be promoted quickly, and/or face the possibility to made redundant.)
          2. 5

            Her name is Rachel and she is not a guy. If you want to use gendered language you should do your research on not jump to conclusions. The name is even visible in the domain.

            1. 2

              Gendering someone based on a personal name that you found in the domain name of the site where you found an article sounds like a lot of making assumptions to me. And in the glorious twenty-first century, anyone and everyone is free to be a “guy”, just like in the 90s we were all dudes.

          1. 14

            What’s going on here? How did this get to the top of lobste.rs with 26 upvotes? I’m happy for the OP that they could get their system to work, but as far as I can tell, the story here is “package manager used to manage packages.” We have been doing that for decades. Is there any way the community can get a lever to push back on thin stories like this one?

            1. 25

              Would it change your opinion if the article mentioned that the nix shell being used here is entirely disposable and this process leaves no mark in your OS setup? Also that even if this required some obscure versions of common system dependencies you could drop into such a shell without worrying about version conflicts or messing up your conventional package manager?

              I agree that the article is thin in content, but I don’t think you can write this story off as “package manager used to manage packages.” , I think nix shell is very magical in the package management world.

              1. 6

                I could do that with docker too and it would not leave a trace either

                1. 17

                  Yes, but then you’d be inside a container, so you’d have to deal with the complexities of that, like mounting drives, routing network traffic etc. With nix shell, you’re not really isolated, you’re just inside a shell session that has the necessary environment variables that provide just the packages you’ve asked for.

                  Aside from the isolation, the nix shell is also much more composable. It can drop you into a shell that simultaneously has a strange Java, python and Erlang environment all compiled with your personal fork of GCC, and you’d just have to specify your GCC as an override for that to happen.

                  1. 4

                    I get that, but I have to go through the learning curve of nix-shell, while I already know docker, since I need it for my job anyway. I am saying that there are more ways to achieve what the article is talking about. It is fine that the author is happy with their choice of tools, but it is very unremarkable for the title and given how many upvotes that article got.

                    1. 5

                      Why not learn nix and then use it at work as well :) Nix knows how to package up a nix-defined environment into a docker container and produce very small images, and you don’t even need docker itself to do that. That’s what we do at work. I’m happy because as far as I’m concerned Nix is all there is and the DevOps folks are also happy because they get their docker images.

                      1. 3

                        I work in a humongous company where the tools and things are less free to choose from atm, so even if I learned nix, it would be a very tough sell..

                  2. 3

                    As someone who hasn’t used Docker, it would be nice to see what that looks like. I’m curious how the two approaches compare.

                    1. 6

                      I think that the key takeaway is that with Docker, you’re actually running a container will a full-blown OS inside. I have a bias against it, which is basically just my opinion, so take it with a grain of salt.

                      I think that once the way to solve the problem of I need to run some specific version of X becomes let’s just virtualize a whole computer and OS because dependency handling is broken anyway, we, as a category simply gave up. It is side-stepping the problem.

                      Now, the approach with Nix is much more elegant. You have fully reproducible dependency graphs, and with nix-shell you can drop yourself in an environment that is suitable for whatever you need to run regardless of dependency conflicts. It is quite neat, and those shells are disposable. You’re not running in a container, you’re not virtualizing the OS, you’re just loading a different dependency graph in your context.

                      See, I don’t use Nix at all because I don’t have these needs, but I played with it and was impressed. I dislike our current approach of just run a container, it feels clunky to me. I think Docker has it’s place, specially in DevOps and stuff, but using it to solve the I need to run Python 2.x and stuff conflicts with my Python 3.x install is not the way I’d like to see our ecosystem going.


                      In the end, from a very high-level, almost stratospheric, point-of-view: both docker and nix-shell workflow will be the developer typing some commands on the terminal, and having what they need running. So from a mechanical standpoint of needing to run something, they’ll both solve the problem. I just don’t like how solving things by doing the evergreen is now the preferred solution.

                      Just be aware that this is an opinion from someone heavily biased against containers. You should play with both of them and decide for yourself.

                      1. 3

                        This comment is a very good description of why I’ve never tried Docker (and – full disclosure – use Nix for things like this).

                        But what I’m really asking – although I didn’t make this explicit – is a comparison of the ergonomics. The original post shows the shell.nix file that does this (although as I point out in another comment, there’s a shell one-liner that gets you the same thing). Is there an equivalent Dockerfile?

                        I was surprised to see Docker brought up at all because my (uninformed) assumption is that making a Docker image would be prohibitively slow or difficult for a one-off like this. I assumed it would be clunky to start a VM just to run a single script with a couple dependencies. But the fact that that was offered as an alternative to nix-shell makes me think that I’m wrong, and that Docker might be appropriate for more ad-hoc things than I expected, which makes me curious what that looks like. It points out a gap in my understanding that I’d like to fill… with as little exertion of effort as possible. :)

                        1. 4

                          But the fact that that was offered as an alternative to nix-shell makes me think that I’m wrong, and that Docker might be appropriate for more ad-hoc things than I expected, which makes me curious what that looks like. It points out a gap in my understanding that I’d like to fill… with as little exertion of effort as possible. :)

                          I think containers is a perfectly capable solution to this. The closest thing you can use would probably be toolbox.

                          https://github.com/containers/toolbox

                          It would allow you to even provide a standardized environment which would be decoupled from the deployment itself (if that makes sense). It also mount $HOME as well.

                          1. 3

                            I use Nix, but also have experience with Toolbox.

                            I would recommend most people to use Toolbox over nix-shell. With toolbox you can create one-off containers in literally seconds (it’s two commands). After entering the container you can just dnf install whatever you need. Your home directory gets mounted, so you do not have to juggle with volumes, etc. If you need to create the same environment more often, you can create a Dockerfile and build your toolbox containers with podman. The upstream containers that Fedora provides are also just built using Dockerfiles.

                            The post shows a simple use case, but if you want to do something less trivial, it often entails learning Nix the language and nixpkgs (and all its functions, idioms, etc.). And the Nix learning curve is steep (though it is much simpler if you are familiar with functional programming). This makes the toolbox approach orders of magnitude easier for most people - you basically need to know toolbox create and toolbox enter and you can use all the knowledge that you already have.

                            However, a very large shortcoming of toolbox/Dockerfiles/etc. is reproducibility. Sure, you can pass around an image and someone else will have the same environment. But Nix allows you to pin all dependencies plus the derivations (e.g. as a git SHA). You can give someone your Nix flake and they will have exactly the same dependency graph and build environment guaranteed.

                            Another difference is that once you know Nix, it is immensely powerful for defining packages. Nix is a turing-complete functional language, so nixpkgs can provide a lot of powerful abstractions. I dread every time I have to create/modify and RPM spec file, because it is so primitive compared to making a Nix derivation.

                            tl;dr: most people will want to use something like Toolbox, it is familiar and provides many of the same benefits as e.g. nix-shell (isolated, throw-away environments, with your home directory available). However, if you want strong reproduciblity across systems and a more powerful packaging/configuration language, learning Nix is worth it.

                          2. 3

                            A cool aspect of Docker is that it has a gazillion images already built and available for it. So depending on what you need, you’ll find a ready-made image you can put to good use with a single command. If there are no images that fill your exact need, then you’ll probably find an image that is close enough and can be customised. You don’t need to create images from scratch. You can remix what is already available. In terms of ergonomics, it is friendly and easy to use (for these simple cases).

                            So, NixPkgs have a steeper learning curve in comparison to dockerfiles. It might be simpler to just run Docker. What I don’t like is what is happening inside Docker, and how the solution for what looks like simple problems involves running a whole OS.

                            I’m aware that you can have containers without an OS like described in this thread, but that is not something I often see people using in the wild.

                          3. 1

                            Nit-pick: AFAIK one doesn’t really need Alpine or any other distro inside the container. It’s “merely” for convenience. AFAICT it’s entirely possible to e.g. run a Go application in a container without any distro. See e.g. https://www.cloudbees.com/blog/building-minimal-docker-containers-for-go-applications

                      2. 3

                        Let’s assume nix shell is actual magic — like sourcerer level, wave my hand and airplanes become dragons (or vice versa) magic — well this article just demonstrated that immense power by pulling a coin out of a deeply uncomfortable kid’s ear while pulling on her nose.

                        I can’t speak for the previous comment’s author, but those extra details, or indeed any meat on the bones, would definitely help justify this article’s otherwise nonsensical ranking.

                        1. 2

                          Yeah, I agree with your assessment. This article could just as well have the title “MacOS is so fragile, I consider this simple thing to be an issue”. The trouble with demonstrating nix shell’s power is that for all the common cases, you have a variety of ad-hoc solutions. And the truly complex cases appear contrived out of context (see my other comment, which you may or may not consider to be turning airplanes into dragons).

                      3. 19

                        nix is not the first thing most devs would think of when faced with that particular problem, so it’s interesting to see reasons to add it to your toolbox.

                        1. 9

                          Good, as it is not supposed to be the first thing. Learning a fringe system with a new syntax just to do something trivial is not supposed to be the first thing at all.

                        2. 4

                          I find it also baffling that this story has more upvotes than the excellent and original code visualization article currently also very high. Probably some nix up vote ring pushing this

                          1. 12

                            Or folks just like Nix I guess? 🤷

                            1. 11

                              Nix is cool and people like it.

                              1. 5

                                I didn’t think this article was amazing, but I found it more interesting than the code visualization one, which lost me at the first, “From this picture, you can immediately see that X,” and I had to search around the picture for longer than it would have taken me to construct a find command to find the X it was talking about.

                                This article, at least, caused me to say, “Oh, that’s kind of neat, wouldn’t have thought of using that.”

                              2. 6

                                This article is useless. It is way simpler (and the python way) to just create a 2.7 virtualenv and run “pip install psycopg2 graphwiz”. No need to write a nix file, and then write a blog post to convince yourself you didn’t waste your time!

                                Considering all nix posts get upvoted regardless of content, it’s about time we have a “nix” tag added to the site.

                                1. 14

                                  This article is not useless just because you don’t see its value.

                                  I work mainly with Ruby and have to deal with old projects. There are multiple instances where the Ruby way (using a Ruby version manager) did not work because it was unable to install an old Ruby version or gem on my new development machine. Using a nix-shell did the job every time.

                                  just create a 2.7 virtualenv and run “pip install psycopg2 graphwiz”

                                  What do you do if this fails due to some obscure dependency problem?

                                  1. 4

                                    What do you do if this fails due to some obscure dependency problem?

                                    Arguably you solve it by pinning dependency versions in the pip install invocation or requirements.txt, as any Python developer not already using Nix would do.

                                    This article is not useless just because you don’t see its value.

                                    No, but it is fairly useless because it doesn’t do anything to establish that value, except to the choir.

                                    1. 2

                                      In my experience there will be a point where your dependencies will fail due to mismatching OpenSSL, glibc versions and so on. No amount of pinning dependencies will protect you against that. The only way out is to update dependencies and the version of your language. But that would just detract from your goal of getting an old project to run or is straight up impossible.

                                      Enter Nix: You pin the entire environment in which your program will run. In addition you don’t pollute your development machine with different versions of libraries.

                                      1. 3

                                        Arguably that’s just shifting the burden of effort based on a value judgement. If your goal is to get an old project to run while emphasizing the value of incurring zero effort in updating it, then obviously Nix is a solution for you and you’ll instead put the effort into pinning its entire runtime environment. If, however, your value to emphasize is getting the project to run then it may well be a more fruitful choice to put the effort into updating the project.

                                        The article doesn’t talk about any of the hairier details you’re speaking to, it just shows someone taking a slightly out of date Python project and not wanting to put any personal effort into updating it… but updating it by writing a (in this case relatively trivial) Python 3 version and making that publicly available to others would arguably be the “better” solution, at least in terms of the value of contributing back to the community whose work you’re using.

                                        But ultimately my argument isn’t with the idea that Nix is a good solution to a specific problem, it’s that this particular article doesn’t really make that point and certainly doesn’t convincingly demonstrate the value of adding another complex bit of tooling to the toolkit. All the points you’ve raised would certainly help make that argument, but they’re not sadly not present in this particular article.

                                    2. 1

                                      Just out of curiosity, I’m also dealing with ancient ruby versions and use nix at work but I couldn’t figure out how to get old enough versions, is there something that helps with that?

                                        1. 1

                                          Thank you, very helpful!

                                          1. 1

                                            Do note this method will get you a ruby linked to dependencies from the same checkout. In many cases this is what you want.

                                            If instead you want an older ruby but linked to newer libraries (eg, OpenSSL) there’s a few extra steps, but this is a great jumping off point to finding derivations to fork.

                                            1. 1

                                              Do note this method will get you a ruby linked to dependencies from the same checkout. In many cases this is what you want.

                                              Plus glibc, OpenSSL and other dependencies with many known vulnerabilities. This is fine for local stuff, but definitely not something you’d want to do for anything that is publicly visible.

                                              Also, note that mixing different nixpkgs versions does not work when an application uses OpenGL, Vulkan, or any GPU-related drivers/libraries. The graphics stack is global state in Nix/NixOS and mixing software with different glibc versions quickly goes awry.

                                        2. 2

                                          This comment mentions having done something similar with older versions by checking out an older version of the nixpkgs repo that had the version of the language that they needed.

                                          1. 2

                                            Like others already said you can just pin nixpkgs. Sometimes there is more work involved. For example this is the current shell.nix for a Ruby on Rails project that wasn’t touched for 5 years. I’m in the process of setting up a reproducible development environment to get development going again. As you can see I have to jump through hoops to get Nokogiri play nicely.

                                            There is also a German blog post with shell.nix examples in case you need inspiration.

                                        3. 4

                                          this example, perhaps. I recently contributed to a python 2 code base and running it locally was very difficult due to c library dependencies. The best I could do at the time was a Dockerfile (which I contributed with my changes) to encapsulate the environment. However, even with the container standpoint, fetching dependencies is still just as nebulous as “just apt install xyz.” Changes to the base image, an ambiently available dependency or simply turning off the distro package manager services for unsupported versions will break the container build. In the nix case, it is sort of forced on the user to spell it out completely what the code needs, combine with flakes and I have a lockfile not only for my python dependencies, but effectively the entire shell environment.

                                          More concretely, at work, the powers to be wanted to deploy python to an old armv7 SoC running on a device. Some of the python code requires c dependencies like openssl, protobuf runtime and other things and it was hard to cross compile this for the target. Yes, for development it works as you describe, you just use a venv, pip install (pipenv, poetry, or whatever as well) and everything is peachy. then comes to deployment:

                                          1. First you need to make a cross-compiled python interpreter, which involves first building the interpreter for your host triple then rebuilding the same source for the target host triple making sure to tell the build process where the host triple build is. This also ignores that some important python interpreter things may not build, like ctypes.
                                          2. Learn every environment variable you need to expose to the setup.py or the n-teenth build / packaging solution for the python project you want to deploy, hope it generates a wheel. We will conveniently ignore how every C depending package may use cmake, or make, or meson, etc, etc…
                                          3. make the wheels available to the image you actually ship.

                                          I was able to crap out a proof-of-concept in a small nix expression that made a shell that ran the python interpreter I wanted with the python dependencies needed on both the host and the target and didn’t even have to think. Nixpkgs even gives you cross compiling capabilities.

                                          1. 1

                                            Your suggested plan is two years out of date, because CPython 2.7 is officially past its end of life and Python 2 packages are generally no longer supported by upstream developers. This is the power of Nix: Old software continues to be available, as if bitrot were extremely delayed.

                                            1. 3

                                              CPython 2.7 is available in debian stable (even testing and sid!), centos and rhel. Even on MacOS it is still the default python, that ships witht he system. I don’t know why you think it is no longer available in any distro other than nix.

                                        1. 3

                                          Finishing a compact TUI task manager I used to start learning Rust. After that I will try to get some feedback on my code. I stuck to cargo clippys feedback but I can learn a lot more by specific suggestions from better Rust programmers. Even without feedback I’m immensely satisfied. Learning Rust was this big daunting task to me. Now I managed to build a working application in a few hours. Really looking forward to dive deeper.

                                          On Saturday I participate in a 24 hours / 400 km cycling race so I will definitely pack and prepare my gear in the evenings.

                                          1. 3

                                            Rust is pretty fun learning! Once the borrowchecker clicks for you, it becomes really pleasant to write.

                                            Are you using any crates in your project? I’m curious on how you interact with the tasks. Are they always single line and you edit them in place? Do you open an external editor? Did you implement and editor withing the application?

                                            1. 1

                                              Thanks for the encouragement. I also realized how little I know about memory management. String and str made that obvious very quickly.

                                              I’m using three crates: termion, uuid and sqlite. There are three screens. Task list, adding a new task and editing an existing task. The input screens are only static text with a single line for the input and cursor. I implemented an editor in the sense that the application handles any character key, backspace and return to finish.

                                              The task list shows the four states (todo, done, in progress, discarded) as color and symbol in front of every task. You can select a specific task with the arrow keys. There are hotkeys to change a tasks state, edit it or add a new task to the end of the list.

                                              It’s really basic but was great for starting to learn Rust. I have a hard time with theory or toy examples. Instead I tend to build real things when learning something new. Sometimes this is a challenge but with Rust it turned out easier than anticipated to get going. The compiler is fantastic, I rarely had to use the internet, better than I imagined.

                                              1. 1

                                                Cool! Learning these rust stuff also made me a better programmer. I’d love to see some screenshots/gifs of it in action!

                                                One “trick” I use when developing in rust and need info on, say, a stdlib struct, “!rust ” will search rust docs (if you have duckduckgo as your default search engine). I always take a look at the available methods (and examples) to see if there’s anything that already solves the problem I have (usually there is).

                                                cheers

                                          1. 2

                                            Talking to someone looking for a summer internship at my company.

                                            Finishing a web application for selling art via Stripe. Having a state machine for the entire process is liberating. You just look at the graphic and see what edge case is not yet implemented. Rinse and repeat.

                                            1. 1

                                              Cool, is it available to see in action?

                                              1. 2

                                                Yes it is: https://www.nowornever.gallery/

                                                Sorry for the late reply.

                                            1. 1

                                              Event sourcing is such an intriguing pattern that has deceptively sharp edges. Ive heard the “you get an audit log for free” a few times but the event store ends up being a very poor audit log and querying the raw events is usually hard/cumbersome or impossible depending on how you store them. My audit logs always end up being yet another projection.

                                              Ive found that event sourcing only gets nasty when you inevitably have a breaking change to an event contract and have to upgrade/downgrade events for different projections/handlers.

                                              I feel like its a powerful pattern even on the small scale but you need a disciplined set of developers and a strong ops strategy to make it work

                                              1. 4

                                                event sourcing only gets nasty when you inevitably have a breaking change to an event contract and have to upgrade/downgrade events for different projections/handlers

                                                Our solution to this was a robust capability model. Capabilities limit access to resources, but a change in a capability is itself an event. So at the point when a contract changes, that change is itself modeled in the event log, and hence only affects events that occur after it.

                                                1. 2

                                                  Our solution to this was a robust capability model.

                                                  That sounds really interesting! Is there something I can read, or can you say a bit more about that?

                                                2. 1

                                                  The article mentions being able to completely rebuild the application state. It makes sense in theory, but how does it work out in practice?

                                                  I imagine that you might have to do event compaction over time or else the event storage would be massive. Seems like an area where those sharp edges might come out.

                                                  1. 3

                                                    A lot of folks end up using snapshots ever n events or something discarding the previous events. Its an optimization that becomes fairly essential quickly

                                                    1. 2

                                                      I run a darts scorer in my spare time. The state of every single game is only stored as an event stream. Account and profile data uses a conventinal relational model.

                                                      I never rebuild the entire application state, only the game I’m interested at. Should I introduce new overall statistics I would just rebuild game after game to backfill the statistics data.

                                                      Storage and volume hasn’t been a problem on my tiny VPS. PostgreSQL sits at 300.000 games with each one having about 300 events. If you want I can lookup the exact numbers in the evening.

                                                  1. 8
                                                    1. Query the database to find all dormant accounts with a balance, which haven’t been charged the fee this month.

                                                    That’s still not idempotent: the worker could crash between charging the fee and marking that it charged the fee.

                                                    (It sounds like in this case they’re moving the money between internal accounts, so could do it all in a transaction, which would make it idempotent. It’s not entirely clear from the context though.)

                                                    1. 13

                                                      I want to add that a transaction is not enough. Its isolation level needs to be set to “serializable” otherwise you still end up with a race condition. Alternatively you can use a different optimistic concurrency mechanism to prevent errors.

                                                      The race condition looks like this. Assuming query and calculation take a duration of X you have a window of X where another process would see and do the exact same thing since the write of the first one did not yet go through. Therefore you need to enforce your idempotency when writing.

                                                      I admit the time window X is tiny and likely never hit. But if you make it idempotent in the first place you should do it right.

                                                    1. 1

                                                      I spent the last year making my darts scorer app remote-friendly. Basically, that people from all over the world can play darts together and everybody enters their score. The most requested feature is a new game mode, cricket to be precise. It’s very intimidating to go from 1 to N without rewriting everything. The first tiny step is adding a game mode selection, after that, I hope that the TypeScript errors will guide me through the tunnel.