1. 2

    Is there a list of SQL operations that are optimal in SQL and that are not?

    What are the operations that SQL truly excels? What are the operations better handled with commands for text streams/files?

    1. 4

      No. There are no SQL operations that are “not optimal” in SQL. A query is a description of a result, not the execution strategy. Any execution strategy can be chosen by a SQL engine, and obviously optimizers aim to choose the optimal one.

      There is only one situation where using text streams is objectively better: the operation you want to do isn’t implemented efficiently by your SQL engine, which is a problem with your engine, not SQL itself. There are even SQL engines that operate on text streams directly, e.g. Hive, or q.

      The author has a different problem, they don’t know enough about SQL / their SQL engine to make this query work. Which the author actually states in the post.

      As experimenting with each suggestion could easily take at least half a day, I proceeded with a way I knew would work efficiently and reliably.

      This query is an “optimal” case where SQL typically excels: most SQL engines are perfectly capable of using this exact execution plan. Given that personal constraint, they made a reasonable choice to run the query outside of their database using an approach they understand.

    1. 2

      I love getting data on myths like this. Mythbusters wasn’t exactly going to tackle this one.

      1. 11

        “We need a more complex build system” seems to be a pretty common trap. “We had our reasons” is always offered, but some years later there’s always a monumental undertaking to replace it. There’s a few projects I’m thinking of where building resembles an iterative process of building tools to build to tools to generate python scripts that create make files that build tools… “But that was the only way to create a debug build for windows!”

        You think you have a problem nobody else has? You open this door, you’re really going to have problems nobody else has.

        1. 3

          I saw several times this pattern and as a junior, I’ve often blamed my naiveness in favor of trusting more experienced people. Although, it still feels like “we’re doing something very complicated to do something that should be simple”. At the end I’m always told that this is because it’s more flexible and will adapt better if the packaging needs shift. Which I find odd, since it’s making the whole company slower everyday for a “future” need that nobody knows what it will be.

          So exacly like you said, it seems a good idea at first but finally everybody has problem with this…

          1. 2

            I have no data to support this theory, but I feel like this mostly happens because people don’t want to learn cmake (or don’t know that they should).

            1. 1

              Yes, people should learn cmake, if only to learn how awfully limiting it is.

              1. 1

                And you would suggest?

                1. 1

                  For real-world use it depends on the use case; after all these years, there is unfortunately still no one-size-fits-all C/C++ build system. For personal projects I usually use GNU make (non-recursive).

                  I keep hoping that something like tup (but a little less minimalist) will take the stage one day, but it hasn’t happened yet.

            2. 1

              You think you have a problem nobody else has? You open this door, you’re really going to have problems nobody else has.

              On the other hand, if you never face problems that nobody else had before, you are not doing anything new.

            1. 5

              tldr; learning

              1. 1

                Thanks. I was looking for the thesis stated in the usual places and couldn’t find it.

              1. 4

                Is this supposed to be designed for female programmers? If so what makes it specific to women?

                1. 12

                  Nothing? But it’s made by an organization called “Django Girls”.

                  1. 8

                    About and contributing

                    This tutorial is maintained by DjangoGirls.

                    Following the link gets us this (emphasis mine):

                    Django Girls is a non-profit organization and a community that empowers and helps women to organize free, one-day programming workshops by providing tools, resources and support. We are a volunteer run organization with hundreds of people contributing to bring more amazing women into the world of technology. We are making technology more approachable by creating resources designed with empathy.

                    To me, it looks like they created high quality approachable documentation as material for their workshops, to further their goal of bringing women into tech. “Designed with empathy” probably means it avoids using exclusive language, anecdotes, analogies, and so on. So less “designed for female programmers” and more “doesn’t assume programmers are typically male.”

                    Otherwise, nothing. I don’t reckon a bunch of women were about to write the “Django Bros Tutorial.”

                    1. 11

                      As others have said, the tutorial name just comes from the organisation.

                      It’s notable in that it is tutorial meant for people with 0 knowledge and possible contact issues with computers.

                      Also, FWIW, the “Girls” name for those organisations (coming from “Rails Girls” is widely regarded a mistake now. The chapter I help out with (Rails Girls Berlin) has recently renamed into “Code Curious”. Turns out that grown women don’t feel spoken to by “Girls”.

                      1. 6

                        Turns out that grown women don’t feel spoken to by “Girls”.

                        This is a cultural thing, by which I mean there are women in the US, at least, who are older than I am (34) who wouldn’t be troubled by being called “girls” or would actively appreciate it as a sign of informality.

                        1. 5

                          It’s not that people saw it insulting or something, we had a lot of people that just didn’t feel addressed at first contact! The amount of people we found passing on the project on first contact for the reason that they thought it was for people under 18 was notable.

                          Interestingly, the US version (and precursor) of Rails Girls is called RailsBridge for reasons of not typecasting.

                          It’s a thing to write books about :D. I’m quite interested how Code Curious turns out. RG is quite a successful brand, which is lost in the process.

                    1. 7

                      It’s great to see OSM has come so far. I used to dismiss it out of hand in favor of Google Maps, but nowadays that’s a big mistake.

                      I didn’t know Mapbox was OSM, but it makes sense. I’ve only used it a little bit, but damn Mapbox made everything so easy. With it I visualized thousands of routes all over the globe in just a couple of hours coding / debugging. It’s absolutely my first choice for anything map-related I need to do in the future.

                      1. 2

                        And contributing to OSM is extremely easy too.

                      1. 12

                        Say what you will about the Ruby community, but I’ve never seen them fly into a moral panic over a tiny syntax improvement to an inelegant and semi-common construct.

                        The person-years and energy spent bikeshedding PEP 572 are just astoundingly silly.

                        1. 7

                          Say what you will about the Ruby community, but I’ve never seen them fly into a moral panic over a tiny syntax improvement to an inelegant and semi-common construct.

                          Try suggesting someone use “folks” instead of “guys” sometime…

                          1. 13

                            I switched to folks and do you know how satisfying of a word it is to say? “Hey folks? How’s it going folks? Listen up folks!” I love it.

                            On the other hand the interns weren’t too keen on “kiddos.”

                            1. 6

                              I’ve gotten used to saying “’sup, nerds” or “what are you nerds up to?”

                              1. 4

                                A man/woman (or nerd, I guess) after my own heart! This has been my go-to for a while, until one time I walked into my wife’s work (a local CPA / tax service) and said, “What up, nerds?” It didn’t go over so well and apparently I offended some people – I guess “nerd” isn’t so endearing outside of tech?

                                Thankfully, I don’t think I learned anything from the encounter.

                                1. 3

                                  It’s not endearing within tech to anyone over 40.

                                  1. 2

                                    I generally only use it in a technical setting – so within my CS friend group from college, other programmers at work, etc… whenever it’s clear that yes, I am definitely not trying to insult people because I too am a nerd.

                            2. 1

                              as @lmm notes above, a minimalist, consistent syntax is an important part of python’s value proposition. in ruby’s case, adding syntactic improvements is aligned with their value proposition of expressiveness and “programmer joy”, so it’s far less controversial to do something like this.

                            1. 5

                              Just git?

                              I was kind of hoping that if we’re going to break the github hegemony, we might also start to reconsider git at least a little. Mercurial has so many good ideas worth spreading, like templates (DSL for formatting the output of every command), revsets (DSL for querying commits), filesets (DSL for querying files and path) and changeset evolution (meta-graph of commit rewriting).

                              1. 10

                                Don’t forget pijul!

                                Seriously, though, I don’t think there is any “github plus something” that is going to break the github hegemony. Github won because it offered easy forking while Sourceforge was locked in a centralized model. Sourceforge won because it was so much easier than hosting your own repo + mailing list.

                                The thing that will get people away from github has to have a new idea, a new use case that isn’t being met by github right now, and which hasn’t been proposed before. That means that adding hg won’t do it – not because hg is worse than git (honestly, git’s terrible, and hg is fine), but because hg’s already been an option and people aren’t using it.

                                Adding email commits won’t do it, because that use case has been available for a long time (as pointed out elsewhere in these comments) and people aren’t using it.

                                Until something new is brought to the table, it’s all “let’s enter a dominated market with a slight improvement over the dominant tech”, and that’s just not going to be enough.

                                1. 7

                                  So, one thing that I would use a new contender for is being able to put my work under my own domain.

                                  The “new thing” here is “have your personal branding on your site” (which is clearly fairly popular given how common personal domain/sites are among developers).

                                  If I could CNAME code.daniel.heath.cc to your host to get my own github, I’d do it today (as long as any issues/wiki/PR state/etc remained usefully portable).

                                  1. 7

                                    That’s a really neat idea. I don’t think I can prioritize it right now but it’s definitely something I would consider implementing.

                                    1. 3

                                      I actually think that GitHub’s lack of branding and customization is a big reason for its success. When I go take a look at a new project on GitHub, I don’t have to figure out how to navigate a new site’s design, and this makes the GitHub ecosystem as a whole easier to use.

                                      1. 3

                                        I don’t mean corporate/design branding.

                                        I want to use my own name (and be able to move providers without breaking links).

                                        1. 1

                                          I want to use my own name (and be able to move providers without breaking links).

                                          But that will happen anyway, unless your new provider uses the same software as the old one.

                                          1. 1

                                            Yep - so it has to be oss too.

                                      2. 2

                                        You can do that with Gitlab (or Gitea if you prefer something lightweight). Only thing is you need to take care of the hosting yourself. But I’m sure there are companies offering a one-click setup, to which you can later point your own domain.

                                        1. 2

                                          If you host your own gitlab instance, can you fork and submit patches to a project that’s hosted on gitlab,com, as easily/seamlessly as if you were hosted there?

                                          Centralization has benefits that self-hosting can’t always provide. If there were some federation which allowed self-hosting to integrate with central and other self-hosting sites, that seems like a new and interesting feature.

                                          1. 4

                                            Git is already federated with email - it’s specific services like GitHub which are incompatible with git’s federation model (awfully conveniently, I might add). sr.ht is going to be designed to accomodate git’s email features, both for incoming and outgoing communication, so you’ll be able to communicate easily between sr.ht instances (or sr.ht and other services like patchworks or LKML).

                                            1. 2

                                              As I mention earlier, though, federation by email has been available for a long time and hasn’t been used (by enough people to replace github). The (vast) majority of developers (and other repo watchers) prefer a web UI to an email UI.

                                              1. 3

                                                I intend to build a web UI which is driven by email underneath.

                                            2. 3

                                              The gitlab, gitea, and gogs developers are working on this but it’s still very much in the discussion stage at this point. https://github.com/git-federation/gitpub/

                                              1. 1

                                                Oh, definitely no federation. Didn’t know that federation was what he was looking for.

                                                1. 2

                                                  I don’t know exactly what he was looking for, but It seemed like one of:

                                                  • hosted, centralized, but with my domain plus some branding, or
                                                  • self-hosted but with the features of centralization, including community, etc

                                                  The latter sounds to me like it would need federation.

                                              2. 1

                                                It’s currently awkward to run multiple domains on most OSS servers which might otherwise be suitable.

                                            3. 3

                                              hg isn’t really an option right now, though. There’s nowhere to host it. There’s bitbucket, and it’s kind of terrible, and they keep making it worse.

                                              If you can’t even host it, people won’t even try it.

                                            4. 13

                                              I’m afraid you’re not going to find a sympathetic ear in sr.ht. I am deeply fond of git and deeply critical of hg.

                                              The GitHug hegemony has nothing to do with its basis on git. If git were the product of GitHub, I might agree, but it’s not. If you really want to break the GitHub hegemony you should know well enough to throw your lot in with the winning tool rather than try to disrupt two things at once.

                                              1. 3

                                                Do you mind expanding on why you are deeply critical of mercurial?

                                                1. 7

                                                  Perhaps some day I’ll write a blog post going into detail. The short of it is that git is more Unixy, Mercurial does extensibility the wrong way, and C is a better choice than Python (or Rust, I hear they’re working on that).

                                                  1. 3

                                                    Git IS more unixy! Using hg feels suspiciously like using a GUI and I can’t figure out why.

                                                    1. 3

                                                      because hg‘s command-line interface was “designed”, whereas git’s command-line interface “evolved” from how it was being used.

                                                2. 2

                                                  The GitHug hegemony has nothing to do with its basis on git.

                                                  Exactly; it’s the other way around. Git got popular because of github.

                                                  Git was much worse before github made it popular. It’s bad now and difficult to use now, but it was much worse before 2008. So if you just want to get away from Github, there’s no need to stay particulary enamoured with git either.

                                                  And whatever criticisms you may have about hg, you have to also consider that it has good ideas (those DSLs above are great). Those ideas are worth spreading, and git for a long time has tried to absorb some of them and hasn’t succeeded.

                                              1. 1

                                                On the EINTR/retry issue, it’s not clear to me why a simple ‘return to userspace and retry the instruction’ is harder to implement.

                                                There is kernel-side state associated with the fd (seek position, network buffers etc) but these need to be maintained anyway for an EINTR return.

                                                Put another way - what work can the kernel avoid in the ‘return EINTR and let the userspace application call the system call again’ scenario which is required in the ‘return to userspace at PC-1’?

                                                1. 5

                                                  There are two big issues: changing entry conditions, and blocking after interrupts.

                                                  Blocking after interrupts is fairly obvious when you think about it. If you are blocking on some syscall in your main thread, receive a signal, and your syscall is restarted automatically, you cannot respond to that signal in your main thread at all. You’d just keep blocking. If the signal was SIGINT and you would want to cleanly shut down, you can’t. You’d be stuck until your syscall unblocks, which could never happen.

                                                  Changing entry conditions are much more tricky. For example, some syscalls have timeouts—the amount of time they should block before returning regardless of their success or failure. If you start a syscall with a 10 second timeout, what happens if that syscall is cancelled and restarted 5 seconds in? If the same arguments are used, it would be as if the timeout had been 15 seconds. If you receive 1 signal per second indefinitely, the syscall will block forever. Unlikely but possible.

                                                  When you call something with a 10 second timeout, you actually mean 10 seconds from now. To restart these syscalls, the kernel would need to preserve that start time entry condition. That’s fairly doable, but there are other entry conditions that aren’t doable at all. If you’re writing size-formatted output to the console, and you receive SIGWINCH, you don’t want to proceed with the write. Instead you need to reformat the output to match the new console dimensions. The kernel certainly can’t do that for you.

                                                  There are so many reasons you might want to change syscall parameters after an interrupt, or do something else entirely. The kernel can’t know all of them. And designing an interface to conveniently accommodate all of them is a lot harder. Thus the worse-is-better solution: never assume what a program wants to do after an interrupt.

                                                  1. 2

                                                    Signals, I think, is now fairly well accepted as one of the “ugly” parts of posix.

                                                    signalfd was an attempt at fixing it…. Here is more discussion of that..

                                                    https://ldpreload.com/blog/signalfd-is-useless

                                                    Having spend the last week battling the fine fine fine corner cases of signal handling….

                                                    Sigh.

                                                    I wish linux had something better.

                                                    1. 1

                                                      Thanks for this.

                                                      If you are blocking on some syscall in your main thread, receive a signal, and your syscall is restarted automatically, you cannot respond to that signal in your main thread at all. You’d just keep blocking.

                                                      I’m still not getting this one. I’d envisage:

                                                      • application calls blocking read()
                                                      • application receives SIGHUP
                                                      • kernel sees incoming signal and stops doing read()
                                                      • kernel calls up into user spaces to run signal handler in user context for SIGHUP. As far as application goes, it’s still doing read(), but signals can happen any time anyway, so no problem here?
                                                      • kernel restarts read()

                                                      If you’re writing size-formatted output to the console, and you receive SIGWINCH, you don’t want to proceed with the write.

                                                      I’m not sure I agree. Given that arriving signals are inherently racy, I think it could be considered to also be correct to re-run the system call without the application making a new choice based on the new information. (The system call could easily have completed before the signal arrived - and the application should be prepared for that eventuality).

                                                      When you call something with a 10 second timeout, you actually mean 10 seconds from now. To restart these syscalls, the kernel would need to preserve that start time entry condition.

                                                      This is a good point. However, the optimist in me would like to think this is always solvable with API design. (In the timeout case, this would involve absolute timeout rather than relative).

                                                      1. 2

                                                        Your scenario doesn’t work, because signal handlers are extremely restricted. Signal handlers must be reentrant with respect to all other signal handlers, meaning they can’t allocate memory, can’t use most syscalls, can’t use any libc functions that set errno, and can’t non-atomically modify non-local variables.

                                                        For this reason, signal handlers usually set a global volatile flag and return immediately, allowing the main thread to catch EINTR, check the flag, and handle the signal without restriction.

                                                        In the SIGWINCH example, doing what you suggest causes significant visual tearing, even though it’s technically correct. But that assumption about racy signals only works if all syscalls are guaranteed to complete.

                                                        However, the optimist in me would like to think this is always solvable with API design.

                                                        Perhaps. Until such an API actually exists, we must write interruptible code somehow.

                                                        (In the timeout case, this would involve absolute timeout rather than relative).

                                                        What absolute time? The system clock can change. The clock that can’t is the monotonic raw clock, which simply ticks upwards from an unspecified time. I don’t think an API that takes a deadline with respect to a meaningless clock beats an API that requires restarting on interrupt.

                                                        1. 1

                                                          Your scenario doesn’t work, because signal handlers are extremely restricted.

                                                          Yes they are, but I don’t think that stops the kernel from invoking them and then restarting the system call afterwards.

                                                          In the SIGWINCH example, doing what you suggest causes significant visual tearing

                                                          Which could already occur. We’re just (slightly) increasing the window in which delivery of the signal will cause it.

                                                          What absolute time?

                                                          The one with the same semantics as a relative timeout - i.e. monotonic. That is the behaviour you would get if you specified “10 seconds from now”.

                                                          1. 2

                                                            Your scenario doesn’t work, because signal handlers are extremely restricted.

                                                            Yes they are, but I don’t think that stops the kernel from invoking them and then restarting the system call afterwards.

                                                            What I think you’re missing here is that since you can’t do much in a signal handler besides setting a flag for the main thread to check, you must have a way to interrupt/cancel/unblock whatever system call is blocking the main thread so that the main thread can start doing whatever the signal called for. If the kernel automatically restarts the system call, the main thread obviously can’t go check that flag and react accordingly.

                                                            1. 2

                                                              Yes, you’re right, thank you. But in that case, the interrupted return is the feature, not the bug?

                                                              The whole ‘worse is better’ thing is cast as “return EINTR” being an undesirable property.

                                                              1. 2

                                                                Yes indeed, it is a feature. But it is a feature that complicates life for every call that doesn’t care and doesn’t want to be interrupted.

                                                                1. 2

                                                                  OK, but it isn’t anything to do with Worse is Better, right? The article characterises it as a deficiency of implementation.

                                                                  In fact, it’s a feature you need if you want to abort a blocking operation.

                                                                  afaics, the only problem is the default. You probably want the SA_RESTART behaviour by default (and perhaps that should be per-fd, rather than per-signal).

                                                                  But I don’t think the characterisation in the article is fair, unless I’ve missed something.

                                                                  1. 1

                                                                    Worse is better means choosing a design that’s worse in some cases because at least it works for all cases. EINTR absolutely embodies that description. It’s a dead simple way to make sure you can always handle interrupts, but extremely tedious to work with in the common case.

                                                                    1. 2

                                                                      That was my understanding of the pt made in the article and the received wisdom regarding this article.

                                                                      However, I thought the conclusion of the discussion above was that it wasn’t any easier than not handling the interrupt. In fact, the interrupted behaviour is required (it is a feature) in many cases.

                                                                      I still fail to see how it is easier for the kernel to return EINTR rather than restart the syscall. (Apart from the API issue mentioned regarding entry conditions, e.g. relative/absolute times).

                                                                      It might help if someone can outline the desired behaviour. It isn’t “restart syscall”, since that has been disposed of as undesirable above.

                                                                      I think my point is: “this article says the Unix EINTR approach is a short cut in kernel implementation which has imposed a cost on userspace since then”. However:

                                                                      a) I can’t see the shortcut being taken (no one has pointed out how it is easier to restart a read() than return EINTR) b) arguments above are for EINTR being a good thing

                                                                      Is there a 3rd way (other than ‘abort read() early and return EINTR’ or ‘restart read()’) which I’m missing here?

                                                                      1. 2

                                                                        I’m totally out of my element here, but did not EINTR evolve over years? Like, it started really cheap and not useful, but it became better (and more complex) over time? Is this what the author referred to when they said that unix/c improved from 50% to 90%?

                                                      2. 0

                                                        Thus the worse-is-better solution: never assume what a program wants to do after an interrupt.

                                                        One might argue that interrupts are a broken IPC system. Errno definitely is broken.

                                                        However syscalls can fail (as any other computation).

                                                      1. 41

                                                        It’s also developer-friendly because of its excellent wiki.

                                                        I learned Linux doing everything by hand on a Slackware system, then moved to Ubuntu after ~8 years when I realized I’d stopped learning new things. Then a couple years ago I realized I didn’t understand how a bunch of things worked anymore (systemd, pulseaudio, Xorg, more). I looked at various distros and went with Arch because its wiki had helped me almost every time I’d had an issue.

                                                        Speaking of distros, I’m currently learning Nix and NixOS. It’s very nice so far. If I can learn to build packages I’ll probably replace lobsters-ansible with it (the recent issues/PRs/commits tell a tale of my escalating frustration at design limitations). Maybe also my personal laptop: I can experiment first with using nix to try xmonad first because it’s mostly configured by editing + recompiling) and deal with python packaging, which has never worked for me, then move completely to NixOS if that goes well.

                                                        1. 9

                                                          I switched from Mac to NixOS and couldn’t be happier. At work we use Nix for building Haskell projects as well.

                                                          1. 9

                                                            The Arch wiki actually seems to be the only good documentation for using the advanced functionality of newer freedesktop components like pulseaudio, or much older software like Xorg.

                                                            But I’ve noticed it’s documentation for enterprise software like ZFS is usually hot garbage. Not surprising given the community. The recommendations are frequently hokey nonsense: imaginary micro-optimizations or blatantly incorrect feature descriptions.

                                                            What do you find better about nix for making packages than, say, making an rpm or deb? I’ve found those package systems valuable for large scale application deployment. Capistrano has also been nice for smaller scale, with its ability to deploy directly from a repo and roll back deployments with a simple symlink swap. And integration libraries are usually small enough that I’m comfortable just importing the source into my project and customizing them, which relieves so many minor tooling frustrations overall.

                                                            Of course in the end the best deployment system is the one you’ll actually use, so if you’re excited about packaging and deploying with nix, and will thus devote more time and energy to getting it just right, then that’s de facto the best option.

                                                            1. 3

                                                              What do you find better about nix for making packages than, say, making an rpm or deb?

                                                              I don’t, yet. The “If I can learn to build packages” sentence links to an issue I’ve filed. I was unable to learn how to do so from the official documentation. I’ve almost exclusively been working in languages (PHP, Python, Ruby, JavaScript) that rpm/deb have not had good support for, prompting those languages to each implement their own package management systems that interface poorly or not at all with system packaging.

                                                              I’ve used Capistrano, Chef, Puppet, and currently use Ansible for deployment. Capistrano and Ansible at least try to be small and don’t have a pretensions to being something other than an imperative scripting tool, but I’ve seen all of them break servers on deployment, let servers drift out of sync with the config, or fail to be able to produce new deployments that match the existing one. Nix/NixOS/NixOps approach the problem from a different direction; it looks like they started from what the idea of system configuration is instead of scripting the manual steps of maintaining one. Unfortunately nix replicates the misfeature of templating config files and providing its own config file on top of them instead of checking complete config files into a repo. Hopefully this won’t be too bad in practice, though it’s not a good sign that they implemented a programming language.

                                                              I appreciate your closing sentiment, but I’m not really trying to reach new heights of system configuration. I’m trying to avoid losing time to misconfiguration caused by services that fundamentally misunderstand the problem, leading to booby traps in common usage. I see almost all of my experience with packaging + deployment tools as a loss to be minimized in the hopes that they waste less time than hand-managing the global variables of public mutable state that is a running server.

                                                              1. 1

                                                                Hmmm. I don’t think the problems you listed are 100% avoidable with any tool, just easier in some rather than others.

                                                                I like Puppet and Capistrano well enough. But I also think packaging a Rails application as a pre-built system package is definitely the way to go, with all gems installed and assets compiled at build time. That at least makes the app deployment reproducible, though it does nothing for things like database migrations.

                                                              2. 1

                                                                What do you find better about nix for making packages than, say, making an rpm or deb?

                                                                Let me show you a minimal nix package:

                                                                pkgs.writeScriptBin "greeter" "echo Hello $1!"
                                                                

                                                                Et voila! You have a fine nix package of a utility called greeter that you can let other nix packages depend on, install to your environment as a user or make available in nix-shell. Here’s a function that returns a package:

                                                                greeting: pkgs.writeScriptBin "greeter" "echo ${greeting} $1!"
                                                                

                                                                What you have here is a lambda expression, that accepts something that you can splice into a string and returns a package! Nix packages in nixpkgs are typically functions, and they offer an a great amount of customizability without much effort (for both the author and the user).

                                                                At work, we build, package and deploy with nix (on the cloud and on premises), and we probably have ~1000 nix packages of our own. Nobody is counting though, since writing packages doesn’t feel like a thing you do with nix. Do you count the number of curly braces in your code, for instance? If you’re used to purely functional programming, nix is very natural and expressive. So much so that you could actually write your application in the language if it’s IO system were designed for it.

                                                                It also helps a lot that nix can seamlessly be installed on any Linux distro (and macOS) without getting in the way of its host.

                                                                1. 1

                                                                  If only ZFS from Oracle hadn’t had the licensing compatibility issues it currently has, it would probably have landed in the kernel by now. Subsequently, the usage would have been higher and so would the quality of the community documentation.

                                                                2. 4

                                                                  If I can learn to build packages I’ll probably replace lobsters-ansible with it

                                                                  Exactly. I don’t have much experience with Nix (none, actually). But in theory it seems like it can be a really nice OS-level replacement for tools like Ansible, SaltStack, etc.

                                                                  1. 1

                                                                    This is exactly what NixOps does! See here.

                                                                    1. 2

                                                                      Thanks for the video. I’ll watch it over the weekend!

                                                                      Curious - are you also running NixOS on your personal machine(s)? I’ve been running Arch for a long time now but considering switching to Nix just because it makes so much more sense. But the Arch documentation and the amount of packages available (if you count the AUR in) is something that’s difficult to leave.

                                                                      1. 1

                                                                        Yes, I’m using it on my personal machine :). I wouldn’t recommend switching to NixOS all at once, what worked for me was to install the Nix package manager, use it for package management and creating development environments, and then only switch once I was fully convinced that NixOS could do everything I wanted from my Ubuntu install. This took me about a year, even with me using it for everything at work. Another approach would be to get a separate laptop and put NixOS on that to see how you like it.

                                                                        1. 1

                                                                          Interesting. I’ll try it out for some time on a VM to get a hang of it. Thanks for the info!

                                                                  2. 3

                                                                    Even as a Ubuntu user, I’ve frequently found the detailed documentation on the Arch wiki really helpful.

                                                                    1. 2

                                                                      I really want to use Nix but I tried installing it last month and it doesn’t seem to have great support for Wayland yet which is a deal breaker for me as I use multiple HiDPI screens and Wayland makes that experience much better. Anyone managed to get Nix working with Wayland?

                                                                      1. 2

                                                                        Arch’s wiki explaining how to do everything piecemeal really seems strange given its philosophy is assuming their users should be able to meaningfully help fix whatever problems cause their system to self-destruct on upgrade. It’s obviously appreciated, but still…confusing, given how many Arch users I’ve met who know nothing about their system except what the wiki’s told them.

                                                                        1. 1

                                                                          I gave up on my nix experiment, too much of it is un- or under-documented. And I’m sorry I derailed this Arch discussion.

                                                                          1. 1

                                                                            I’m happy to help if I can! I’m on the DevOps team at work, where use it extensively, and I did a presentation demonstrating usage at linux.conf.au this year. All my Linux laptops run NixOS and I’m very happy with it as an operating system. My configuration lives here.

                                                                            1. 2

                                                                              Ah, howdy again. I’m working my way through the “pills” documentation to figure out what’s missing from the nix manual. If you have a small, complete example of how to build a single package that’d probably be pretty useful to link from the github issue.

                                                                              1. 2

                                                                                I made a small change to the example to get it to build, and I’ve added it as a comment to your issue.

                                                                          1. 1

                                                                            Inefficient Rendering (IR). This one was a surprise to me in the measured impact. It’s very common in Rails to do something like this, which calls link_to within a loop to generate an anchor:

                                                                            […]

                                                                            Replacing it with one call to link_to outside of the loop, and the use of gsub within the loop is faster.

                                                                            I wonder which is causing the bigger speed up: inlining the partial view, or caching the link_to and using gsub?

                                                                            1. 19

                                                                              I’ve tried several linux distribution in the last 10years (ubuntu,debian,fedora,arch,nixos) and honestly, NixOS is way above the others for a developer-friendly OS. It’s:

                                                                              • super lightweight
                                                                              • customizable
                                                                              • with an awesome package manager
                                                                              • well documented

                                                                              I love being able to drop in a shell having the package or the lib I want and test things. In comparison, Arch feels like a totally standard linux distribution.

                                                                              1. 9

                                                                                I’ve been using NixOS everywhere around me for about 3.5 years and totally agree. I now work on Atlassian Marketplace which deployed using Docker images that are built from Nix.

                                                                                1. 3

                                                                                  The nix model definitely seems like a great way to build docker images. Reproducible, minimal, flexible, it seems like a perfect fit.

                                                                                  1. 3

                                                                                    That sounds strange to me; if you’re already set up to use nix, why bother with docker? Maybe I’m overlooking some things?

                                                                                    1. 2

                                                                                      Because Atlassian has an internal PaaS which requires Docker. I use NixOS for everything but deploy our systems to that.

                                                                                      1. 1

                                                                                        I’m not using NixOS. And it wouldn’t be for running locally, it would be for deploying on something like Kubernetes. But nix is a flexible, useful tool even when NixOS isn’t involved.

                                                                                      2. 1

                                                                                        How can it be both good as my workstation, and good as a minimal container runtime?

                                                                                        Not for the sake of being argumentative (I have yet to try Nix), just confused because those two seem like opposites.

                                                                                        1. 3

                                                                                          Flexible is the key adjective that makes it work for both. Nix allows you to install a package tree into a target directory, using binary packages. Analogous to debootstrap / kickstart. But it also lets you ad hoc add / update / remove packages in that directory like apt / yum does on a running system. It can also do all this according to a package spec a la bundler / npm / maven.

                                                                                          And it can do all this live on a workstation too! So that’s why it works for both.

                                                                                          It also does a great job of keeping things clean by installing packages into versioned directories, and symlinking the active package into the base system. Similar to what homebrew does on MacOS. That makes cleaning old versions a breeze, and allows multiple versions to be installed, which nix lets you switch between easily.

                                                                                          That being said, I run more conventional distros to keep familiar with the server installs used by customers. I find the utility of that expertise greater than any utility NixOS provides. But since nix is also a standalone tool, it works on less interesting distros and can be used for homedir installs or building docker images.

                                                                                          1. 2

                                                                                            That being said, I run more conventional distros to keep familiar with the server installs used by customers.

                                                                                            Interesting. I find that I do this as well. I.e. I use a minimal vimrc, bash (not zsh/fish), etc., to not confuse my muscle memory of my day job (which is bog-standard Linux/Debian sysadmin).

                                                                                        2. 1

                                                                                          Here’s a good blog post about using Nix to build Docker images: http://lethalman.blogspot.com/2016/04/cheap-docker-images-with-nix_15.html

                                                                                        3. 1

                                                                                          So all the images are built from the nixos base image? It’s the first big company that I hear is using nixos+docker!

                                                                                          1. 2

                                                                                            No base image - just x86_64-linux built Nix binaries.

                                                                                      1. 18

                                                                                        I’ll add to the chorus of folks with good experiences using Archlinux. I’ve been using it for almost 10 years now, and it’s been extremely solid. I use it at home, at work, on my laptop and on my various media PCs. My wife uses it now too. There is nothing in particular that has really made me want to switch to something else. NixOS has piqued my curiosity since I read their paper many years ago, but I’ve never given it a fair shake because Arch has been so good to me, and because I tend to be a “worse is better” kind of a guy, but only when necessary. ;-) I love me some PKGBUILD!

                                                                                        I’m not sure I’d sing praises for the Archlinux community. I do like their trial by fire approach to things. But my personal interactions with them haven’t been a shining beacon of friendliness, I can tell you that. I never really fit in with the Archlinux community, so I tend to avoid it. But I think that’s OK.

                                                                                        1. 3

                                                                                          Arch Linux is (mostly) friendly towards experienced users that also strives towards keeping things simple. How can a distro communicate that it’s not for beginners, nor for maximalists (excluding large groups of users, on purpose) while also communicating that it is friendly?

                                                                                          1. 6

                                                                                            I don’t know how to answer your question, but I will clarify my statement. I said that my personal interactions with people in the Archlinux community have not been what I’d call “friendly.” I’m talking about conversing with people, not the official documentation materials, which I’ve found do a great job of being friendly while simultaneously being a great example of the trial-by-fire approach to learning that really works.

                                                                                            To be fair, I think it is very hard to mix trial-by-fire learning (which I think is one of many great ways to learn) with friendliness in an online community. It’s too easy to slip. But that’s just my belief based on what I think I understand about humans.

                                                                                            Like I said, I just don’t fit in with that style of communication. But there are a lot of places that I don’t fit into. And I really do think that’s OK.

                                                                                            EDIT: Also to add a positive note, not all of my interactions with Archlinux community members have been bad. There have been (very) good ones too. I guess it’s just much easier to focus/remember the bad. :-/

                                                                                            1. 3

                                                                                              I guess it’s just much easier to focus/remember the bad

                                                                                              This is a known psychological effect called Negativity Bias. In theory it protects you to from forgetting things that could harm you, and helps you notice potentially dangerous situations.

                                                                                              1. 2

                                                                                                Yeah, this happens. Mainly because too many people forget how rolling works, and the forums are tired of answering the same questions :/

                                                                                          1. 5

                                                                                            Because knock packets use a timestamp to limit knock-reuse attacks, servers and clients must have synchronized clocks. Clock skew greater than 60 seconds is likely to cause the knock process to fail, resulting in a “connection refused” error when establishing the TCP connection.

                                                                                            Given this, if my server’s clock does fall out of sync, how do I connect to it to fix it if Oxy is my only access method?

                                                                                            1. 1

                                                                                              Remember to set up ntp when you set up your server.

                                                                                              1. 1

                                                                                                Software crashes and networks disconnect. Even if you start NTP you can still end up with an inaccessible server.

                                                                                                1. 0

                                                                                                  So if you have no network connectivity for ntp to update, how are you going to log in with any remote access tool?

                                                                                                  Software crash of ntpd is unrealistic.

                                                                                                  1. 1

                                                                                                    NTP has an exponential back off mechanism for retrying connection, it’s possible that it could get out of sync and not have ticked over when the network is back up, causing time skew but still having the machine allow network connections. Depending on the configuration, it could also plain shut down when it cannot get a connection and not restart when the connection is back.

                                                                                                    Saying that software crashes are unrealistic is in itself an unrealistic view in my opinion. Software WILL crash.

                                                                                                    Allowing clock skew to prevent connections to the machine definitely adds more moving parts to the remote access process and adds risk to that process failing.

                                                                                            1. 1
                                                                                              1. You can merge in both directions

                                                                                              You can do this in svn too. Nevertheless it’s a good thing to know for git users.

                                                                                              Great post, this is extraordinarily helpful as someone who teaches people git often.

                                                                                              1. 34

                                                                                                I think you could reimplement it easily yourself with a small shell script and some calls to mount; but I haven’t bothered.

                                                                                                I don’t have the expertise to criticize the content itself, but statements like the above make me suspect that the author doesn’t know nearly as much about the problem as they think they know.

                                                                                                1. 32

                                                                                                  This reminds me of a trope in the DIY (esp. woodworking DIY) world.

                                                                                                  First, show video of a ludicrously well equipped ‘starter shop’ (it always has a SawStop, Powermatic Bandsaw, and inexplicably some kind of niche tool that never really gets used, and a CNC router).

                                                                                                  Next, show video of a complicated bit of joinery done using some of the specialized machines.

                                                                                                  Finally, audio: “I used for this, but if you don’t have one, you can do the same with hand tools.”

                                                                                                  No, asshole, no I can’t. Not in any reasonable timeframe. Usually this happens in the context of the CNC. “I CNC’d out 3 dozen parts, but you could do the same with hand tools.”

                                                                                                  I get a strong whiff of that sort of attitude from this. It may be that the author is capable of this. It may be possible to ‘do this with hand tools’ like Shell and some calls to mount. It might even be easy! However, there is a reason docker is so popular, it’s because it’s cheap, does the job, and lets me concentrate on the things I want to concentrate on.

                                                                                                  1. 9

                                                                                                    As someone who can do “docker with hand tools,” you and @joshuacc are completely correct. Linux does not have a unified “container API,” it has a bunch of little things that you can put together to make a container system. And even if you know the 7 main namespaces you need, you still have to configure the namespaces properly.

                                                                                                    For example, it isn’t sufficient to just throw a process in its own network namespace, you’ve got to create a veth pair and put one end of that into the namespace with the process, and attach the other end to a virtual bridge interface. Then you’ve got to decide if you want to allocate an IP for the container on your network (common in kubernetes), or masquerade (NAT) on the local machine (common in single box docker). If you masquerade you must make snat and dnat iptables rules to port forward to the veth interface, and enable the net.ipv4.ip_forward sysctl.

                                                                                                    So the “small shell script” is now also a management interface for a network router. The mount namespace is even more delightful.

                                                                                                    1. 8

                                                                                                      Exactly this! One of the most egregious things about the ‘… you could do it with hand tools’ is that it is dismissive of people who really can do it with hand tools and dismissive of the folks that can do it with CNC.

                                                                                                      In woodworking, CNC work is complicated, requires a particular set of skills and understanding, and is prone to a totally different, equally painful class of errors that hand tools are not.

                                                                                                      Similarly, Hand tool work is complicated, requires a particular set of skills and understanding, and is prone to a totally different, equally painful class of errors that power/CNC work is not.

                                                                                                      Both are respectable, and both are prone to be dismissive of the other, but a hand-cut, perfect half-blind dovetail drawer is amazing. Similarly, a CNC cut of 30 identical, perfect half-blind dovetail drawers is equally amazing.

                                                                                                      The moral of this story: I can use the power tool version of containers. It’s called docker. It lets me spit out dozens of identically configured and run services in a pretty easy way.

                                                                                                      You are capable of ‘doing it with hand tools’, and that’s pretty fucking awesome, but as you lay out, it’s not accomplishing the same thing. The OP seems to believe that building it artisinally is intrinsically ‘better’ somehow, but it’s not necessarily the case. I don’t know what OP’s background is, but I’d be willing to bet it’s not at all similar to mine. I have to manage fleets of dozens or hundreds of machines. I don’t have time to build artisanal versions of my power tools.

                                                                                                    2. 2

                                                                                                      And then you have Paul Sellers. https://www.youtube.com/watch?v=Zuybp4y5uTA

                                                                                                      Sometimes, doing things by hand really is faster on a small scale.

                                                                                                      1. 2

                                                                                                        He’s exactly the guy I’m talking about though in my other post in this tree – he’s capable of doing that with hand tools and that’s legitimately amazing. One nice thing about Paul though is he is pretty much the opposite of the morality play from above. He has a ludicrously well-equipped shop, sure, but that’s because he’s been doing this for a thousand years and is also a wizard.

                                                                                                        He says, “I did this with hand tools, but you can use power tools if you like.” Which is also occasionally untrue, but the sentiment is a lot better.

                                                                                                        He also isn’t elitist. He uses the bandsaw periodically, power drillmotors, and so on. He also uses panel saws and brace-and-bit, but it’s not an affectation, he just knows both systems cold and uses whatever makes the most sense.

                                                                                                        Paul Sellers is amazing and great and – for those people in the back just watching – go watch some Paul Sellers videos, even if you’re not a woodworker (or a wannabe like me), they’re great and he’s incredible. I like the one where he makes a joiner’s mallet a lot. Also there’s some floating around of him making a cabinet to hold his planes.

                                                                                                    3. 1

                                                                                                      My reaction was “if you had to write this much to convince me that there are easier ways than Docker, then it sounds like this is why Docker has a market.”

                                                                                                      I’m late to the Docker game - my new company uses it heavily in our infrastructure. Frankly, I was impressed at how easy it was for me to get test environments up and running with Docker.

                                                                                                      I concede it likely has issues that need addressing but I’ve never encountered software that didn’t.

                                                                                                    1. 4

                                                                                                      I wasn’t sure if practices is the best tag for this. Overall I think it’s relevant to remind us not to over-automate things and always leave enough options for manual intervention. Perhaps https://lobste.rs/s/5frjuu/youtube_blocks_blender_videos_worldwide also was a result of too “much” automation?

                                                                                                      1. 3

                                                                                                        Manual intervention yes, but observability the most. It’s ridiculous that no one could figure out what was happening.

                                                                                                        1. 2

                                                                                                          It’s ridiculous that no one could figure out what was happening.

                                                                                                          It’s not ridiculous, it’s a well known phenomenon dismissively called “the paradox of automation” (not a paradox at all, just a fundamental UI law: users are humans).

                                                                                                      1. 3

                                                                                                        I experience pain of using it every time I have to edit i18n strings in Rails project. Even XML would be more convenient.

                                                                                                        Other markup formats with the same philosophy (multiple ways to represent things, indentation-based, concise) has similar problems.

                                                                                                        1. 6

                                                                                                          Can your editor not fold YAML? If not I highly recommend finding one that can.

                                                                                                        1. 10

                                                                                                          Ruby’s load() just loads the first document, and as near as I can tell, doesn’t have a way to load multiple documents.

                                                                                                          Psych.load_stream for anyone who’s wondering.