1. 5

    I’m wondering where the author researched his opinions, I don’t see how boot environments are not supported on Linux.

    First of all they are essentially just a function of set of ZFS clones, so if ZFS is supported, so are boot environments.

    Second of all there are boot environment managers on Linux - zedenv (disclaimer I am the author) - being one of them.

    1. 3

      Also, it is weird (also in this respect) to summarize all of Linux in one column and/or using such narrow features. E.g., NixOS, Guix, and Fedora may not have boot environments, but they also do not need them. Since they do not use imperative, mutable package management, you do not need filesystem snapshots to persist older system versions, atomic upgrades/rollbacks are core to their system design.

      tl;dr you can always pick the rows/columns such that the feature table looks beneficial to your system.

    1. 8

      I’ve said it before and I’ll say it again: ZFS should be the default on all Linux distros. It’s in a league of its own, and makes all other existing Linux filesystems irrelevant, bizarre licensing issues be damned.

      1. 7

        I use ZFS and love it. But I disagree that ZFS should be the default as-is. It requires a fair bit of tuning. For non-server workloads, the ARC in particular. ZFS does not use Linux’ buffer cache and while ARC size adapts, I have often seen on lower memory machines that the ARC takes too much memory at a given point, leaving too little memory for the OS and applications. So, most users would want to tune zfs_arc_max for their particular workload.

        I do think ZFS should be available as an option in all Linux distributions. It is simply better than the filesystems that are currently provided in the kernel. (Maybe bcachefs will be a competent alternative in the future.)

        1. 2

          I agree.

          I remember installing FreeBSD 11 once (with root on ZFS) because I needed a machine remotely accessible via SSH to handle files on an existing disk with ZFS.

          No shizzle, FreeBSD defaults, the machine had 16G of RAM, and during an hours long scp run, ARC decided to eat up all the memory, triggering the kernel into killing processes… including SSH.

          So I lost access, had to restart scp again (no resume, remember), etc. This is a huge show stopper and it should never happen.

          1. 1

            That seems like a bug that should be fixed. Don’t see any reason why that should prevent it from being the default though.

          2. 1

            That’s definitely something to consider, however, Apple has made APFS (ZFS inspired) the default on macOS, so there’s got to be a way to make it work for ZFS + Linux Desktop too. ZFS is all about making things work without you having to give it much thought. Desktop distros can pick reasonable defaults for desktop use, and ZFS could possibly make the parameter smarter somehow.

          3. 2

            I think the licensing issue is the primary problem for Linux distros.

            1. 1

              I agree on technical superiority. What about the Oracle threat given its owner pulled off that API trick? Should we take the risk of all owing Oracle’s lawyers money in some future case? Or rush to implement something different that they don’t control with most of its strengths? I think the latter makes the most sense in the long-term.

              1. 3

                Oracle is not a problem, as the ZFS license is not being violated – it is the Linux license.

                1. 1

                  “Oracle is not a problem, as the ZFS license is not being violated”

                  That’s a big claim to make in the event large sums of money are ever involved. Oracle threw massive amounts of lawyers at Google with the result being API’s were suddenly a thing they could copyright. Nobody knew that before. With enough money and malicious intent, it became a thing that could affect FOSS developers or anyone building on proprietary platforms. What will they do next?

                  I don’t know. Given they’re malicious, the safest thing is to not use anything they own or might have patents on. Just stay as far away from every sue-happy party in patent and copyright spaces. Oracle is a big one that seeks big damages for its targets on top of trying to rewrite the law in cases. I steer clear of their stuff. We don’t even need it, either. It’s just more convenient than alternatives.

                  1. 8

                    The CDDL, an OSI-approved open source licensed, includes both a copyright and patent grant for all of the code released by Sun (now Oracle). Oracle have sued a lot of people for a lot of things, but they haven’t come after illumos or OpenZFS and there are definitely companies using both of those bodies of software to make real money.

                    1. 2

                      I think you’re missing the implications of they effectively rewrote the law in the case I referenced. If they can do that, it might not matter what their agreements say if it’s their property. The risk might be low enough that it never plays out. One just can’t ever know if they depend on legal provisions with a malicious party that tries to rewrite laws in its favor with lobbyists and lawyers.

                      And sometimes succeeds unlike basically everyone doing open source and free software. Those seem to barely enforce their agreements and/or be vulnerable to patent suits in case of the permissive licenses. Plus, could the defenders even afford a trial at the current rates?

                      I bet 10 years ago you wouldn’t have guessed a mobile supplier using an open-ish platform would be fighting to avoid giving over $8 billion dollars to an enterprise-focused, database company. Yet, untrustworthy dependencies let that happen. And we got lucky it was a rich company that depended on OSS/FOSS stuff defending. The rulings could’ve been worse for us if it wasn’t Google.

                      1. 6

                        Seeing as Sun gave ZFS away before Oracle bought it, Oracle would have a LOT of legal wackiness to get the CDDL license revoked somehow. But for the safe of argument, let’s assume they do manage somehow to make it invalidated, and went nuts and decided to try and charge everyone currently using ZFS pay bajillions of dollars for “their” tech. Laws would have to change significantly for that to happen, and with such a significant change in current law, there is basically zero chance it would be retro-active from the moment you started using ZFS, so worst case you’d have to pay from the time of the law change. That is if you didn’t just move off of ZFS after the law changed and be out zero dollars.

                        Also, the OSS version of ZFS is significantly different from Oracle’s version that they are sort of kissing cousins at best anymore. ZFS has been CDDL licensed since 2005, so a long history of divergence from the Oracle version. I think Oracle would have a VERY hard time getting the OSS version back under the Oracle banner(s). Even with very hypothetical significant law changes.

                        I’m in favour of things competing against ZFS, but currently nothing really does.. BTRFS tries, but their stability record is pretty miserable for anything besides the simplest workloads. ZFS has had wide production usage since 2001. Maybe in another 5 or 10 years we will have a decent stable competitor to some of ZFS’s feature-sets.

                        But regardless if you are a large company with something to lose, your lawyers will be the ones advising you about using ZFS or not, and Canonical’s lawyers clearly decided there was nothing to worry about, Along with Samsung(who own Joyent, the people behind Illumos). There are also many other large companies that have bet big on Oracle having basically zero legal leg to stand on.

                        Of course the other side of the coin is the ZFS <-> Linux marriage, but that’s easy just don’t run ZFS under Linux, or use the Canonical shipped version and let Canonical take all the legal heat.

                        1. 2

                          Best counterpoints so far. I’ll note this part might not be as strong as you think:

                          “and Canonical’s lawyers clearly decided there was nothing to worry about, Along with Samsung(who own Joyent, the people behind Illumos)”

                          The main way companies dodge suits is to have tons of money and patents themselves to make the process expensive as hell for anyone that tries. Linux companies almost got patent sued by Microsoft. IBM, a huge patent holder, stepped up saying they’d deal with anyone that threatened it. They claimed they were putting a billion dollars into Linux. Microsoft backed off. That GPL companies aren’t getting sued made Canonical’s lawyers comfortable but not an actual assurance. Samsung is another giant, patent holder with big lawyers. It takes an Apple-sized company to want to sue them.

                          So, big, patent holders or projects they protect are outliers. That might work to ZFS’s advantage here. Especially if IBM used it. They don’t prove what will happen with smaller companies, though.

                          1. 2

                            I agree with you in theory, but not in practice because of the CDDL (which ZFS is licensed under). This license explicitly grants a “patent peace” see: https://en.wikipedia.org/wiki/Common_Development_and_Distribution_License

                            I know most/many OSS licenses sort of wimp out on patents and ignore the problem, CDDL doesn’t. Perhaps it could have even stronger language, and there might be some wiggle room for some crazy lawyering.. I just don’t really see Oracle being THAT crazy. Oracle, being solely focused on $$$$, would have to see some serious money bags to go shake loose, I doubt they would ever bother going after anyone not the size of a Fortune 500, the money just isn’t there. Google has giant bags full of money they don’t even know what to do with, so Oracle trying to steal a few makes sense. :P

                            Oracle going after Google makes sense knowing Oracle, and it was , like you said, brand new lawyering, trying to create API’s out of Copyrights. Patents are not remotely new. So some lawyer for Oracle would have to dream up some new way to screw up laws to their advantage. Possible sure, but it would be possible for any other crazy lawyer to go nuts here (wholly unrelated to ZFS or even technology), it’s not an Oracle exclusive idiocy. Trying to avoid unknown lawyering that’s not even theoretical at this point would be sort of stupid I would think… but I’m not a lawyer.

                            1. 1

                              “I know most/many OSS licenses sort of wimp out on patents and ignore the problem, CDDL doesn’t.”

                              That would be re-assuring on patent part.

                              “Possible sure, but it would be possible for any other crazy lawyer to go nuts here (wholly unrelated to ZFS or even technology), it’s not an Oracle exclusive idiocy. Trying to avoid unknown lawyering”

                              Oracle was the only one to flip software copyright on its head like this. So, I don’t think it’s an any company thing. Either way, the threat I’m defending against isn’t unknown lawyering in general: it’s unknown lawyering of a malicious company whose private property I may or may not depend on. When you frame it that way, one might wonder why anyone would depend on a malicious company at all. Avoiding that is a good pattern in general. Then, the license negates some amount of that potential malice for a great product with unknown, residual risk.

                              I agree the residual risk probably won’t affect individuals, though. An Oracle-driven risk might affect small to mid-sized businesses depending on how it plays out. Good news is swapping filesystems isn’t very hard on Linux and BSD’s. ;)

                    2. 4

                      AFAIK, it’s the GPL that’s being violated. But I’m really tired and the SFC does mention something about Oracle suing so 🤷.

                      Suing based on the use of works derived from Oracle’s CDDL sources would be a step further than the dumb Google Java lawsuit because they haven’t gone after anyone for using OpenJDK-based derivatives of Java. Oracle’s lawsuit-happy nature would, however, mean that a reimplementation of ZFS would be a bigger target because it doesn’t have the CDDL patent grant. Of course, any file system that implements one of their dumb patents could be at risk….

                      I miss Sun!

                2. 1

                  What does ZFS have that is so much better than btrfs?

                  I’m also not sure these types of filesystems are well suited for databases which implement their own transactions and COW, so I’m not sure I would go as far as saying they are all irrelevant.

                  1. 11

                    ZFS is extremely stable and battle-tested, while that’s not a reason in itself to make it a better filesystem, it makes it a extremely safe option when what you’re looking for is something stable to keep your data consistent.

                    It is also one of the most cross-platform file system. Linux, FreeBSD, MacOS, Windows Illumos. It has a huge amount of development behind it, and as of recently the community has come together significantly across the platforms. Being able to export your pool on FreeBSD and import it on Linux or another platform makes it a much better option if you want to avoid lock-in.

                    Additionally, the ARC

                    Problems with btrfs that make it not ready:

                    1. 0

                      If I don’t use/want to use RAID5 then I don’t see the problem with btrfs.

                      1. 3

                        I ran btrfs in production on my home server for ~3-4 years, IIRC. If you want to use btrfs as a better ext4, e.g. just for the compression and checksumming and maybe, maybe snapshotting, then you’re probably fine. If you want to do anything beyond that, I would not trust it with your data. Or at the very least, I wouldn’t trust it with your data that’s not backed up using something that has nothing to do with btrfs (i.e. is not btrfs snapshots and is not btrfs send/receive).

                        I had three distinct crashes/data corruption problems that damaged the filesystem badly enough that I had to back up and run mkfs.btrfs again. These were mostly caused by interruptions/power failures while I was making changes to the fs, for example removing a device or rebalancing or something. Honestly I’ve forgotten the exact details now, otherwise I’d say something less vague. But the bottom line is that it simply lacks polish. And mind you, this is from the filesystem that is supposed to be explicitly designed to resist this kind of corruption. I know at least the last case of corruption I had (which finally made me move to ZFS) was obviously preventable but that failure handling hadn’t been written yet and so the fs got into a state that the kernel didn’t know how to handle.

                    2. 3

                      well, I don’t know about better, but ZFS has the distinct disadvantage of being out of tree filesystem so it can and will break depending completely on the whims of kernel development. How anyone can call this stable and safe for production use is beyond me.

                      1. 3

                        I think the biggest argument is mature implementations used by large numbers of people. That catches lots of common and uncommon problems. In reliability-focused filesystems, that the reliability is field-proven then constantly maintained is more important to me than about anything. The only reason I don’t use it is that it came from Oracle with all the legal unknowns that can bring down the line.

                        1. 3

                          When you say “Oracle”, are you referring to ZFS or btrfs? ;)

                          1. 1

                            Oh shit! I didn’t know they designed both! Glad I wasn’t using btrfs either. Thanks for the tip haha.

                        2. 2

                          On a practical level, ZFS is a lot more tested (in Solaris/Illumos, FreeBSD, and now Linux); more different people have put more terabytes of data in and out of ZFS than they seem to have for btrfs. This matters because we seem to be unable to build filesystems that don’t run into corner cases sooner or later, so the more time and data a filesystem has handled, the more corner cases have been turned up and fixed.

                          On a theoretical level, my personal view is that ZFS picked a better internal structure for how its storage is organized and managed than btrfs did (unless btrfs drastically changed things since I last looked several years ago). To put it simply, ZFS is a volume manager first and then a filesystem manager second (on top of the volumes), while btrfs is (or was) the other way around (you manage filesystems and volumes are a magical side effect). ZFS’s model does more (obvious) violence to Linux IO layering than I think btrfs’s does, but I strongly believe it is the better one and gives you cleaner end results.

                        3. 0

                          Why would I want to run ZFS on my laptop?

                          1. 1

                            Why wouldn’t you want to run it on your laptop?

                        1. 2

                          I don’t get the attraction. Why would a developer pick this over something like Ubuntu or Fedora? The features cited sound like neat yum enhancements, but is that really worth the price of admission?

                          1. 11

                            RHEL is an “approved distribution” for a lot of enterprises, meaning products and solutions based on it get approved more easily than others (this is of course true of Ubuntu as well, but very often less true of community-supported distros). Being able to run the same operating system on your workstation as your final product is often beneficial.

                            Not saying it’s great, but a large amount of code that I have to support is officially only supported on RHEL, so this is big news for me at least. :)

                            1. 3

                              Paid support, so a developer wouldn’t, but a business owner might.

                              1. 2

                                You can actually get a developer license for free. It wouldn’t be my choice for development either, but if you are in the RHEL ecosystem, I can see the benefits of developing on the system you’re deploying to.

                                1. 2

                                  I guess the 10 year+ support makes it attractive for some use cases

                                1. 2

                                  Very interesting. Reminds me of Kata containers. Could be really useful for applications where security is paramount. I think the idea of reusing the flatpak infrastructure that already existed was a great idea.

                                  1. 2

                                    Nice catch! Kata and Qubes OS are my main inspirations for this project. In fact, flatkvm-linux is a variant of Kata’s packaged Linux kernel with some additions (sound, virtio-9p, ia32 emulation…).

                                  1. 4

                                    That is a very strange URL. Is there some sort of reasoning behind https://https.www.google.com.tedunangst.com anyone can think of?

                                    1. 6

                                      It’s trolling (in the fun way, not the bad way).

                                      See the recent post about displaying URLs: https://lobste.rs/s/2e5min/chrome_guidelines_for_displaying_urls#c_x2xv8k.

                                      1. 3

                                        (sighs) Another victim of a cert strategy followed by a URL that’s partly intended to cause exactly these reactions in comment threads. Now, we’re 3 to 0 off-topic in comments as tedu watches with a wise-ass grin. There also may be no stopping this on any site with new users coming in. Gotta fix the 0 if nothing else.

                                        1. 2

                                          If they’re going to go through the trouble of making that domain name they could at least take half an hour and make the site display with a reasonable font size on mobile. Sigh indeed.

                                          1. 1

                                            The font is small, but not too small I think? Maybe I hold my phone too close, but I get kind of annoyed at sites that can only fit two sentences on a screen. Maybe it could go another .2 em or so.

                                      1. 1

                                        I’ve enjoyed experimenting with sr.ht up until now, and I’m very excited to see how things will develop now that it’s an alpha. I’m honestly amazed by what you’ve been able to build so far, especially the build service. Keep up the good work Sir!

                                        1. 1

                                          Stow is cool and I remember looking at it when I was trying to find something better than git submodules, and symlinking for managing my dotfiles. I ended up going with vcsh, and myrepos, and have so far been really happy with it. It’s a bit more of a minimal solution, and a wrapper around git but it does what I want.

                                          1. 2

                                            That’s really cool, I’m impressed by the amount of common layers that can be achieved. I guess that makes sense though since there is so much reproducibility with nix.

                                            1. 2

                                              Flatpak is a definite no for me as long as they think it’s acceptable to dump things into $HOME. It’s 2018. No new application should do this.

                                              1. 3

                                                Can you elaborate on this? What do they dump in $HOME and where exactly? You can’t change it?

                                                1. 0

                                                  Flatpak creates its own .var directory in $HOME.

                                                  1. 3

                                                    What’s wrong with that?

                                                    1. 0

                                                      It’s my home directory, not the application’s.

                                                2. 3

                                                  I have the same question as @andyc. Do you think of applications that create files, like rc-files, or folders on $HOME directory level or does this even include subfolders of the XDG base directories, e.g. XDG_CONFIG_HOME (~/.config/<application>)?

                                                  Update:

                                                  I just installed an application via flatpak and checked which folders where created/modified and it showed that flatpak does not respect the XDG directories specification, instead the application was installed into .var/app/. I assume that this is what you’re referring to?

                                                  1. 3

                                                    Yes, I was referring to the .var directory.

                                                    But according to the Flatpak developers Flatpak adheres to the XDG spec and .var is “nothing to see here”: https://github.com/flatpak/flatpak.github.io/issues/191

                                                  2. 2

                                                    While I agree with you that they should have used a directory for ~/.var that adheres to the XDG spec - like ~/.local/var - they aren’t dumping configuration or any files besides that directory into $HOME. I would however like to see an explanation as to why it was necessary to use the ~/.var directory. Apparently after a discussion including XDG devs, they decided to go that route.

                                                    1. 3

                                                      It has been a common issue of application developers to believe that their app is special and should be exempt from the rules. I have seen it many times, but the Flatpak devs invented a whole new level of entitlement.

                                                      1. 2

                                                        What’s supposed to be the correct way to do this?

                                                  1. 5

                                                    git is intended to be usable in a non graphical setting, anything that doesn’t render properly as text only has no place in my opinion. Emotions don’t belong in commit messages.

                                                    1. 2

                                                      So stoked to finally see this come to ZFS. Adding the wrong device and causing problems has been one of the big things people point to being a common mistake for people with ZFS.

                                                      1. 3

                                                        Also makes migrating a pool to new hardware way easier. Remove the old vdev, wait for the new one to be built, no downtime in between.

                                                      1. 3

                                                        There’s something about break statements I’ve never liked, for some reason they always seem wrong. Not that there’s anything programmatically incorrect about using them.

                                                        I personally use next() instead whenever feasible, and like that python gives you a concise alternative.

                                                        1. 2

                                                          I have felt the same way for many years about break, but I can’t put my finger on it. Often it’s possible to factor the loop into a separate function and use return in place of break; for some reason this feels less dirty.

                                                        1. 1

                                                          Really excited to see this, I’ve been running games through wine using flatpak, and I found they run quite well. It would be awesome to see the full steam catalog available on Linux.

                                                          Really stoked that it’s open source, I look forward to seeing their improvements pushed upstream, hopefully this will benefit all users of wine.

                                                          1. 3

                                                            I’m John, I write about projects I’m interested in at the time or working on which are mostly related to System Administration, Unix and Linux.

                                                            https://ramsdenj.com/

                                                            1. 7

                                                              Now we have bpf too, which is supposed to be a higher performance firewall that will support the nftables API.

                                                              1. 1

                                                                While it reduces readability, I’ve definitely had cases where having assignment expressions would have meant one less function call. Used sparingly, I think it’s a useful tool to have.

                                                                1. 12

                                                                  As someone who uses arch on all my developer machines, arch is a horrible developer OS, and I only use it because I know it better than other distros.

                                                                  It was good 5-10 years ago (or I was just less sensitive back then), but now pacman Syu is almost guaranteed to break or change something for the worse, so I never update, which means I can never install any new software because everything is dynamically linked against the newest library versions. And since the arch way is to be bleeding edge all the time, asking things like “is there an easy way to roll back an update because it broke a bunch of stuff and brought no improvements” gets you laughed out the door.

                                                                  I’m actually finding myself using windows more now, because I can easily update individual pieces of software without risking anything else breaking.

                                                                  @Nix people: does NixOS solve this? I believe it does but I haven’t had a good look at it yet.

                                                                  1. 14

                                                                    Yes, Nix solves the “rollback” problem, and it does it for your entire OS not just packages installed (config files and all).

                                                                    With Nix you can also have different versions of tools installed at the same time without the standard python3.6 python2.7 binary name thing most place do: just drop into an new nix-shell and install the one you want and in that shell that’s what you have. There is so much more. I use FreeBSD now because I just like it more in total, but I really miss Nix.

                                                                    EDIT: Note, FreeBSD solves the rollback problem as well, just differently. In FreeBSD if you’re using ZFS, just create a boot environment before the upgrade and if the upgrade fails, rollback to the pre-upgrade boot environment.

                                                                    1. 9

                                                                      Being a biased Arch Developer, I rarely have Arch break when updating. Sometimes I have to recompile our own C++ stack due to soname bumps but for the rest it’s stable for me.

                                                                      For Arch there is indeed no rollback mechanism, although we do provide an archive repository with old versions of packages. Another option would be BTRFS/ZFS snapshots. I believe the general Arch opinion is instead of rolling back fixing the actual issue at hand is more important.

                                                                      1. 8

                                                                        I believe the general Arch opinion is instead of rolling back fixing the actual issue at hand is more important.

                                                                        I can see some people might value that perspective. For me, I like the ability to plan when I will solve a problem. For example I upgraded to the latest CURRENT in FreeBSD the other day and it broke. But I was about to start my work day so I just rolled back and I’ll figure out when I have time to address it. As all things, depends on one’s personality what they prefer to do.

                                                                        1. 2

                                                                          For me, I like the ability to plan when I will solve a problem.

                                                                          But on stable distros you don’t even have that choice. Ubuntu 16.04, (and 18.04 as well I believe) ships an ncurses version that only supports up to 3 mouse buttons for ABI stability or something. So now if I want to use the scroll wheel up, I have to rebuild everything myself and maintain some makeshift local software repository.

                                                                          And that’s not an isolated case, from a quick glance at my $dayjob workstation, I’ve had to build locally the following: cquery, gdb, ncurses, kakoune, ninja, git, clang and other various utilities. Just because the packaged versions are ancient and missing useful features.

                                                                          On the other hand, I’ve never had to do any of this on my arch box because the packaged software is much closer to upstream. And if an update break things, I can also roll back from that update until I have time to fix things.

                                                                          1. 2

                                                                            I don’t use Ubuntu and I try to avoid Linux, in general. I’m certainly not saying one should use Ubuntu.

                                                                            And if an update break things, I can also roll back from that update until I have time to fix things.

                                                                            Several people here said that Arch doesn’t really support rollback which is what I was responding to. If it supports rollback, great. That means you can choose when to solve a problem.

                                                                            1. 1

                                                                              I don’t use Ubuntu and I try to avoid Linux, in general. I’m certainly not saying one should use Ubuntu.

                                                                              Ok, but that’s a problem inherent to stable distros, and it gets worse the more stable they are.

                                                                              Several people here said that Arch doesn’t really support rollback

                                                                              It does, pacman keeps local copies of previous versions for each package installed. If things break, you can look at the log and just let pacman install the local package.

                                                                              1. 1

                                                                                It does, pacman keeps local copies of previous versions for each package installed. If things break, you can look at the log and just let pacman install the local package.

                                                                                Your description makes it sound like pacman doesn’t support roll backs, but you can get that behaviour if you have to and are clever enough. Those seem like very different things to me.

                                                                                Also, what you said about stable distros doesn’t seem to match my experience in FreeBSD. FreeBSD is ‘stable’ however ports packages tend to be fairly up to date (or at least I rarely run into it except for a few).

                                                                                1. 1

                                                                                  I’m almost certain any kind of “rollback” functionality in pacman is going to be less powerful than what’s in Nix, but it is very simple to rollback packages. An example transcript:

                                                                                  $ sudo pacman -Syu
                                                                                  ... some time passes, after a reboot perhaps, and PostgreSQL doesn't start
                                                                                  ... oops, I didn't notice that PostgreSQL got a major version bump, I don't want to deal with that right now.
                                                                                  $ ls /var/cache/pacman/pkg | rg postgres
                                                                                  ... ah, postgresql-x.(y-1) is sitting right there
                                                                                  $ sudo pacman -U /var/cache/pacman/pkg/postgres-x.(y-1)-x86_64.pkg.tar.xz
                                                                                  $ sudo systemctl start postgres
                                                                                  ... it's alive!
                                                                                  

                                                                                  This is all super standard, and it’s something you learn pretty quickly, and it’s documented in the wiki: https://wiki.archlinux.org/index.php/Downgrading_packages

                                                                                  My guess is that this is “just downgrading packages” where as “rollback” probably implies something more powerful. e.g., “rollback my system to exactly how it was before I ran the last pacman -Syu.” AFAIK, pacman does not support that, and it would be pretty tedious to actually do it if one wanted to, but it seems scriptable in limited circumstances. I’ve never wanted/needed to do that though.

                                                                                  (Take my claims with a grain of salt. I am a mere pacman user, not an expert.)

                                                                                  EDIT: Hah. That wiki page describes exactly how to do rollbacks based on date. Doesn’t seem too bad to me at all, but I didn’t know about it: https://wiki.archlinux.org/index.php/Arch_Linux_Archive#How_to_restore_all_packages_to_a_specific_date

                                                                    2. 12

                                                                      now pacman Syu is almost guaranteed to break or change something for the worse

                                                                      I have the opposite experience. Arch user since 2006, and updates were a bit more tricky back then, they broke stuff from time to time. Now nothing ever breaks (I run Arch on three different desktop machines and two servers, plus a bunch of VMs).

                                                                      I like the idea of NixOS and I have used Nix for specific software, but I have never made the jump because, well, Arch works. Also with Linux, package management has never been the worst problem, hardware support is, and the Arch guys have become pretty good at it.

                                                                      1. 3

                                                                        I have the opposite experience

                                                                        I wonder if the difference in experience is some behaviour you’ve picked up that others haven’t. For example, I’ve found that friend’s children end up breaking things in ways that I would never do just because I know enough about computers to never even try it.

                                                                        1. 2

                                                                          I think it’s a matter of performing Syu update often (every few days or even daily) instead of once per month. Rare updates indeed sometimes break things but when done often, it’s pretty much update and that’s it.

                                                                          I’m an Arch user since 6 years and there were maybe 3 times during those 6 years where something broke badly (I was unable to boot). Once it was my fault; second & third one is related to nvidia driver and Xorg incompatibility.

                                                                          1. 3

                                                                            Rare updates indeed sometimes break things but when done often, it’s pretty much update and that’s it.

                                                                            It’s sometimes also a matter of bad timing. Now every time before doing a pacman -Syu I check /r/archlinux and the forums to see if someone is complaining. If so then I tend to wait for a day or two before the devs push out updates to broken packages.

                                                                          2. 1

                                                                            That’s entirely possible.

                                                                        2. 4

                                                                          I have quite a contrary experience, I have pacman run automated in the background every 60 minutes and all breakage I suffer is from human-induced configuration errors (such as misconfigured boot loader or fstab)

                                                                          1. 1

                                                                            Things like Nix even allow rolling back from almost all user configuration errors.

                                                                            1. 3

                                                                              Would be nice, yeah, though I never understood or got Nix really. It’s a bit complicated and daunting to get started and I found the documentation to be lacking.

                                                                          2. 3

                                                                            How often were you updating? Arch tends to work best when it’s updated often. I update daily and can’t remember the last time I had something break. If you’re using Windows, and coming back to Arch very occasionally and trying to do a huge update you may run into conflicts, but that’s just because Arch is meant to be kept rolling along.

                                                                            I find Arch to be a fantastic developer system. It lets me have access to all the tools I need, and allows me to keep up the latest technology. It also has the bonus of helping me understand what my system is doing, since I have configured everything.

                                                                            As for rollbacks, I use ZFS boot environments. I create one prior to every significant change such as a kernel upgrade, and that way if something did happen go wrong, and it wasn’t convenient to fix the problem right away, I know that I can always move back into the last environment and everything will be working.

                                                                            1. 2

                                                                              How do you configure ZFS boot environments with Arch? Or do you just mean snapshots?

                                                                              1. 3

                                                                                I wrote a boot environment manager zedenv. It functions similarly to beadm. You can install it from the AUR as zedenv or zedenv-git.

                                                                                It integrates with a bootloader if it has a “plugin” to create boot entries, and keep multiple kernels at the same time. Right now there’s a plugin for systemdboot, and one is in the works for grub, it just needs some testing.

                                                                                1. 2

                                                                                  Looks really useful. Might contribute a plugin for rEFInd at some point :-)

                                                                                  1. 1

                                                                                    Awesome! If you do, let me know if you need any help getting started, or if you have any feedback.

                                                                                    It can be used as is with any bootloader, it just means you’ll have to write the boot config by hand.

                                                                          1. 5

                                                                            Last time I checked the, ZFS was actually broken in the installer and hasn’t worked for quite a while. I haven’t heard the best things about how well it’s worked for people who have tried installing Antergos on a ZFS root. With how easy it is to install on Arch, I don’t think I’d recommend this to anyone, especially if it’s their first look at zfsonlinux, it might put someone off.

                                                                            1. 2

                                                                              the only complaint I have with arch is that pip install and npm install fuck the system up. the solution is you have to use them inside virtualenv, nvm.

                                                                              1. 7

                                                                                Which is not a distro problem, but a problem with the defaults of these third party language package managers. Everyone assumes sudo pip install requests is a great idea while have these issues when installing the requests lib from the repository.

                                                                                I’ve configured my npm to install in ~ for globals things, so it doesn’t mess up my system :)

                                                                                1. 4

                                                                                  The problem isn’t really an arch issue, it’s related to the defaults of pip and npm. I’ve never had an issue with either negatively affecting my system if I set it up as the arch wiki recommends.

                                                                                  For pip, use --user if you aren’t in a venv, for npm, set $npm_config_prefix to be located in ~.

                                                                                  1. 4

                                                                                    You install pip or npm with the package manager

                                                                                    then you use them according to the man pages and it borks your system.

                                                                                    This is a fault

                                                                                    If special configuration should be done this should come when you install it with the distro package manager. That’s the whole point of a linux distribution. If you’re supposed to use the virtual env you should at least get a warning about that when you install the tool. UX matters.

                                                                                    1. 7

                                                                                      If you’re supposed to use the virtual env you should at least get a warning about that when you install the tool.

                                                                                      Actually almost every popular Python package suggests using virtualenvs (or more recently, pipenv). AFAIK, sudo pip install is considered bad practice in the Python community.

                                                                                  2. 7

                                                                                    Yeah, my solution to the npm problem is to stay far far away from npm.

                                                                                    1. 1

                                                                                      Not exactly a universally applicable solution.

                                                                                      1. 2

                                                                                        As a society, however, we’d all be better off if we could.

                                                                                  1. 4

                                                                                    IMO, the TrueOS folks are being way to optimistic. I recently switched back to FreeBSD from TrueOS, and a big part of it is that I don’t think they have a track record of being able to deliver their layer as a finished high quality project.

                                                                                    1. 4

                                                                                      I agree. The TrueOS guys are working on some interesting stuff but my experience, TrueOS never quite feels finished. Sometimes I think they’re trying to do too much, the big switch with OpenRC, meaning things are quite different from FreeBSD, a server version, desktop version, Lumina…

                                                                                      I’d be more inclined to use TrueOS if they stuck closer to FreeBSD, but just made it easier to set up a graphical system. Right now they’re not exactly FreeBSD, but they’re not exactly something else. You cant quite apply what you know from FreeBSD though, because of how they do differ.