1. 24

    Suggest -a11y

    I also use set -e quite a lot. There is a very large and useful BashFAQ describing ways that set -e doesn’t do what authors might expect: https://mywiki.wooledge.org/BashFAQ/105

    1. 9

      Suggest -a11y

      If you mean anything other than “accessibility”, could you please define what you’re talking about, or provide a link to a write-up of it? The abbreviation “a11y” is so universally used for “accessibility” (in the same way “i18n” is “internationalization”) that it’s effectively impossible to perform a search for other potential meanings.

      1. 4

        The article was originally tagged a11y, and:

        Suggest -a11y

        is as far as I know common syntax to instruct other lobsters to make a suggestion to remove the a11y tag.

        You can see the moderation log here: https://lobste.rs/moderations?utf8=%E2%9C%93&moderator=%28Users%29&what%5Bstories%5D=stories

        Edit: Seeing as the automatic moderation event took place 5 minutes after my comment, it appears that my understanding of it being common syntax is true.

        1. 25

          Here I was thinking -a11y was a Bash option.

          1. 3

            Same here :)

          2. 3

            OK. I didn’t realize this was trying to suggest something about the metadata of the post; the context of the rest of the comment made it seem as if you were suggesting some strange set of bash flags that I’d never heard of.

            1. 1

              My bad. I don’t know how I ended up adding -a11y

            2. 2

              This is exactly why I think these type of abbreviations are i5c [1]. It’s just a way to obfuscate words and confuse people. (Sorry for the overly salty reaction, but it’s a pet peeve.)

              [1] idiotic

              1. 0

                Same here, I would love to learn about this as well.

              2. 5

                Thanks for that link, I never suspected there were so many undefined corner cases with set -e

                1. 4

                  Yes, great link. I think especially example 5 demonstrates that set -e might be more trouble than it’s worth.

                  1. 4

                    I wonder how many security issues are lurking underneath these inconsistent and weird semantics. This is the kind of muddled thinking that used to cause (and probably still causes) problems in PHP scripts.

                2. 1

                  I also use set -e quite a lot. There is a very large and useful BashFAQ describing ways that set -e doesn’t do what authors might expect: https://mywiki.wooledge.org/BashFAQ/105

                  Thanks. This is great learning for me. Fortunately, I switch to Python before such complications usually arise but you are right, one should not blindly add “set -e” to an existing BASH script they didn’t write.

                1. 3

                  I’m a big fan of luadns.com. They have a simple Lua based DSL for authoring zone data and the records are kept in Git. It isn’t so expensive, either.

                  1. 2

                    This is a great, favorite trick of mine. It has some surprising edge cases, however! For example:

                    [grahamc@Petunia:~]$ cat <(date)
                    Sat Dec 14 21:08:17 EST 2019
                    
                    [grahamc@Petunia:~]$ sudo cat <(date)
                    cat: /proc/self/fd/13: No such file or directory
                    
                    1. 1

                      It wasn’t there already?? I’ve assumed that it was merged like back when the hype about it started..

                      1. 3

                        To my knowledge they had quite long arguments about including Zinc in the kernel or not. The author of WireGuard wanted to bring Zinc into the kernel, the kernel maintainers did not want to include it and instead told WireGuard to use the already existing crypto functionality. To my knowledge the agreement now is that they will not bring Zinc into the kernel.

                        1. 3

                          So Wireguard will require some out of tree module(s) for Zinc, or is Wireguard now using in-tree crypto functionality instead of Zinc?

                          1. 4

                            Some parts of Zinc were adopted in to the kernel, and some parts of Wireguard were updated to use the existing kernel APIs: https://www.phoronix.com/scan.php?page=news_item&px=Crypto-API-Doing-Some-Zinc

                            1. 1

                              Oh, thanks for finding/sharing that! Seems like a good compromise.

                      1. 2

                        I once wrote a much smaller version as a tutorial for writing Prometheus exporters :)

                        https://opensource.com/article/19/4/weather-python-prometheus

                        1. 1

                          Pretty cool! I also instrumented the service itself: https://weather.gsc.io/metrics

                        1. 3

                          The evaluation criteria are interesting:

                          it is appropriate for writing large programs

                          Some days I think it would be interesting to see a shell language designed for small, shell-size programs.

                          1. 1

                            I see this in Bash in almost every way. I wonder what you think would need to change for Bash to be more appropriate for only small programs?

                          1. 25

                            Who cares if the servers are in Iceland? Your business is not, and is subject to your local laws.

                            1. 1

                              True, but if the authorities come knocking and request data, it might be out of reach because it’s located in Iceland.

                              1. 22

                                If you are within reach, your data is within reach. The authorities will lock the business owner up until they provide the data.

                                1. 5

                                  Exactly. I made a similar argument about ProtonVPN being in Switzerland but having Americans in key positions. There’s some leverage right there.

                                2. 5

                                  If SimpleAnalytics is as privacy-friendly as it claims then it shouldn’t really matter, since the data they store can’t be used to identify individuals.

                                  1. 4

                                    That’s why the CLOUD Act was passed last year.

                                1. 42

                                  To quote a friend: To stop offering Mercurial hosting is bad. To delete the repositories is evil.

                                  1. 12

                                    I’m not sure about evil, but yeah this sounds bad. It doesn’t seem that either of these two options would be huge amounts of extra investment:

                                    • automatically convert hg repos to git repos
                                    • archive hg repos but keep serving a read-only mirror

                                    I wonder, does archive.org have the means to mirror the public bitbucket hg repositories?

                                    1. 14

                                      I do a lot of work around reproducible builds, and find the deletion of public source code to be quite severe. A lot of important projects don’t get regular maintenance, and it takes quite a lot of work to archive source. Converting to git means the inputs to the build process have changed. This might not be a huge deal for people today, but if you’re trying to rebuild something from a decade ago this is a serious problem.

                                      1. 6

                                        I do a lot of work around reproducible builds

                                        Don’t most big shops (Linux distros, Mozilla, Google) vendor the universe anyway? Specifically to avoid vanishing source code, or even minor network flakiness during build?

                                        1. 6

                                          I’m not sure what you mean by “vendor the universe”, but what I have seen is creation of private forks of public open source projects even if there is no intent to modify the code. This has two benefits.

                                          1. If the author or the hosting provider (e.g. bitbucket) deletes the repository, you still have access to it
                                          2. Performing a build only requires that one hosting provider be up rather than N
                                          1. 7

                                            Vendoring the universe means to, in your builds or deployments etc, pull all dependencies from source you control.

                                            1. 2

                                              “Vendoring” is the act of taking source code from your dependencies and including it in your own source tree. Doing that for the whole universe is what you do if you want to be sure that nobody else can break your build.

                                            2. 1

                                              Yes, Google has a third_party/ directory where mirrors of OSS code are stored. There is a team that works on the tools to keep things in sync.

                                          2. 4

                                            I think back to Gitorious and how they went down and everything they hosted is gone as well. That’s slightly different as the entire company folded, but there are still some things on there which probably didn’t exist anywhere else, which are now gone.

                                            I remember looking through my creative commons music once, finding a song I liked and trying to lookup the artist and see if they had other stuff. No only could I not find the artist, I couldn’t find the track! After some digging I found their old ccMixter account, from which they deleted all their tracks. The CC song I had literally didn’t existed anywhere I could find (at least under that named) that was indexed by Google/DDG or Bing.

                                            We look at how much new stuff is created each day. I wonder how much stuff is deleted forever.

                                            1. 1

                                              I think back to Gitorious and how they went down and everything they hosted is gone as well. That’s slightly different as the entire company folded, but there are still some things on there which probably didn’t exist anywhere else, which are now gone.

                                              Interesting in this context is the work by Guix and the Software Heritage to store all source archives/repositories used in Guix.

                                        1. 7

                                          Calling it “the real reason” is a bit of stretch (it’s more like “what I realized by accident after N years in the industry”), but the reason does actually sound quite compelling – to the point that I makes so much sense that struggled to remember why I preferred spaces all this time.

                                          I think the reason is alignment – I do things like

                                          $foo->bar()
                                              ->baz()
                                              ->etc(...)
                                          

                                          or

                                          sprintf("%s %s %s\n",
                                                  $foo
                                                  $bar
                                                  $baz);
                                          

                                          regularly, and spaces guarantee that these will actually show up consistently. You could have the cake and eat it by using tabs for indentation and spaces for alignment – which makes sense when you think about it, but somehow “feels dirty” to mix the two.

                                          1. 8

                                            tabs for indentation and spaces for alignment

                                            This sounds reasonable to me.

                                            but somehow “feels dirty” to mix the two

                                            I think I can learn not to fee dirty if I remember that I wont actually be mixing the two. The tabs are strictly in prefix.

                                            1. 7

                                              The tough part is that if you are writing Python, mixed tabs and spaces are illegal, and the standard style guide for the language specifies spaces only. Outside of Python, it feels like tabs for indentation and spaces for alignment requires editor support: at a minimum, visible whitespace, but preferably also for automating the number of tabs and number when reindenting.

                                              I have always been a partisan of spaces-only, but this is the first argument for tabs that I find really strong. I do feel like I need to work towards adopting tabs.

                                              1. 12

                                                The tough part is that if you are writing Python, mixed tabs and spaces are illegal

                                                “Indentation is rejected as inconsistent if a source file mixes tabs and spaces in a way that makes the meaning dependent on the worth of a tab in spaces; a TabError is raised in that case.” That is, you can use spaces for alignment if you consistently use tabs for scoping.

                                                e.g. This works

                                                $ cat -t x.py
                                                def a(x):
                                                ^Iprint("1",
                                                ^I      "2"
                                                ^I      "3")
                                                a(1)
                                                

                                                And python does not complain.

                                                $ python -t x.py
                                                1 23
                                                
                                                1. 3

                                                  Thanks, yes, you’re right. This leaves just the task of always maintaining perfect consistency in using tabs for scoping and spaces for alignment. Plus being willing to ignore the standard style guide (PEP 8) and avoid using code formatting tools like black.

                                            2. 4

                                              somehow “feels dirty” to mix the two.

                                              Why? Tabs are for indentation (semantic) whereas spaces are for alignment (visual). They’re different things.

                                              1. 1

                                                It’s “dirty” when different lines use different characters for indentation and alignment. e.g. line 1 uses 2 spaces for a single level of indentation, and line 2 uses 1 tab for a single level of indentation. (I’m not disagreeing with or discounting your point about having a rule to consistently abide by.)

                                                1. 2

                                                  That is a different situation than what was described and called “dirty”.

                                              2. 4

                                                The main downside to using tabs + spaces is editor support. Back when I used it, I think kate was one of the few editors that didn’t mess up my code. I eventually moved to spaces only out of frustration with different editors.

                                                The other downside is all the derision you get for using tabs.

                                                1. 2

                                                  You can’t use spaces for alignment for the same reason that tabs are apparently more accessible: different people will use tabs to mean different things. If you use space for alignment you are just moving the problem around, insisting that a tab is equal to N spaces – to keep your precious alignment correct.

                                                  1. 14

                                                    If you use tabs only for indentation, then your spaces will always line up. E.g. if you are 2 levels of indentation deep, and want to line up with the 3rd character, you have 2 tabs and 2 spaces. Regardless of tab length, 2 tabs and 2 spaces will end up at the same column as 2 tabs and 2 characters.

                                                    Rule of thumb is if you are visually lining up to a previous line, use spaces. If you are adding a level of indentation, use a tab.

                                                    1. 4

                                                      That makes sense, I’m wrong :)

                                                      I suppose a follow-up question is, does spaces for alignment satisfy the usability concern which tabs help with?

                                                      1. 2

                                                        That’s a good question. I would imagine the answer isn’t so much that spaces for alignment helps, it’s the tabs for indentation that help. Spaces for alignment satisfies the “consistent alignment” part of the problem.

                                                  2. 1

                                                    If you drop the first method call/arg to a new line, indented one level more you get a similar effect with tabs only. And you save some horizontal space when the var/symbol on the first line is longer than one tab-width.

                                                    JetBrains tools call this style “chop down” I believe.

                                                  1. 1

                                                    This is because of the recently announced package registry feature: https://github.com/features/package-registry

                                                    1. 1

                                                      Seems strange that you’d be able to log in to this whole second/nth copy, and visit issues and what-not: https://npm.pkg.github.com/NixOS/nixpkgs/issues/63839

                                                      1. 1

                                                        It seems to be just additional domain. And as it is additional domain that is not meant to be used as a web UI it do not shares cookies with the main domain. So only problem I see is that it is indexed (aka it do not specify canonical URL).

                                                    1. 2

                                                      suggest title: “LSTM: How to Train Neural Networks to Write Like Lovecraft”

                                                      1. 1

                                                        I have made that suggestion already.

                                                        1. 1

                                                          Is there a way to edit a post title ?

                                                          1. 2

                                                            If enough people suggest the same change, it will be applied automatically.

                                                            Mine was apparently the last one needed here. ;)

                                                            For future reference, you can ping one of the sysops (look them up on the hat list) to edit a post title.

                                                      1. 1

                                                        He touches in the beginning on a well-studied problem: the programs will have to run on systems that are partly broken and being repaired. If people want to study it more, the best examples I saw in that were in papers about the challenges of exascale computing. Here’s an example (pdf) that talks about resilience around page 52. Here’s an example showing how quickly failure accumulates as resources are massively scaled up:

                                                        “Requiringfloating point performance that is 1000×that of Roadrunner will thus require1000/64≈16 times as many CPUs and, consequently, 16 times as many CPUsockets. The growth of socket count for high performance computers over time isshown in Figure 4-21. The failure rate of such a system would be 16 times higherthan that of one of today’s petascale machines. For example, if a current petascalemachine has a reasonable mean time to system interrupt (MTSI) of 64 hours, anexascale machine would have an MTSI of merely 4 hours!”

                                                        I’m not sure what HPC clusters currently require in terms of maintenance, though.

                                                        1. 2

                                                          This is the crux of all the work he’s done with the Movable Feast Machine. His focus is on developing systems which continue to be effective through even massive system failures. I recommend checking out his other videos for more information on this work, and also his other channel, The T2 Tile Project where he is working towards hundreds of small computers each computing together, to build a robust system: https://www.youtube.com/channel/UC1M91QuLZfCzHjBMEKvIc-A

                                                          1. 1

                                                            I was having trouble following what was going on in the first video. That link has a lot of videos. Do you have one or more in his collection that are good intros?

                                                            Edit: Nevermind. Found the paper describing the work.

                                                        1. 8

                                                          I’ve said it before and I’ll say it again: ZFS should be the default on all Linux distros. It’s in a league of its own, and makes all other existing Linux filesystems irrelevant, bizarre licensing issues be damned.

                                                          1. 7

                                                            I use ZFS and love it. But I disagree that ZFS should be the default as-is. It requires a fair bit of tuning. For non-server workloads, the ARC in particular. ZFS does not use Linux’ buffer cache and while ARC size adapts, I have often seen on lower memory machines that the ARC takes too much memory at a given point, leaving too little memory for the OS and applications. So, most users would want to tune zfs_arc_max for their particular workload.

                                                            I do think ZFS should be available as an option in all Linux distributions. It is simply better than the filesystems that are currently provided in the kernel. (Maybe bcachefs will be a competent alternative in the future.)

                                                            1. 2

                                                              I agree.

                                                              I remember installing FreeBSD 11 once (with root on ZFS) because I needed a machine remotely accessible via SSH to handle files on an existing disk with ZFS.

                                                              No shizzle, FreeBSD defaults, the machine had 16G of RAM, and during an hours long scp run, ARC decided to eat up all the memory, triggering the kernel into killing processes… including SSH.

                                                              So I lost access, had to restart scp again (no resume, remember), etc. This is a huge show stopper and it should never happen.

                                                              1. 1

                                                                That seems like a bug that should be fixed. Don’t see any reason why that should prevent it from being the default though.

                                                              2. 1

                                                                That’s definitely something to consider, however, Apple has made APFS (ZFS inspired) the default on macOS, so there’s got to be a way to make it work for ZFS + Linux Desktop too. ZFS is all about making things work without you having to give it much thought. Desktop distros can pick reasonable defaults for desktop use, and ZFS could possibly make the parameter smarter somehow.

                                                              3. 2

                                                                I think the licensing issue is the primary problem for Linux distros.

                                                                1. 1

                                                                  I agree on technical superiority. What about the Oracle threat given its owner pulled off that API trick? Should we take the risk of all owing Oracle’s lawyers money in some future case? Or rush to implement something different that they don’t control with most of its strengths? I think the latter makes the most sense in the long-term.

                                                                  1. 3

                                                                    Oracle is not a problem, as the ZFS license is not being violated – it is the Linux license.

                                                                    1. 1

                                                                      “Oracle is not a problem, as the ZFS license is not being violated”

                                                                      That’s a big claim to make in the event large sums of money are ever involved. Oracle threw massive amounts of lawyers at Google with the result being API’s were suddenly a thing they could copyright. Nobody knew that before. With enough money and malicious intent, it became a thing that could affect FOSS developers or anyone building on proprietary platforms. What will they do next?

                                                                      I don’t know. Given they’re malicious, the safest thing is to not use anything they own or might have patents on. Just stay as far away from every sue-happy party in patent and copyright spaces. Oracle is a big one that seeks big damages for its targets on top of trying to rewrite the law in cases. I steer clear of their stuff. We don’t even need it, either. It’s just more convenient than alternatives.

                                                                      1. 8

                                                                        The CDDL, an OSI-approved open source licensed, includes both a copyright and patent grant for all of the code released by Sun (now Oracle). Oracle have sued a lot of people for a lot of things, but they haven’t come after illumos or OpenZFS and there are definitely companies using both of those bodies of software to make real money.

                                                                        1. 2

                                                                          I think you’re missing the implications of they effectively rewrote the law in the case I referenced. If they can do that, it might not matter what their agreements say if it’s their property. The risk might be low enough that it never plays out. One just can’t ever know if they depend on legal provisions with a malicious party that tries to rewrite laws in its favor with lobbyists and lawyers.

                                                                          And sometimes succeeds unlike basically everyone doing open source and free software. Those seem to barely enforce their agreements and/or be vulnerable to patent suits in case of the permissive licenses. Plus, could the defenders even afford a trial at the current rates?

                                                                          I bet 10 years ago you wouldn’t have guessed a mobile supplier using an open-ish platform would be fighting to avoid giving over $8 billion dollars to an enterprise-focused, database company. Yet, untrustworthy dependencies let that happen. And we got lucky it was a rich company that depended on OSS/FOSS stuff defending. The rulings could’ve been worse for us if it wasn’t Google.

                                                                          1. 6

                                                                            Seeing as Sun gave ZFS away before Oracle bought it, Oracle would have a LOT of legal wackiness to get the CDDL license revoked somehow. But for the safe of argument, let’s assume they do manage somehow to make it invalidated, and went nuts and decided to try and charge everyone currently using ZFS pay bajillions of dollars for “their” tech. Laws would have to change significantly for that to happen, and with such a significant change in current law, there is basically zero chance it would be retro-active from the moment you started using ZFS, so worst case you’d have to pay from the time of the law change. That is if you didn’t just move off of ZFS after the law changed and be out zero dollars.

                                                                            Also, the OSS version of ZFS is significantly different from Oracle’s version that they are sort of kissing cousins at best anymore. ZFS has been CDDL licensed since 2005, so a long history of divergence from the Oracle version. I think Oracle would have a VERY hard time getting the OSS version back under the Oracle banner(s). Even with very hypothetical significant law changes.

                                                                            I’m in favour of things competing against ZFS, but currently nothing really does.. BTRFS tries, but their stability record is pretty miserable for anything besides the simplest workloads. ZFS has had wide production usage since 2001. Maybe in another 5 or 10 years we will have a decent stable competitor to some of ZFS’s feature-sets.

                                                                            But regardless if you are a large company with something to lose, your lawyers will be the ones advising you about using ZFS or not, and Canonical’s lawyers clearly decided there was nothing to worry about, Along with Samsung(who own Joyent, the people behind Illumos). There are also many other large companies that have bet big on Oracle having basically zero legal leg to stand on.

                                                                            Of course the other side of the coin is the ZFS <-> Linux marriage, but that’s easy just don’t run ZFS under Linux, or use the Canonical shipped version and let Canonical take all the legal heat.

                                                                            1. 2

                                                                              Best counterpoints so far. I’ll note this part might not be as strong as you think:

                                                                              “and Canonical’s lawyers clearly decided there was nothing to worry about, Along with Samsung(who own Joyent, the people behind Illumos)”

                                                                              The main way companies dodge suits is to have tons of money and patents themselves to make the process expensive as hell for anyone that tries. Linux companies almost got patent sued by Microsoft. IBM, a huge patent holder, stepped up saying they’d deal with anyone that threatened it. They claimed they were putting a billion dollars into Linux. Microsoft backed off. That GPL companies aren’t getting sued made Canonical’s lawyers comfortable but not an actual assurance. Samsung is another giant, patent holder with big lawyers. It takes an Apple-sized company to want to sue them.

                                                                              So, big, patent holders or projects they protect are outliers. That might work to ZFS’s advantage here. Especially if IBM used it. They don’t prove what will happen with smaller companies, though.

                                                                              1. 2

                                                                                I agree with you in theory, but not in practice because of the CDDL (which ZFS is licensed under). This license explicitly grants a “patent peace” see: https://en.wikipedia.org/wiki/Common_Development_and_Distribution_License

                                                                                I know most/many OSS licenses sort of wimp out on patents and ignore the problem, CDDL doesn’t. Perhaps it could have even stronger language, and there might be some wiggle room for some crazy lawyering.. I just don’t really see Oracle being THAT crazy. Oracle, being solely focused on $$$$, would have to see some serious money bags to go shake loose, I doubt they would ever bother going after anyone not the size of a Fortune 500, the money just isn’t there. Google has giant bags full of money they don’t even know what to do with, so Oracle trying to steal a few makes sense. :P

                                                                                Oracle going after Google makes sense knowing Oracle, and it was , like you said, brand new lawyering, trying to create API’s out of Copyrights. Patents are not remotely new. So some lawyer for Oracle would have to dream up some new way to screw up laws to their advantage. Possible sure, but it would be possible for any other crazy lawyer to go nuts here (wholly unrelated to ZFS or even technology), it’s not an Oracle exclusive idiocy. Trying to avoid unknown lawyering that’s not even theoretical at this point would be sort of stupid I would think… but I’m not a lawyer.

                                                                                1. 1

                                                                                  “I know most/many OSS licenses sort of wimp out on patents and ignore the problem, CDDL doesn’t.”

                                                                                  That would be re-assuring on patent part.

                                                                                  “Possible sure, but it would be possible for any other crazy lawyer to go nuts here (wholly unrelated to ZFS or even technology), it’s not an Oracle exclusive idiocy. Trying to avoid unknown lawyering”

                                                                                  Oracle was the only one to flip software copyright on its head like this. So, I don’t think it’s an any company thing. Either way, the threat I’m defending against isn’t unknown lawyering in general: it’s unknown lawyering of a malicious company whose private property I may or may not depend on. When you frame it that way, one might wonder why anyone would depend on a malicious company at all. Avoiding that is a good pattern in general. Then, the license negates some amount of that potential malice for a great product with unknown, residual risk.

                                                                                  I agree the residual risk probably won’t affect individuals, though. An Oracle-driven risk might affect small to mid-sized businesses depending on how it plays out. Good news is swapping filesystems isn’t very hard on Linux and BSD’s. ;)

                                                                        2. 4

                                                                          AFAIK, it’s the GPL that’s being violated. But I’m really tired and the SFC does mention something about Oracle suing so 🤷.

                                                                          Suing based on the use of works derived from Oracle’s CDDL sources would be a step further than the dumb Google Java lawsuit because they haven’t gone after anyone for using OpenJDK-based derivatives of Java. Oracle’s lawsuit-happy nature would, however, mean that a reimplementation of ZFS would be a bigger target because it doesn’t have the CDDL patent grant. Of course, any file system that implements one of their dumb patents could be at risk….

                                                                          I miss Sun!

                                                                    2. 1

                                                                      What does ZFS have that is so much better than btrfs?

                                                                      I’m also not sure these types of filesystems are well suited for databases which implement their own transactions and COW, so I’m not sure I would go as far as saying they are all irrelevant.

                                                                      1. 11

                                                                        ZFS is extremely stable and battle-tested, while that’s not a reason in itself to make it a better filesystem, it makes it a extremely safe option when what you’re looking for is something stable to keep your data consistent.

                                                                        It is also one of the most cross-platform file system. Linux, FreeBSD, MacOS, Windows Illumos. It has a huge amount of development behind it, and as of recently the community has come together significantly across the platforms. Being able to export your pool on FreeBSD and import it on Linux or another platform makes it a much better option if you want to avoid lock-in.

                                                                        Additionally, the ARC

                                                                        Problems with btrfs that make it not ready:

                                                                        1. 0

                                                                          If I don’t use/want to use RAID5 then I don’t see the problem with btrfs.

                                                                          1. 3

                                                                            I ran btrfs in production on my home server for ~3-4 years, IIRC. If you want to use btrfs as a better ext4, e.g. just for the compression and checksumming and maybe, maybe snapshotting, then you’re probably fine. If you want to do anything beyond that, I would not trust it with your data. Or at the very least, I wouldn’t trust it with your data that’s not backed up using something that has nothing to do with btrfs (i.e. is not btrfs snapshots and is not btrfs send/receive).

                                                                            I had three distinct crashes/data corruption problems that damaged the filesystem badly enough that I had to back up and run mkfs.btrfs again. These were mostly caused by interruptions/power failures while I was making changes to the fs, for example removing a device or rebalancing or something. Honestly I’ve forgotten the exact details now, otherwise I’d say something less vague. But the bottom line is that it simply lacks polish. And mind you, this is from the filesystem that is supposed to be explicitly designed to resist this kind of corruption. I know at least the last case of corruption I had (which finally made me move to ZFS) was obviously preventable but that failure handling hadn’t been written yet and so the fs got into a state that the kernel didn’t know how to handle.

                                                                        2. 3

                                                                          well, I don’t know about better, but ZFS has the distinct disadvantage of being out of tree filesystem so it can and will break depending completely on the whims of kernel development. How anyone can call this stable and safe for production use is beyond me.

                                                                          1. 3

                                                                            I think the biggest argument is mature implementations used by large numbers of people. That catches lots of common and uncommon problems. In reliability-focused filesystems, that the reliability is field-proven then constantly maintained is more important to me than about anything. The only reason I don’t use it is that it came from Oracle with all the legal unknowns that can bring down the line.

                                                                            1. 3

                                                                              When you say “Oracle”, are you referring to ZFS or btrfs? ;)

                                                                              1. 1

                                                                                Oh shit! I didn’t know they designed both! Glad I wasn’t using btrfs either. Thanks for the tip haha.

                                                                            2. 2

                                                                              On a practical level, ZFS is a lot more tested (in Solaris/Illumos, FreeBSD, and now Linux); more different people have put more terabytes of data in and out of ZFS than they seem to have for btrfs. This matters because we seem to be unable to build filesystems that don’t run into corner cases sooner or later, so the more time and data a filesystem has handled, the more corner cases have been turned up and fixed.

                                                                              On a theoretical level, my personal view is that ZFS picked a better internal structure for how its storage is organized and managed than btrfs did (unless btrfs drastically changed things since I last looked several years ago). To put it simply, ZFS is a volume manager first and then a filesystem manager second (on top of the volumes), while btrfs is (or was) the other way around (you manage filesystems and volumes are a magical side effect). ZFS’s model does more (obvious) violence to Linux IO layering than I think btrfs’s does, but I strongly believe it is the better one and gives you cleaner end results.

                                                                            3. 0

                                                                              Why would I want to run ZFS on my laptop?

                                                                              1. 1

                                                                                Why wouldn’t you want to run it on your laptop?

                                                                            1. 3

                                                                              Suggest correcting Depreciation to Deprecation.

                                                                              Suggest correcting to Amazon S3 Path Deprecation Plan – The Rest of the Story to match the exact title (as michrider points out nicely.)

                                                                              1. 2

                                                                                Also it’s only S3 paths being deprecated, not S3 entirely.

                                                                                1. 2

                                                                                  Folks please consider clicking the “Suggest” link right below the title above and making your suggested changes :)

                                                                                  1. 2

                                                                                    The way I interpret these “Suggest” comments going is I make a suggestion through that link, and readers who agree take my hint and also go through the “Suggest”

                                                                                    1. 2

                                                                                      Indeed. And for something like tags, it serves as an opportunity to help educate new users. :)

                                                                                      1. 1

                                                                                        I thought the suggestions went to the moderators who then either make the changes or not.

                                                                                        1. 4

                                                                                          Not sure. My impression was if several people make the suggestion, it was applied automatically – but I have no idea.

                                                                                          edit: https://lobste.rs/moderations you can see some changes are automatic.

                                                                                        2. 1

                                                                                          Uh, personally, I didn’t even know/realize there’s a “suggest” link/button! :/ TIL; that said, for education of others like me, @grahamc what would you think about e.g. linkifying the “Suggest” word in future? Say, something like: “Suggest correcting…” — or, as @feoh elaborated, “Folks please consider clicking “Suggest” and correcting to…”

                                                                                    1. 20

                                                                                      Well, this is infuriating. I hate that my browser just became essentially useless to me because someone at Mozilla messed something up. Anyone know if there’s a way to opt out of the extension verification stuff?

                                                                                      1. 11

                                                                                        I’m seriously considering just switching to Chromium (ungoogled-chromium maybe?) as a workaround. I don’t feel like Mozilla is doing too well in general with regards to being pro-user and pro-privacy lately;

                                                                                        • There’s this issue, leaving everyone vulnerable to tracking and disabling protections for tor users.
                                                                                        • The fact that this feature exists at all, and the only supported way to disable signing requires nightly, takes a lot of control out of users’ hands.
                                                                                        • Mozilla have bought companies with closed source products (such as pocket), integrated them into Firefox, promised to open-source those products, and just never open-sourced them, leaving Firefox still with built-in integrations with potentially privacy-breaching inauditable closed-source products.
                                                                                        • They have plans to move away from DNS, where a query first consults my OS (and its hosts file) and then consults my ISP which is a norwegian company following strong privacy laws, to just sending queries directly to a random American company which follows the US’ seemingly non-existant privacy laws.
                                                                                        • It seems likely that they’ll move from IRC to discord or slack, which will be pretty bad if it happens (though this point is invalid if they end up moving to something free and open source). They should at least have come out and clearly stated that they’re not moving to a closed-source solution.
                                                                                        • And, well, Chromium just has better performance on machines I’ve tested it on; having a worse experience for a good cause is worth it, but having a worse experience just to support a company which doesn’t really stand for anything might not be.

                                                                                        I honestly really want to support Mozilla, and to do my small part in avoiding a complete browser monopoly by not using chromium, and I really don’t want to support Google. Mozilla just does so many stupid things which flies in the face of the values they claim to hold.

                                                                                        1. 27

                                                                                          I definitely plan to stay with Firefox. They are sometimes failing, but at least they’re trying to fight. There’s a saying, that “if somebody’s not failing, they’re not trying hard enough”. Whereas Chrome has a fundamental conflict of interest against many user protection mechanisms, because paid by Google Ads.

                                                                                          1. 7

                                                                                            You’re probably right. I’m on chromium right now, but I will probably honestly end up switching back to Firefox when this whole thing is over. It just sucks that Mozilla has to put themselves in the position of being the least bad of two evils, instead of just being plain good.

                                                                                            1. 4

                                                                                              It just sucks that Mozilla has to put themselves in the position of being the least bad of two evils, instead of just being plain good.

                                                                                              You’ve hit the nail on the head. I just want a browser that’s privacy-respecting and good.

                                                                                            2. 5

                                                                                              Mozilla is also paid by Google Ads.

                                                                                            3. 20

                                                                                              Can you not be a drama llama? They goofed up. They will probably fix it soon. So you are without addons for a few days.

                                                                                              As for their decisions, they are clearly straddling a line between purity and a little bit of the dirty stuff to make it more convenient for the non-0.1% of users who are ‘technical’. Meanwhile Google is ACTIVELY TRYING TO FUCK YOUR SHIT UP to maximise their control and profit.

                                                                                              Perfect is the mortal enemy of the good.

                                                                                              1. 2

                                                                                                I think the problem here is that not only do they enforce the signing, but they also make it impossible for the user to turn it off, unless the user downloads non-stable or non-official versions of software, taking control out of the hands of the user.

                                                                                                Sure, Google is worse, but what excuse does Mozilla have for the workaround (e.g., disabling the feature) not working on stable versions of Firefox? I see that as the very definition of the lesser of the two evils.

                                                                                                1. 3

                                                                                                  I think I’ve seen some article long ago, basically saying how users will do everything they’re told by a website if this means they get to watch one more funny cat video - including changing settings in about:config, in OS, etc. Unfortunately I can’t seem to find the article with google nor ddg.

                                                                                                  1. 3

                                                                                                    This rings a bell, I read that too. I think the term you are looking for is “dancing pigs”. The Wikipedia page for dancing pigs cites a few sources for it. The one I think you and I both read is probably one of the Bruce Schneier articles. Wiki suggests the first publicly available written thing using the term was a chapter in a book about the Java security model. Which is kind of funny when one thinks about it because it’s hard to think of a piece of technology that did a worse job of what it was supposed to do than the Java security model.

                                                                                                    1. 1

                                                                                                      You’re saying the users are the only one gullible here?! What about the developers? A couple of folks at Mozilla and Google tell devs to trust LetsEncrypt with all your SSL needs, and pretty much every single developer restricts access to their websites now through LetsEncrypt now. Talking about the folks being gullible!

                                                                                                      1. 1

                                                                                                        Hm, I see now that the way I wrote it may be seen as more ambiguous than I expected! :) Basically, what I meant, and what the article I mention tried to convey AFAIR, was that as a developer, one sometimes needs to protect users from themselves; in this case, I guess the “[Mozilla] mak[ing] it impossible for the user to turn [addon signing verification] off” decision might have been to protect users from themselves. That is, to protect users from being conned into disabling the verification feature “to see this one funny cat video”, and installing some malware addon.

                                                                                                        As to LetsEncrypt, I don’t think I want to engage in a discussion completely (in my opinion) unrelated to the original post/article :)

                                                                                                  2. 1

                                                                                                    this isn’t the only thing they’ve done. it’s part of a longer trend of user-hostility which tells us that the mainstream web will not be compatible with freedom, so long as google controls what standards are implemented.

                                                                                                  3. 5

                                                                                                    Mozilla just does so many stupid things which flies in the face of the values they claim to hold.

                                                                                                    Yeah, remember that “auto install” of the LookingGlass/Mr.Robot thing a while back (end of 2017 I think…)?
                                                                                                    wtf Mozilla. I am going to check out some alternatives.

                                                                                                    Anyone here tried Brave or Vivaldi? If so, any good?

                                                                                                    1. 3

                                                                                                      Been working with Brave and Firefox for quite some time now.

                                                                                                      Brave is less polished and is missing quite a lot of sync-related-features I tend to use quite often on firefox. But the fact that firefox broke at a critical moment on my smartphone, right this morning, was the turning point.

                                                                                                      I haven’t tried Vivaldi as extensively though.

                                                                                                    2. 3

                                                                                                      The fact that this feature exists at all, and the only supported way to disable signing requires nightly

                                                                                                      https://twitter.com/SwiftOnSecurity/status/1124545069078536192

                                                                                                      There’s no solution here that doesn’t involve making normal users more vulnerable to malware. It’s been tried.

                                                                                                      Chrome has had similar problems in the past.

                                                                                                      They have plans to move away from DNS …. to just sending queries directly to a random American company

                                                                                                      Nobody has said that it will be a random American company. Mozilla’s testing this feature out with Cloudflare. I suspect this will be pretty configurable if it becomes an actual thing, and probably more local.

                                                                                                      It seems likely that they’ll move from IRC to discord or slack

                                                                                                      Mozilla’s moving away from IRC, but from the requirements here it doesn’t seem like slack or discord are likely solutions.

                                                                                                      1. 2

                                                                                                        Nobody has said that it will be a random American company. Mozilla’s testing this feature out with Cloudflare.

                                                                                                        Cloudflare is the random American company I’m talking about.

                                                                                                        1. 2

                                                                                                          Right, operative term being “testing this feature out”. There’s no indication that if this feature becomes a thing it will be only cloudflare that it uses. There’s just a lot of FUD around it.

                                                                                                          My comment is not correcting “random American company” to cloudflare, it is correcting your statement about Mozilla plans around this. They have not ever stated that this is the plan. It’s just what they’re testing out, because you have to bootstrap an ecosystem somehow.

                                                                                                      2. 1

                                                                                                        Mozilla isn’t moving away from DNS, you can disable DoH in the network settings and you can set any other DoH endpoint you want in the same dialog (so for example, you could set your Norwegian ISP or no DoH at all).

                                                                                                        The Pocket extension is open source to my knowledge, I do recall a github repo floating around. What isn’t open source (yet) is the backend.

                                                                                                        1. 4

                                                                                                          Sure, it will probably be possible to disable DoH, but how many non-American Firefox users will know to do that, compared to how many will not even know it’s something they have to worry about and send all their queries to a US corporation?

                                                                                                          The pocket extension is open source, but it’s the backend which is interesting, and it’s the backend they promised to open-source a long time ago. (Look at this comment from a Mozilla employee 2 years ago: https://www.reddit.com/r/firefox/comments/5wio45/mozilla_acquires_pocket/deadcf7/ - that didn’t say that the Pocket extension would become open source, but Pocket itself.)

                                                                                                          1. 1

                                                                                                            To my knowledge the current default and to keep it disabled, the DoH provider setting currently defaults to only using standard DNS as well, I don’t know of any plans to change that, Mozilla is still very early in testing the waters on how to deploy it.

                                                                                                      3. 2

                                                                                                        See the description of this post for a workaround.

                                                                                                        1. -9

                                                                                                          Well, this is infuriating. I hate that my browser just became essentially useless to me because someone at Mozilla messed something up. Anyone know if there’s a way to opt out of the extension verification stuff?

                                                                                                          LOL, says a person who’s website is “protected” by a time-bombed HTTPS and is unavailable via HTTP.

                                                                                                          You are aware that your website suffers from the same issues that you appear to condemn in this very comment? That it’s up to external third parties on whether or not the user is allowed to see it, because you decided to cave in to their pressure to “secure” your static website, and yourself made a choice to prohibit folks from accessing it via HTTP through your own policy?

                                                                                                          How are you then act surprised that Mozilla does same?!

                                                                                                          1. 6

                                                                                                            Well firstly, my website is not a tool that people depend on to do work. Firefox is. Secondl, I have automated systems in place to renew the SSL certs & get warned when they’re near to expiry. Thirdly, if you had my site open & the certs somehow expired, you could still see the content; Firefox just disabled a bunch of functionality while it was running without giving me any chance to intervene. Finally, if a website’s certificates are expired, you still have the ability to say “show me anyway”; there doesn’t seem to be any ability to do the same with stable Firefox.

                                                                                                            Glad to see you’re enough of a fan of mine to look into how I configure my site though!

                                                                                                            1. 2

                                                                                                              But how’s a website different from a tool? Firefox is still made by people just like you. The fact that one can click “show me anyway” on your website is merely omission on the part of site’s operator to not install HSTS. With proper HSTS, the user is guaranteed to not have any way to access your site even if you decide to cancel your https policy. There is no way to intervene, either, if HSTS is setup correctly. If you click reload and a new connection has to be established, pretty certain things won’t work no more, either.

                                                                                                              “Automated systems in place to renew SSL certs”? Are they autonomous and self-contained, or do they depend on any third parties? Are the third-parties they depend upon by any chance related to the very same party that caused the incident at stake? Isn’t Mozilla the biggest backer behind LetsEncrypt? This has got to be a joke! The most classic example of #TooBigToFail!

                                                                                                              1. 2

                                                                                                                Firefox is only a tool you depend on because people serve websites which require a modern browser to be usable. HSTS contributes to this monoculture.

                                                                                                              2. 3

                                                                                                                HTTPS is a bit different; with a website, you’re inherently relying on someone else paying the bills for the server and domain name continuously anyways, and if they don’t, you can’t use their website even if it”smnot HTTPS. Relying on the owner to renew their certs too doesn’t really change anything. If you want to have access to a website without relying on anyone else, you need to download it and access it locally, whether it’s HTTP or HTTPS.

                                                                                                                There’s no such expectation for addons I have downloaded to my personal machine which don’t inherently need to rely on any third-party.

                                                                                                                1. 6

                                                                                                                  This is a personal attack and not something that contributes to the conversation.

                                                                                                                  1. 0

                                                                                                                    How’s something a personal attack if it applies to pretty much every site operator nowadays? The comment purposefully doesn’t even contain any PII, either.

                                                                                                                    1. 2

                                                                                                                      There are better ways of discussing the merits and problems involved with the https certificate system than dismissing what someone said with “LOL, says a person who [..]” and doubting the person’s sincerity with “issues that you appear to condemn”.

                                                                                                                      1. -1

                                                                                                                        the dismissal or questioning of their sincerity is something you’re adding with your interpretation. it doesn’t follow from the parts you quoted.

                                                                                                                        maybe his goal was not to discuss the merits and problems of the https certificate system, but to actually lessen the spread of this scourge.

                                                                                                                    2. -3

                                                                                                                      Pointing out hypocrisy is a good tool when discussing moral issues.

                                                                                                                    3. -4

                                                                                                                      Good post, sad to see it got swarmed by haters.

                                                                                                                  1. 7

                                                                                                                    You can add Nix to the list of offenders. I actually did read the script first (okay, fine, I skimmed it) and I noticed that it at least has protection against the “partial content” problem the OP mentions. The entire script is wrapped in { and }, which means that if it ends up only half-downloaded you’ll get a syntax error and none of the commands will be run.

                                                                                                                    Overall, though, yeah—I agree that this is a disturbingly common practice.

                                                                                                                    1. 15

                                                                                                                      Yeah, it is true. I don’t love that Nix uses a curl|sh install. However. Something interesting about this list is three out of the five concerns (hidden text, user agent sniffing, not knowing what the script will do) are predicated on the remote server being malicious. As you noticed we’re not subject to the partial content problem, and the fifth (not using TLS) is a complete failure of applying basic security.

                                                                                                                      One alternative was publishing instructions which included sha256 verification. However if you trust the website to give you the right hash, you can trust the website to give you the right script.

                                                                                                                      Another alternative is publishing GPG verification installations, which is an option listed immediately below the curl|sh instruction, in bold text, on https://nixos.org/nix/ (click Get Nix.) But still, unless you have a WoT connection to Eelco (who signs our releases and also invented Nix) what does it even mean?

                                                                                                                      Going back to the list of concerns – the only ones at concern here is not knowing the script (yes, please do download and read the script) or not trusting Nix’s install code in the first place. Frankly, if you don’t trust nixos.org to distribute a clean and safe install, you probably don’t want to use Nix at all.

                                                                                                                      I’m not thrilled at the article saying that by simply having these instructions present that we are cutting corners – we’ve pretty carefully considered the implications of this mechanism of installation and have explored alternatives. Yes, being packaged in a distro’s package manager is a great route for many pieces of software. Many distributions provide packages for Nix at this point, but for some time Nix’s /nix directory was a non-starter for breaking FHS.

                                                                                                                      If you have suggestions on how to improve the security of the Nix installation, I would be happy to hear about it.

                                                                                                                      1. 1

                                                                                                                        All you need to do is change the bit that says curl https://nixos.org/nix/install | sh to just be a link to the script https://nixos.org/nix/install. Telling people that they can download it and use it to install nixos.

                                                                                                                        1. 2

                                                                                                                          I don’t think this makes any meaningful difference. If the user doesn’t check any checksums or signatures or read and understand the script, then you’ve just made the user part of the automated process. If the user is savvy enough to do a meaningful check on the script before executing it, they can see curl https://nixos.org/nix/install | sh and decide to download the script to a file instead.

                                                                                                                          1. 1

                                                                                                                            the difference is that you’re explicitly encouraging and condoning a bad practice

                                                                                                                            1. 0

                                                                                                                              Do you read all Makefiles before you run make? Or all setup.pys before you run pip install?

                                                                                                                        2. 1

                                                                                                                          Everything you say is true. And please understand that I wasn’t trying to slag on Nix so much as point out that it’s another prominent project that uses this pattern.

                                                                                                                          However if you trust the website to give you the right hash, you can trust the website to give you the right script.

                                                                                                                          Totally fair point.

                                                                                                                          unless you have a WoT connection to Eelco… what does it even mean?

                                                                                                                          I don’t have a WoT connection to the guy who signs Nginx releases, either. In fact, his key is the only key in my GPG keychain on my web server. The important thing is that every release is signed with that key. If I downloaded a release and found that it was signed by a different key that purported to be from the same person, I would hold off on that version until I could figure out what was going on. (I know this is a much weaker form of security than PGP was supposed to give us, but it’s what I’ve got.) Of course, the same is true of Nix.

                                                                                                                          My main worry with curl | sh installs is that they usually redirect to some GitHub URL that points to the project’s master branch. All it would take to compromise this installation procedure would be for one bad commit to make it to master somehow. Some projects merge to master a lot. Maybe the only actionable advice I have here (and I don’t know whether this is applicable to Nix or not) is that if you’re going to offer a download script to be piped into sh, host it somewhere that is immune from random, malicious PRs on GitHub.

                                                                                                                          By the way, I did install and start using Nix. I recognize that if you wanted to install malware on my machine, you have several more subtle options at your disposal than serving me a bad install script :-) The same goes for projects like Docker. What baffles me is ordinary programs like Calibre (which I do use) recommending piping wget --no-check-certificate into sudo sh 😱

                                                                                                                      1. 4

                                                                                                                        No. I don’t feel comfortable using Mastodon until it is very easy to put my account on my own domain. Being tied to a host feels counter to the goals of federation. There are some hosted mastodon-on-your-domain systems, but they’re more expensive than I’m interested in paying. I plan on staying away from Mastodon until either:

                                                                                                                        • accounts can be moved
                                                                                                                        • it becomes very cheap to get a hosted version
                                                                                                                        • it becomes trivial to run a self-hosted version
                                                                                                                        1. 1

                                                                                                                          I hear that pleroma is a lot easier to self-host than mastodon. Have you looked into that?

                                                                                                                        1. 4

                                                                                                                          I wish there was a way to simply choose a set of programs I need (say for example, jupyter with a set of libraries) and download a nixos configuration that I can simply apply to a virtual machine, and get a machine exactly according to what I need (with sane defaults). Does something like this exist now?

                                                                                                                          1. 6

                                                                                                                            I haven’t done this myself, but I’ve seen this talked about on several occasions. I get the impression that you can build a bootable iso from any given config, or you can build directly into a virtualization target.

                                                                                                                            Relevant links:

                                                                                                                            1. 2

                                                                                                                              This is such a cool idea! I at one point built a “try nixos” service where you got 1hr of free NixOS-in-a-VM, but I stopped working on it due to abuse concerns. Maybe it could live on in this way.

                                                                                                                              1. 1

                                                                                                                                This is the question I also wanted to ask here, glad I am not the only one. Is there some way of generating secure configs somewhere, based on your needs? I tried to use Nixos like five times or so, but I always have the feeling I miss entries in the config file that make it less secure compared to my normal hardening of distro’s like CentOS and Debian that I know very well. I don’t know what gets done automagically (quite a bit it seems!) And what not.

                                                                                                                                And I can find and use other’s config files of course, but who says they are any good? I really think some documented (and up to date) wiki/site/library of sane default examples for a lot of different use cases (i.e. apache/webserver/security headers, database/mysql, certbot/let’s encrypt etc. etc.) would greatly benefit users like me.

                                                                                                                                But maybe that already exists and I can just not find it. So, if someone knows of something like this, let us know :)

                                                                                                                                1. 3

                                                                                                                                  There’s generally the wiki, at: https://nixos.wiki/ (officially unofficial… but unofficially it’s the main one …kinda like in: the only one). There’s also https://nixos.org/nixos/security.html. But given that the NixOS community is not super large in itself, the security sub-community seems even smaller. I think you could try asking on the NixOS security discourse if you have some detailed questions; but given the slow traffic, it may be hit or miss. Or just asking on the main NixOS discourse.