Threads for zk

  1. 9

    I’m still sad about the state of messengers. One one hand you have Telegram and Signal which are two walled gardens with a closed-source server. (Signal being slightly worse as the author does not want it to be packaged for fdroid, and released a few versions without publishing the source code prior to integrating his cryptocurrency-scam in it.)

    On the other hand, the decentralised alternatives are miles behind WhatsApp/Telegram/Signal. Jami just does not work for me and my friends (messages are not going through) and is a battery killer. (Messages are not being received) Tox has one of the worst UI/UX I’ve ever experienced (let alone push notifications being broken) And Matrix’s support for E2EE is not widespread.

    I appreciate the community effort, but unfortunately I still use Telegram :( .

    1. 25

      I’m not sure it’s fair to place Telegram and Signal in the same category, at least not if E2EE is important to you.

      The vast majority of Telegram’s chats are stored in plaintext on their servers. Group chats cannot be end-to-end encrypted, and you have to manually start a secret chat between two people if you want E2EE. In spite of this, Telegram continues to be seen as a secure messenger – which is why a 2017 study found that many Telegram users mistakenly think their chats are end-to-end encrypted when they’re not.

      1. 2

        I did not realise this and now it makes sense to me that they can delete group chats.

        1. 2

          Yes. Telegram has to be used with over mitts, I agree. I don’t do any group chat, and only secure chats.

        2. 10

          I have no expectation that Telegram is actually secure. I just find it moderately interesting it’s one of the few modern IM services with open client APIs.

          1. 9

            I’m with you on the Telegram and Signal, though I still use both for some contacts. What has been working better for me, though, was setting up a hosted Snikket instance and then I got my family using that. It’s XMPP/Jabber (with OMEMO for the e2ee, which is based on the Signal protocol), and actually has good mobile clients. It’s one of the few ways the iOS family members get consistent notifications and experience. Might be worth a try if you’re ever up for trying another one.

            1. 8

              matrix works great for me.

              1. 12

                People keep saying this, but I don’t know when it’ll actually be the case. Every time I try it, I get disappointed by mediocre clients, resource hogging server, and inadequate moderation tools for groups.

                1. 6

                  I tried matrix about 3 years ago I think, it worked, but it wasn’t great. In fact I self-hosted a matrix server, and while playing it filled my server with garbage from IRC-bridges, etc… So after one year or so, I uninstalled matrix, and just used a self-hosted IRC server, and for my group of geeky friends it was ok.

                  Then, after a while, IRC show some limits, so we’ve tried to use matrix again, about 1 year ago, and there were a lot of progress. The client is a lot better, the server too, I just forgot about it, and it “just works”. The e2ee user interface is a lot better than a few years ago.

                  So if you haven’t tried matrix in a while you should really try it again. Really, now I think matrix is ready.

                  1. 4

                    resource hogging server

                    conduit.rs has been running flawlessly for me, for a couple months now - way less resource-hungry than the Synapse stack (albeit, with some missing features).

                    1. 2

                      If you haven’t tried element in, say the last 3-4 months, then you aren’t up to date.

                      1. 7

                        I have and it’s still not great. The interface is a bit confusing (i.e. the multiple levels of hierarchy for chat, the weird spaces thing that’s trying to paper over federation, etc.) and lacks this je ne sais quoi other clients have that makes it feel wrong for lack of a better term.

                    2. 6

                      just wanted to say: matrix/element has come a long way. Due to it being used across orgs without federation (work vs family) I’d say the biggest downside is that there is no recommended way for multiple accounts on one device (especially mobile). Otherwise it works great.

                      1. 5

                        It REALLY has. But sadly I think the fact that @acatton did a whole ‘state of messengers’ and didn’t even mention Matrix is a great example of how it’s gaining mindshare, but VERY slowly and not with non technical users.

                        The clients are IMO finally ready for that to happen, now we just need more people to actually try the updated versions and figure out how good it can be.e

                        Meanwhile sadly 99% of the communities I actually want to chat on are Discord or Slack :(

                    3. 6

                      If you’re ever in the mood to try again, I suggest exploring Snikket.

                      1. 3

                        I use prosody xmpp server with Conversations from f-droid on mobile. It works great for me and my friends and family. It is also very fast and easy to setup. Server to server works flawlessly so I can connect with others easily and without creating an account.

                        1. 3

                          Can you provide more context or links around Signal’s cryptocurrency-scam that you mention? I could only locate a Verge article, but nothing more technical. Thanks.

                            1. 2

                              Thank you. Yes, it does.

                          1. 3

                            miles behind WhatsApp/Telegram/Signal.

                            I know it wasn’t your intention, but just to clarify, this makes it seem as if these three are of similar quality. However, Telegram is miles ahead of WhatsApp.

                            1. 2

                              Signal being slightly worse as the author does not want it to be packaged for fdroid

                              This is pretty valid though. F-droid’s whole build process is really strange. It also isn’t built with security in mind.

                            1. 5

                              Actual article on the subject this HackADay links to.

                              1. 2

                                Thanks for this!

                              1. 4

                                I would rather see better tools than PGP for new standards.

                                1. 4

                                  If you’re talking about GnuPG, then absolutely. Why can’t we have something simple like signify, eh? If you’re talking about OpenPGP then, well, even with its issues, a suitable replacement would still needs its own RFC first :^)

                                  1. 1

                                    Yes, makes sense.

                                    1. 1

                                      Yeah, Signify would probably be much better.

                                      1. 2

                                        Or reop, but you get the gist ;^)

                                        1. 1

                                          I’m surprised reop hasn’t caught on like signify has (first time I’m hearing of it). Seems like a great tool.

                                  1. 6

                                    I’ve been avoiding Brave because they’re not based on Gecko, but this is yet another thing starting to make me look that way.

                                    1. 4

                                      I love Mozilla and only use FIrefox and Thunderbird, but it has been more and more difficult to hold this position when I see their management continuously trashing their employees and their users.

                                      On the other side, I’m uneasy with Brave, they’re trying to fight Google by using a browser engine written and maintained at ~80% (my guesstimate) by Google…

                                      1. 3

                                        I still don’t know how to feel about Brave. On one hand their privacy features seem miles ahead of the competition. But on the other hand there’s their crypto currency stuff I’d rather have no part in.

                                        I’ve recently warmed up to Chromium (and by proxy Chromium-based browsers like Brave) because it’s sandboxing is so clearly much better than Firefox (even with FF’s recent improvements - which I love to see, don’t get me wrong!).

                                        1. 5

                                          Yeah. Brave’s privacy is only rivaled by Librewolf. Thankfully, the cryptocurrency nonsense is opt-in. And yes, I’m hoping Firefox gets caught up with Chromium security, but it’s not there yet. But the strides they’ve made with RLBox is good to see.

                                          1. 3

                                            It’s opt-in, unless you make some content for which they’d like to collect donations on your behalf. When they opt in for you, it’s no longer opt-in.

                                            1. 2

                                              Indeed, the RLBox improvements are good to see!

                                        1. 1

                                          This looks pretty reasonable for a plaintext note format.

                                          However I feel as though for me, ‘notes’, and ‘comments’ have a sort of overlap in how I’d expect to be using them. I don’t think I’d ever need to write a ‘note’, if I can write a comment instead.

                                          Curious to hear the rationale for both types and how you use them differently, @dewyze

                                            1. 3

                                              Cool stuff!

                                              In the FAQ I’d love to see mention of the cryptographic algorithms used (in this case Argon2id for key derivation and XChaCha20Poly1305 for AEAD - both popular choices these days).
                                              Whenever a project involves encryption that’s the first thing I want to know. Had to dig into source code to figure it out :)

                                              1. 2

                                                Good point, thanks, will add those soon.

                                            1. 1

                                              Note to author: please, add [7 CSS declarations] (http://bettermotherfuckingwebsite.com/) for us mobile readers.

                                              The content is very interesting and I assume very useful, but not simple to use on a phone.

                                              1. 2

                                                It reads fine to me on Bromite (Chromium android fork).

                                                The edges could use some padding but it’s totally tolerable.

                                                1. 2

                                                  Same with me using firefox mobile on Android. OP, what browser are you using?

                                              1. 4

                                                Anyone looking for a secure+private android phone should definitely look to running GrapheneOS on a newer Google Pixel model.

                                                The stuff the GrapheneOS team work on is very important in this day and age if you ask me. Check out the list of GrapheneOS features too.

                                                1. 7

                                                  As long as you’re going to store your TOTP secrets directly on your machine, this feels way “crappier” than just using KeePassXC or similar. Am I missing something?

                                                  1. 6

                                                    It does seem to be missing the point somewhat. TOTP is a protocol for doing crypto with a human in the middle. Putting the human in the middle is there precisely so that the private key can be on a completely different machine to the one that you’re using to log in. The ‘crappy’ authenticator apps (I’ve used the one from F-Droid, which is a fork of the last open-source release of the Google one) manage this for you.

                                                    If you’re willing to store the shared secret on the same machine that you use for logging in, then an attacker who compromises that machine can just exfiltrate it directly. If it’s stored in cyphertext on disk (protected by some key) then you’re not vulnerable to offline attacks if someone steals your computer but you are vulnerable to online attacks. If someone compromises your computer then they can read the secret out of memory and impersonate you.

                                                    With TOTP and a separate device then they need to do online, live attacks: just exfiltrating the one-time password doesn’t help because it’s stops being valid a few seconds later and they need to log into the remote system in that time window.

                                                    You get a small amount of extra security from OTP vs passwords by storing the secret on the same machine. If the attacker is able to compromise your browser then they can leak your password but they can’t leak the secret without a separate sandbox escape that allows them to compromise the credentials manager.

                                                    Both are weaker than using WebAuthn. Now that pretty much all desktop and mobile browsers support WebAuthn with platform authenticators, there’s no excuse for using passwords or TOTP. With a TPM, Windows can protect your credentials with biometrics and require a full OS compromise to gain the ability to fake signatures (but not to . Android and iOS / macOS both provide mechanisms. Android will use a separate hardware store if available, a TrustZone-isolated component if not. iOS and macOS will use the Secure Element, on older Macs there’s an emulator that stores credentials in the Keychain, which is much less good for security, but the best you can do on the older hardware. Not sure what the situation is on Linux / *BSD.

                                                    1. 2

                                                      There is another angle to this which might be of more practical relevance to most people: on some services, enabling 2FA makes it materially harder for support personnel to reset your credentials. I suspect that the vast majority of people are more likely to have their accounts hacked via social engineering than any sort of credential theft, at least if they’re switched-on enough to avoid installing banking trojans. In that light, enabling 2FA is just a way to opt into something like AAA service you don’t have to be an eight-figure customer for, and it doesn’t really matter how secure your TOTP secret is.

                                                      1. 1

                                                        makes it materially harder for support personnel to reset your credentials

                                                        Yep, ask microsoft about that if they think you’re not using that account often enough. Fun support call times.

                                                    2. 3

                                                      I think that really comes down to what exactly you’re trying to secure with TOTP or rather 2FA.

                                                      In the original sense: An attacker has to have one more thing than your password and at best more control than only over your PC (with which you’re logging in). Except I don’t trust my phone more than I can throw it and I do use a password manager with long randomized passwords. And I do use a yubikey as 2FA for my password manager. (with this I have probably now leaked everything important about me)

                                                      So TOTP to me is mostly a pain that protects me against websites that counter brute force attacks with 2FA*. And it makes those websites shut up about setting up TOTP (let’s be honest: “please enter your phone number”).

                                                      And then there is the issue with most people logging into websites also on mobile. So you’re down to one device and the hopes that the per-app isolation of iOS/Android is better than on your PC (which may be true). But that doesn’t help the case that an attacker only requires access to your phone now.

                                                      *Except there are backup-codes which are essentially static passwords. Which also highlights the problem of using only your phone for 2FA, I wouldn’t bet on recovering a google account with 2FA after you lost your phone and backup codes - even if you did send them your passport to verify your youtube-over-18-age / bought a lot of stuff with some form of traceable credit card you can verify yourself with. In fact I had to go though multiple hoops because microsoft found my account usage weird and didn’t trust my TOTP to identify myself.

                                                      1. 2

                                                        I never understood putting 2FA into your primary password manager.

                                                        This makes it a single point of failure. If your password manager is hacked, leaked, or your database is lost/deleted, then you’ve lost everything all at once.

                                                        Maybe I’m wrong on this - but I keep my TOTP stuff in a separate app to my password manager (Aegis, in this case). And backup the encrypted keys/database when I add a new TOTP code (which doesn’t happen very often these days).

                                                        1. 1

                                                          You’re completely correct here. Some prefer to take the risk in exchange for the convenience.

                                                          1. 1

                                                            There is not much convenience in needing to enter a second form, and no added security either, since if your password is already in a password manager, your TOTP code is too.

                                                            1. 1

                                                              Well, there is still security against weird unlikely scenarios like “you’ve entered the credentials into a public PC that was keylogged”. (Some would consider the “compromised personal device” scenarios just as unlikely…) And as mentioned in a comment above, recovery policy improvements that happen in some places due to enabling any 2FA. But yeah, TOTP is really unimpressive in terms of what security it can provide.

                                                        2. 1

                                                          Sometimes you’re happy with a password but someone else has decided that you have to have 2FA.

                                                          1. 1

                                                            I meant to suggest that using oathtool was a “crappier” approach to using TOTP than using KeePassXC or similar. Not that using TOTP altogether was crappy.

                                                            1. 1

                                                              Right, and if you think that TOTP altogether is crappy, then it’s just fine to use oathtool :)

                                                        1. 121

                                                          I used to give the same advice, but I completely changed my opinion over the past 10 years or so. I eventually put in the time and learned shell scripting. These days my recommendation is:

                                                          1. Learn to use the shell. It’s a capable language that can take you very far.
                                                          2. Use ShellCheck to automatically take care of most of the issues outlined in the article.

                                                          I really don’t want to figure out every project’s nodejs/python/ruby/make/procfile abomination of a runner script anymore. Just like wielding regular expressions, knowing shell scripting is a fundamental skill that keeps paying dividends over my entire career.

                                                          1. 60

                                                            Bingo.

                                                            My advice is:

                                                            • Always use #!/usr/bin/env bash at the beginning of your scripts (change if you need something else, don’t rely on a particular path to bash though).
                                                            • Always add set -eou pipefail after that.
                                                            • Always run shellcheck.
                                                            • Always run shfmt.
                                                            • Always pay attention to what version of bash you need to support, and don’t go crazy with “new” features unless you can get teammates to upgrade (this is particularly annoying because Apple ships an older version of bash without things like associative arrays).
                                                            • Always use the local storage qualifier when declaring variables in a function.
                                                            • As much as possible, declare things in functions and then at the end of your script kick them all off.
                                                            • Don’t use bash for heavy-duty hierarchical data munging…at that point consider switching languages.
                                                            • Don’t assume that a bashism is more-broadly acceptable. If you need to support vanilla sh, then do the work.

                                                            While some people like the author will cry and piss and moan about how hard bash is to write, it’s really not that bad if you take those steps (which to be fair I wish were more common knowledge).

                                                            To the point some folks here have already raised, I’d be okay giving up shell scripting. Unfortunately, in order to do so, a replacement would:

                                                            • Have to have relatively reasonable syntax
                                                            • Be easily available across all nix-likes
                                                            • Be guaranteed to run without additional bullshit (installing deps, configuring stuff, phoning home)
                                                            • Be usable with only a single file
                                                            • Be optimized for the use case of bodging together other programs and system commands with conditional logic and first-class support for command-line arguments, file descriptors, signals, exit codes, and other nixisms.
                                                            • Be free
                                                            • Don’t have long compile times

                                                            There are basically no programming languages that meet those criteria other than the existing shell languages.

                                                            Shell scripting is not the best tool for any given job, but across every job it’ll let you make progress.

                                                            (Also, it’s kinda rich having a Python developer tell us to abandon usage of a tool that has been steadily providing the same, albeit imperfect, level of service for decades. The 2 to 3 switch is still a garbage fire in some places, and Python is probably the best single justification for docker that exists.)

                                                            1. 26

                                                              While some people like the author will cry and piss and moan about how hard bash is to write, it’s really not that bad if you take those steps (which to be fair I wish were more common knowledge).

                                                              I think “nine steps” including “always use two third-party tools” and “don’t use any QoL features like associative arrays” does, in fact, make bash hard to write. Maybe Itamar isn’t just “cry and piss and moan”, but actually has experience with bash and still think it has problems?

                                                              1. 2

                                                                To use any language effectively there are some bits of tribal knowledge…babel/jest/webpack in JS, tokio or whatever in Rust, black and virtualenv in Python, credo and dialyzer in Elixir, and so on and so forth.

                                                                Bash has many well-known issues, but maybe clickbait articles by prolific self-pronoters hat don’t offer a path forward also have problems?

                                                                1. 15

                                                                  If your problem with the article is that it’s clickbait by a self-promoter, say that in your post. Don’t use it as a “gotcha!” to me.

                                                                  1. 2

                                                                    I think there’s merit here in exploring the criticism, though room for tone softening. Every language has some form of “required” tooling that’s communicated through community consensus. What makes Bash worse than other languages that also require lots of tools?

                                                                    There’s a number of factors that are at play here and I can see where @friendlysock’s frustration comes from. Languages exist on a spectrum between lots of tooling and little tooling. I think something like SML is on the “little tooling” where just compilation is enough to add high assurance to the codebase. Languages like C are on the low assurance part of this spectrum, where copious use of noisy compiler warnings, analyzers, and sanitizers are used to guide development. Most languages live somewhere on this spectrum. What makes Bash’s particular compromises deleterious or not deleterious?

                                                                    Something to keep in mind is that (in my experience) the Lobsters userbase seems to strongly prefer low-tooling languages like Rust over high-tooling languages like Go, so that may be biasing the discussion and reactions thereof. I think it’s a good path to explore though because I suspect that enumerating the tradeoffs of high-tooling or low-tooling approaches can illuminate problem domains where one fits better than the other.

                                                                    1. 2

                                                                      I felt that I sufficiently commented about the article’s thesis on its own merits, and that bringing up the author’s posting history was inside baseball not terribly relevant. When you brought up motive, it became relevant. Happy to continue in DMs if you want.

                                                                    2. 6

                                                                      You’re really quite hostile. This is all over scripting languages? Or are you passive aggressively bringing up old beef?

                                                                  2. 9

                                                                    Integrating shellcheck and shfmt to my dev process enabled my shell programs to grow probably larger than they should be. One codebase, in particular, is nearing probably like 3,000 SLOC of Bash 5 and I’m only now thinking about how v2.0 should probably be written in something more testable and reuse some existing libraries instead of reimplementing things myself (e.g., this basically has a half-complete shell+curl implementation of the Apache Knox API). The chief maintenance problem is that so few people know shell well so when I write “good” shell like I’ve learned over the years (and shellcheck --enable=all has taught me A TON), I’m actively finding trouble finding coworkers to help out or to take it over. The rewrite will have to happen before I leave, whenever that may be.

                                                                    1. 11

                                                                      I’d be interested in what happens when you run your 3000 lines of Bash 5 under https://www.oilshell.org/ . Oil is the most bash compatible shell – by a mile – and has run thousands of lines of unmodified shell scripts for over 4 years now (e.g. http://www.oilshell.org/blog/2018/01/15.html)

                                                                      I’ve also made tons of changes in response to use cases just like yours, e.g. https://github.com/oilshell/oil/wiki/The-Biggest-Shell-Programs-in-the-World


                                                                      Right now your use case is the most compelling one for Oil, although there will be wider appeal in the future. The big caveat now is that it needs to be faster, so I’m actively working on the C++ translation (oil-native passed 156 new tests yesterday).

                                                                      I would imagine your 3000 lines of bash would be at least 10K lines of Python, and take 6-18 months to rewrite, depending on how much fidelity you need.

                                                                      (FWIW I actually wrote 10K-15K lines of shell as 30K-40K lines of Python early in my career – it took nearly 3 years LOL.)

                                                                      So if you don’t have 1 year to burn on a rewrite, Oil should be a compelling option. It’s designed as a “gradual upgrade” from bash. Just running osh myscript.sh will work, or you can change the shebang line, run tests if you have them, etc.

                                                                      There is an #oil-help channel on Zulip, liked from the home page

                                                                      1. 2

                                                                        Thanks for this nudge. I’ve been following the development of Oil for years but never really had a strong push to try it out. I’ll give it a shot. I’m happy to see that there are oil packages in Alpine testing: we’re deploying the app inside Alpine containers.

                                                                        Turns out that I was very wrong about the size of the app. It’s only about 600 SLOC of shell :-/ feels a lot larger when you’re working on it!

                                                                        One thing in my initial quick pass: we’re reliant on bats for testing. bats seemingly only uses bash. Have you found a way to make bats use Oil instead?

                                                                        1. 1

                                                                          OK great looks like Alpine does have the latest version: https://repology.org/project/oil-shell/versions

                                                                          I wouldn’t expect this to be a pain-free experience, however I would say should definitely be less effort than rewriting your whole program in another language!

                                                                          I have known about bats for a long time, and I think I ran into an obstacle but don’t remember what it was. It’s possible that the obstacle has been removed (e.g. maybe it was extended globs, which we now support)

                                                                          https://github.com/oilshell/oil/issues/297

                                                                          In any case, if you have time, I would appreciate running your test suite with OSH and letting me know what happens (on Github or Zulip).

                                                                          One tricky issue is that shebang lines are often #!/bin/bash, which you can change to be #!/usr/bin/env osh. However one shortcut I added was OSH_HIJACK_SHEBANG=osh

                                                                          https://github.com/oilshell/oil/wiki/How-To-Test-OSH

                                                                        2. 1

                                                                          Moving away from Python? Now it has my interest… in the past I skipped past know it’d probably take perf hits and have some complicaged setup that isn’t a static binary.

                                                                          1. 2

                                                                            Yes that has always been the plan, mentioned in the very first post on the blog. But it took awhile to figure out the best approach, and that approach still takes time.

                                                                            Some FAQs on the status here: http://www.oilshell.org/blog/2021/12/backlog-project.html

                                                                            Python is an issue for speed, but it’s not an issue for setup.

                                                                            You can just run ./configure && make && make install and it will work without Python.

                                                                            Oil does NOT depend on Python; it just reuses some of its code. That has been true for nearly 5 years now – actually since the very first Oil 0.0.0. release. Somehow people still have this idea it’s going to be hard to install, when that’s never been the case. It’s also available on several distros like Nix.

                                                                            1. 1

                                                                              What is the status of Oil on Windows (apologies if it’s in the docs somewhere, couldn’t find any mentioning of this). A shell that’s written in pure C++ and has Windows as a first class citizen could be appealing (e.g. for cross-platform build recipes).

                                                                              1. 1

                                                                                It only works on WSL at the moment … I hope it will be like bash, and somebody will contribute the native Windows port :-) The code is much more modular than bash and all the Unix syscalls are confined to a file or two.

                                                                                I don’t even know how to use the Windows sycalls – they are quite different than Unix! I’m not sure how you even do fork() on Windows. (I think Cygwin has emulation but there is way to do it without Cygwin)

                                                                                https://github.com/oilshell/oil/wiki/Oil-Deployments

                                                                      2. 4

                                                                        To the point some folks here have already raised, I’d be okay giving up shell scripting. Unfortunately, in order to do so, a replacement would: […] There are basically no programming languages that meet those criteria other than the existing shell languages.

                                                                        I believe Tcl fits those requirements. It’s what I usually use for medium-sized scripts. Being based on text, it interfaces well with system commands, but does not have most of bash quirks (argument expansion is a big one), and can handle structured data with ease.

                                                                        1. 4

                                                                          Always use #!/usr/bin/env bash at the beginning of your scripts (change if you need something else, don’t rely on a particular path to bash though).

                                                                          I don’t do this. Because all my scripts are POSIX shell (or at least as POSIX complaint as I can make them). My shebang is always #!/bin/sh - is it reasonable to assume this path?

                                                                          1. 4

                                                                            you will miss out on very useful things like set -o pipefail, and in general you can suffer from plenty of subtle differences between shells and shell versions. sticking to bash is also my preference for this reason.

                                                                            note that the /usr/bin/env is important to run bash from wherever it is installed, e.g. the homebrew version on osx instead of the ancient one in /bin (which doesn’t support arrays iirc and acts weirdly when it comes across shell scripts using them)

                                                                            1. 4

                                                                              My shebang is always #!/bin/sh - is it reasonable to assume this path?

                                                                              Reasonable is very arbitrary at this point. That path is explicitly not mandated by POSIX, so if you want to be portable to any POSIX-compliant system you can’t just assume that it will exist. Instead POSIX says that you can’t rely on any path, and that scripts should instead be modified according to the system standard paths at installation time.

                                                                              I’d argue that these days POSIX sh isn’t any more portable than bash in any statistically significant sense though.

                                                                              1. 2

                                                                                Alpine doesn’t have Bash, just a busybox shell. The annoying thing is if the shebang line fails because there is no bash, the error message is terribly inscrutable. I wasted too much time on it.

                                                                                1. 2

                                                                                  nixos has /bin/sh and /usr/bin/env, but not /usr/bin/bash. In fact, those are the only two files in those folders.

                                                                                2. 3

                                                                                  https://mkws.sh/pp.html hardcodes #!/bin/sh. POSIX definitely doesn’t say anything about shs location but I really doubt you won’t find a sh at /bin/sh on any UNIX system. Can anybody name one?

                                                                                3. 2

                                                                                  I would add, prefer POSIX over bash.

                                                                                4. 18

                                                                                  I checked, and shellcheck (at least the version on my computer) only catches issue #5 of the 5 I list.

                                                                                  1. 14

                                                                                    That’s because the other ones are options and not errors. Yes, typically they are good hygiene but set -e, for example, is not an unalloyed good, and at least some experts argue against using it.

                                                                                    1. 3

                                                                                      Not for lack of trying: https://github.com/koalaman/shellcheck/search?q=set+-e&type=issues

                                                                                      There are tons of pedants holding us back IMO. Yes, “set -e” and other options aren’t perfect, but if you even know what those situations are, you aren’t the target audience of the default settings.

                                                                                    2. 17

                                                                                      I eventually put in the time

                                                                                      Yup, that’s how you do it, It’s a good idea to put in the the time to understand shell scripting. Most of the common misconceptions come out of misunderstanding. The shell is neither fragile (it’s been in use for decades, so it’s very stable) nor ugly (I came from JavaScript to learning shell script, and it seemed ugly indeed at first, now I find it very elegant). Keeping things small and simple is the way to do it. When things get complex, create another script, that’s the UNIX way.

                                                                                      It’s the best tool for automating OS tasks. That’s what it was made for.

                                                                                      +1 to using ShellCheck, I usually run it locally as

                                                                                      shellcheck -s sh
                                                                                      

                                                                                      for POSIX compliance.

                                                                                      I even went as far as generating my static sites with it https://mkws.sh/. You’re using the shell daily for displaying data in the terminal, it’s a great tool for that, why not use the same tool for displaying data publicly.

                                                                                      1. 6

                                                                                        No, it really is ugly. But I’m not sure why that matters

                                                                                        1. 13

                                                                                          I believe arguing if beauty is subjective or not is off topic. 😛

                                                                                      2. 16

                                                                                        I went the opposite direction - I was a shell evangelist during the time that I was learning it, but once I started pushing its limits (e.g. CSV parsing), and seeing how easy it was for other members of my team to write bugs, we immediately switched to Python for writing dev tooling.

                                                                                        There was a small learning curve at first, in terms of teaching idiomatic Python to the rest of the team, but after that we had much fewer bugs (of the type mentioned in the article), much more informative failures, and much more confidence that the scripts were doing things correctly.

                                                                                        I didn’t want to have to deal with package management, so we had a policy of only using the Python stdlib. The only place that caused us minor pain was when we had to interact with AWS services, and the solution we ended up using was just to execute the aws CLI as a subprocess and ask for JSON output. Fine!

                                                                                        1. 15

                                                                                          I tend to take what is, perhaps, a middle road. I write Python or Go for anything that needs to do “real” work, e.g. process data in some well-known format. But then I tie things together with shell scripts. So, for example, if I need to run a program, run another program and collect, and then combine the outputs of the two programs somehow, there’s a Python script that does the combining, and a shell script that runs the three other programs and feeds them their inputs.

                                                                                          I also use shell scripts to automate common dev tasks, but most of these are literally one-ish line, so I don’t think that counts.

                                                                                          1. 2

                                                                                            This makes sense to me

                                                                                          2. 8

                                                                                            we immediately switched to Python for writing dev tooling.

                                                                                            FWIW when shell runs out of steam for me, I call Python scripts from shell. I would say MOST of my shell scripts call a Python script I wrote.

                                                                                            I don’t understand the “switching” mentality – Shell is designed to be extended with other languages. “Unix philosophy” and all that.

                                                                                            I guess I need to do a blog post about this ? (Ah I remember I have a draft and came up with a title – The Worst Amounts of Shell Are 0% or 100%https://oilshell.zulipchat.com/#narrow/stream/266575-blog-ideas/topic/The.20Worst.20Amount.20of.20Shell.20is.200.25.20or.20100.25 (requires login)

                                                                                            (Although I will agree that it’s annoying that shell has impoverished flag parsing … So I actually write all the flag parsers in Python, and use the “task file” pattern in shell.)

                                                                                            1. 2

                                                                                              What is the “task file” pattern?

                                                                                              1. 5

                                                                                                It’s basically a shell script (or set of scripts) you put in your repo to automate common things like building, testing, deployment, metrics, etc.

                                                                                                Each shell function corresponds to a task..

                                                                                                I sketched it in this post, calling it “semi-automation”:

                                                                                                http://www.oilshell.org/blog/2020/02/good-parts-sketch.html

                                                                                                and just added a link to:

                                                                                                https://lobste.rs/s/lob0rw/replacing_make_with_shell_script_for

                                                                                                (many code examples from others in that post, also almost every shell script in https://github.com/oilshell/oil is essentially that pattern)

                                                                                                There are a lot of names for it, but many people seem to have converged on the same idea.

                                                                                                I don’t have a link handy not but Github had a standard like this in the early days. All their repos would have a uniform shell interface so that you could get started hacking on it quickly.

                                                                                          3. 5

                                                                                            You should investigate just for task running. It’s simple like make but none of the pitfalls of it for task running.

                                                                                          1. 1

                                                                                            Not sure why Wiby doesn’t index my site!

                                                                                            1. 1

                                                                                              Unlike others in the list, Wiby and SearchMySite are mostly limited to user-submitted sites. You’ll have to submit your own site and wait.

                                                                                              If that doesn’t work out, you could contact the admins.

                                                                                              1. 1

                                                                                                I did submit it, months ago.

                                                                                                1. 1

                                                                                                  I submitted mine months ago as well but it seems to show up. Try submitting again?

                                                                                                  1. 1

                                                                                                    I wonder if there was a grudge or other personal reason for them nixing it.

                                                                                            1. 2

                                                                                              The static site generator script I wrote for my site is also called shite.

                                                                                              I too couldn’t resist some cheeky naming ;)

                                                                                              1. 9

                                                                                                I’ve read way too many redesign articles like this that don’t actually link to the website that’s been redesigned.
                                                                                                I baffles me why in the first line they couldn’t add a hyperlink to the new MDN like so;

                                                                                                If you’ve accessed the MDN website today, you probably noticed that it looks quite different. We hope it’s a good different. Let us explain!

                                                                                                That being said, the new design is pretty good. But the main change I’d personally make would be to give headings a font-weight slightly more than body text (say like 500). Also I agree with most people saying the new logo is generic but what can you do shrug.

                                                                                                1. 1

                                                                                                  This is really comprehensive. Nice.

                                                                                                  What do you think the chances are of Wayland running in OpenBSD?

                                                                                                  1. 3

                                                                                                    Thanks!

                                                                                                    I don’t want to speak on behalf of OpenBSD developers, but personally I would not hold my breath for Wayland, given that a) they maintain their own port of X called xenocara, and b) desktop is not the main point or focus of OpenBSD.

                                                                                                    1. 1

                                                                                                      Thank you, This is good to know.

                                                                                                    2. 1

                                                                                                      Wayland has been imported into the ports tree [0]. AFAIK it isn’t usable yet, but as you can see by the commits it’s being worked on…

                                                                                                      0: https://github.com/openbsd/ports/tree/master/wayland

                                                                                                    1. 1

                                                                                                      Why is the word “sonos” highlighted in the text every time ? Looks like an ad..

                                                                                                      1. 1

                                                                                                        Every other brand name is also in bold, must be a formatting choice.

                                                                                                      1. 1

                                                                                                        Barring the cryptographic security issues already mentioned in this thread, this is a pretty cool toy and idea.

                                                                                                        1. 2

                                                                                                          Rather than attempting to estimate the entropy of a scheme, wouldn’t it be better to just define a scheme that represents randomness directly in the encoding? If you target N bits of randomness, then you will not fall into these sorts of traps if you directly encode those bits in a symmetric number scheme, such as base16, base32, or diceware (base ~12.92).

                                                                                                          The math is sort of interesting, but I’m not sure why anyone would in practice want to do anything other than directly encoding randomness.

                                                                                                          1. 1

                                                                                                            I became interested in this because I was trying to develop highly memorable passwords. For example, you might want to choose a random grammatical English sentence as your passphrase. A natural way to do this might be to choose a random parse tree of a given size, but if you do that you’ll have some duplicated sentences (with ambiguous parses).

                                                                                                            1. 2

                                                                                                              There are also the EFF word lists, which are designed to be memorable, real, and chosen by dice roll. I particularly like the list where each word can be uniquely identified by its first three letters: https://www.eff.org/deeplinks/2016/07/new-wordlists-random-passphrases

                                                                                                              1. 1

                                                                                                                I’ve created rpass for this. It generates random mnemonics, ie. rpass 128 yields:

                                                                                                                juthor kezrem xurvup kindit puxpem vaszun bok

                                                                                                                1. 1

                                                                                                                  Is it really English if the majority of the words are made up?

                                                                                                                  1. 1

                                                                                                                    You’re right, isn’t English, but “juthor” and “krezrem”, and, “puxpem” are pretty memorable non-words IMO.

                                                                                                            1. 1

                                                                                                              Good rant. I feel similarly in some regards. However, in regard to “electron apps”, the author says:

                                                                                                              They constantly crash and have no value over a native desktop application what so ever - well, perhaps with the only exception that now a 2 year old baby can make something shiny that you can click on with your mouse.

                                                                                                              I hate the popularity, sluggishness and ram gobbling of electron applications as much as the next guy, however they do have one significant value over native applications: they are essentially write once, build+run on almost anything. That is why companies keep making them - they don’t really need a development team for each OS, they just need one electron dev team.

                                                                                                              1. 3

                                                                                                                One suggestion is to rebind the default prefix (C-b) to C-z instead: while C-b moves backwards one character in the default shell config (and hence I use it all the time), C-z backgrounds a job, which I do rarely enough that typing C-z z to do so is perfectly fine.

                                                                                                                1. 5

                                                                                                                  I background+foreground jobs pretty frequently. So I need C-z to be free.

                                                                                                                  Personally I set my prefix to C-a.
                                                                                                                  I think it’s usually used to go to the start of the input line in most shells by default, but I set -o vi in my shell so that doesn’t apply to me.

                                                                                                                  A friend of mine sets their prefix to a backtick. Which I thought was interesting, but I like to use backticks now and then…

                                                                                                                  1. 7

                                                                                                                    C-a is a super common choice, as it’s the same prefix that screen uses by default. The screen folks, in turn, either had the right idea or it was a pretty lucky guess: C-a takes you to the beginning of the line, which is not needed too frequently in a shell.

                                                                                                                    On the other hand it’s the “go to the beginning of the line” in Emacs, too so, uhh, I use C-o for tmux. I suppose it might be a good choice for the unenlightened vim users out there :-).

                                                                                                                    Another prefix binding that I found to be pretty good is C-t. Normally, it swaps two characters before the cursor, a pretty neat thing to have over really slow connections but also frequently unused. Back when I used Ratpoison, I used it as the Ratpoison prefix key.

                                                                                                                    I think C-i (normally that’s a Tab) and C-v (escape the next character) are also pretty good, particularly the former one, since Tab is easily reachable from the home row and available on pretty much any keyboard not from outer space.

                                                                                                                    I’ve no idea why I spent so much time thinking about these things, really. Yep, I’m really fun at parties!

                                                                                                                    1. 2

                                                                                                                      I use C-o for tmux

                                                                                                                      Yeah, I’ve used C-o for screen since I guess the mid-90s because I couldn’t deal with C-a being snarfed, possibly because I was using a lot of terminal programs which used emacs keybindings at the time… Now I’m slowly moving over to tmux and keeping the C-o even though I rarely use anything terminal-based these days.

                                                                                                                      1. 1

                                                                                                                        C-o is pretty important in vim actually. Personally I use M-t, which doesn’t conflict with any vim bindings (vim doesn’t use any alt/meta bindings by default) or any of my i3 bindings (I do use alt for my i3 super key)

                                                                                                                        1. 1

                                                                                                                          Oh, I had no idea, the Church of Emacs does not let us meddle with the vim simpletons, nor dirty our hands with their sub-par editor :-P. I just googled it and, uh, yeah, it seems important, I stand corrected!

                                                                                                                      2. 2

                                                                                                                        Personally I set my prefix to C-a. I think it’s usually used to go to the start of the input line in most shells by default …

                                                                                                                        And in Emacs; I use it multiple times an hour, so unfortunately that is out for me.

                                                                                                                        I think that I have experimented with backtick in screen, after I started using Emacs. I have a vague memory of problems when copy-pasting lines of shell which led me to use C-z instead.

                                                                                                                        1. 2

                                                                                                                          I’ve used C-a in screen for ages and carried it over tmux. Hitting. C-a a to get to the start of the line is so ingrained now that it trips me up when I’m not using tmux.

                                                                                                                      3. 2

                                                                                                                        I just use C-a, primarily because I moved to tmux from using screen, which uses that binding for the prefix.

                                                                                                                        1. 1

                                                                                                                          Yeah I found C-a much better than C-b, much less of a stretch, but eventually I started getting firearm pain in my left arm from all the pinky action. I’ve moved to M-t, most often using right hand for alt and left hand for t.

                                                                                                                        2. 1

                                                                                                                          C-f

                                                                                                                          1. 1

                                                                                                                            Unfortunately that is forward-char🙂

                                                                                                                            1. 1

                                                                                                                              the lesser of all evil

                                                                                                                          2. 1

                                                                                                                            I use C-q. It’s extremely easy for me to hit the keys together, since I remap capslock to be control, and is quite comfortable.

                                                                                                                            This leads to fun when I accidentally killed my other applications, but at this point it’s ingrained so I don’t mess up.

                                                                                                                            1. 1

                                                                                                                              Interesting! I might give that a shot. I do use C-q to quote characters, but not that often, only once or twice every couple of days.

                                                                                                                          1. 4

                                                                                                                            Automounters existed since the 80s but sure, let’s just reinvent the wheel… hang on a minute it’s not round any more - it’s square!

                                                                                                                            1. 10

                                                                                                                              yawn systemd bashing, how boring, and how expected

                                                                                                                              anyway, this is a typical strawman response.

                                                                                                                              automounting is not a novel concept… but noone claimed it is. the post shows how systemd nicely integrates this concept with other systemd concepts so that, for example, it becomes easy to start services that depend on such a network mount, ensuring the right ordering during boot, etc.

                                                                                                                              1. 3

                                                                                                                                yawn systemd bashing, how boring, and how expected

                                                                                                                                My comment wasn’t systemd-specific - it was aimed at any solution in need of a problem - my point was that automounters have existed for well over three decades and they do well what they’re good at.

                                                                                                                                Also, unless I’m missing something, the article describes automating an fstab(5) mount - it isn’t a real automount as in, mount on request/access and unmount if not in use, not to mention other features such us various substitutions (key, wildcard, variable, etc.), etc.

                                                                                                                              2. 4

                                                                                                                                None of the auto mounters I know of integrated with system service ordering and dependencies very well. Also systemd already is the place where fstab handling happens, so it effectively needs to be an (auto)mounter anyway. I don’t think reinventing the wheel criticism really applies here.

                                                                                                                                1. 1

                                                                                                                                  I’m not the biggest fan of systemd… But I do think your comment isn’t very constructive nor a good criticism of this feature of systemd.

                                                                                                                                  1. 1

                                                                                                                                    I have moved from autofs to using the systemd built-in feature in the cases where we are automounting. The advantage is that systemd is already installed, so an additional package is not needed any more, it uses the same syntax for defining mounts as we’re already used to for services and service startup order can be nicely integrated into the auto mount availability.

                                                                                                                                    This as done away with quite a few sleep hacks in old style scripts and so far worked perfectly and did not necessitate new hacks.

                                                                                                                                    Do I need my init system to have built-in automount support? No. But it’s very convenient and very robust, so I’m more than happy to use it.

                                                                                                                                    If you are concerned by bloat or unhappy with the functionality provided by systemd, feel free to continue using autofs or anything else for your mounts.

                                                                                                                                    The one thing I wish was different is if I could have the mount and automount unit in a single file. systemd is very boilerplaty that way and while I see that in some cases the split makes sense, in my simple use cases, it’s just baggage