Threads for nhooyr

  1. 2

    I’ve found the git reflog to pretty much always have the info that I need to get out of trouble. There seems to be quite a few start on Github, so not trying to put this project down. But legitimately, what does this give someone who can use the reflog to reverse unwanted changes?

    1. 2

      What if you didn’t commit those changes and just wrote to the file and then undoed with your editor? This has happened to me before though I’ve just used Neovim’s tree history to go back to my changes.

      1. 1

        I think the target audience (judging by the author’s comment about reflog), is people that can’t or don’t want to use reflog. Seems legitimate enough.

        1. 8

          Originally, my reasoning was that it covers cases where you simply didn’t commit often enough. reflog won’t help at all if there’s no Git object. In hindsight, the prospect of never using reflog again is a sweet one.

      1. 22

        The biggest problem with X11 is the amount of paper my teletype goes through now. It’s really just not economical at this point, and I’m starting to worry about my carbon footprint.

        1. 2

          Huh? Can someone explain? This seems like a joke but I’m not sure what it’s referencing.

          1. 2

            A GUI on a teletype is sub-par. Redrawing windows (or even just moving the mouse) has to print the entire screen again, and that uses a lot of paper.


            Yes, it was a joke. 😉

            I have seen text mode 25x80 (?) “windowing systems” (or some-such) on a teletype, which is the memory that brought this humor to the front of my mind. Fortunately, dumb terminals (aka “glass teletypes”) were inexpensive enough to displace most teletypes pretty quickly. I haven’t even seen one in at least 30 years now.

        1. 13

          I’m certain OpenBSD would’ve “just worked” on this hardware with minimal faffing around.

          1. 3

            I feel the same.

            And I also feel that Dragonfly would have been closer to openbsd’s “just works” than the FreeBSD experience. And it’d likely have performed better.

            1. 1

              Might do. It is on my list to have a look at some time.

            2. 2

              NomadBSD too, which is based on FreeBSD and is installable (despite the name).

              1. 2

                It probably would.

                I reviewed it recently:

                I found, exactly as I expected, that while FreeBSD has a distinctly BDSM flavour, OpenBSD drops the “D” and “M” parts and enforces working exactly its way.

                Yeah, not my bag, thanks.

                1. 6

                  What do you mean FreeBSD has a BDSM flavour? What do you mean OpenBSD drops the D and M? Genuinely curious.

                    1. 1

                      Sounds like BS to me.

                1. 21

                  A useful shorthand I find for this kind of thing is to tag shell commands with comments.

                  $ my awesome | unix pipe | that I want to --remember # frobnicate the bar

                  Now the command is searchable with Ctrl-R > frob...

                  1. 6

                    When I’m in a trolling mood, I show that to people while omitting the space after the # comment and tell them shells support Twitter-style hashtags. :D

                    $ long command to do something #k8s-cheat
                    1. 5

                      Been there, done that ;) My .bash_history

                      1. 7

                        I’m mostly enjoying fish’s completion today. I like the fact that it takes the directory into account. It has its annoyances, but it has never driven me mad to the point that I would start looking for other mechanisms, so far.

                        Regarding your public .bash_history, what do you do to prevent sensitive information from accidentally ending up there?

                        1. 4

                          Crazy you keep yours public. I have mine in a separate private repo from my dotfiles. Too easy to leak a secret.

                          1. 5

                            It seems like it is not a real history file. For example, it is alphabetically sorted. I assume that they use it as a place to store common commands, searchable by ^R.

                            1. 1

                              Yeah, otherwise it would murder my productivity if I had to stop after typing an alias- and shortcut-ladden sausage of a line to add a comment. I spam that enter like crazy, the commenting would never work.2

                      1. 6

                        Go does let you register a callback for when a value’s memory is collected.

                        See runtime.SetFinalizer.

                        It’s used in the standard library to automatically close files if the garbage collector collects the os.File structure.

                        I also use it in my WebSocket library to close the connection if the websocket.Conn structure is collected.

                        1. 8

                          The problem with finaliser-based cleanup is that it relies on GC being run promptly. You can’t use it for things like lock release because it will run an unbounded amount of time in the future. It’s dangerous to use it for file descriptor cleanup because you can rapidly accept a lot of connections and then drop the last reference to the file descriptor object but if the GC doesn’t catch up fast enough then you don’t close the file descriptors and the will hit OS-defined limits (in the worst case, consume a lot of kernel resources in the best case). The Java documentation explicitly tells you not to use finalisers for file descriptor cleanup except as a last resort (i.e. if you’ve accidentally forgotten to close an open descriptor) for precisely this reason.

                          1. 2

                            Fully agreed, it’s not something I’d recommend using as your primary cleanup mechanism either. Just wanted to put it out there as it’s not a well known feature.

                        1. 3

                          I find git’s bundled diff-highlight paired with the occasional –color-words to be sufficient for this use case with the strong benefit of working across all files.


                            # Requires /usr/local/share/git-core/contrib/diff-highlight in $PATH
                            # or the full path to diff-highlight.
                            # See
                            pager = diff-highlight | less


                          git diff --color-words
                          # or
                          git log --color-words
                          1. 5

                            That’s definitely a great option, although difftastic will do a similar word-based highlight for files it doesn’t understand too.

                            --color-words can get confused. For example, it assumes all whitespace is insignificant, which isn’t true in e.g. Python.

                          1. 10

                            I use .lan only because it was the default in openwrt, but I like that this exists.

                            1. 2

                              You should avoid that and other made-up TLDs unless you’ve configured your DNS server/root with a .lan top-level domain zone file.

                              1. 4

                                Confused by this comment. Wouldn’t they have to do that to even use any made up TLD like .lan?

                                1. 2

                                  I believe openwrt just uses that as the default name, but doesn’t do any DNS setup to handle it.

                                  1. 4

                                    That’s not true. dnsmasq in openwrt is configured to route foo.lan to whichever device advertised their name as foo with DHCP (or the ipv6 equivalent).

                                    1. 1

                                      Got it. I meant that if you did start using it, it wouldn’t work anyway so you’d have to set it up correctly. So you can’t use (or avoid) made up TLDs without DNS configuration.

                                      I guess I’m just being particular about the language but the OP to me communicated there’s a way to use a made up TLD without configuring DNS and that you should avoid doing so which doesn’t make sense.

                                      1. 1

                                        Is this a bug in openwrt that could be fixed with a submitted patch?

                                1. 50

                                  To be honest, I’m really sad and concerned about people opting to use WSL. It really looks like another EEE scheme. If you want to have freedom in the long run, please, please participate in the FOSS desktop ecosystem: help debug driver issues, help maintain HCLs to make it easy for new users to pick knowingly well-supported hardware, at the very least, report the bugs you see.

                                  WSL opens up a way for MS to push the restricted boot for desktops without people noticing who would otherwise be the first to notice. Monopolistic proprietary software vendors are not friends of FOSS. Never were, and never will be. Don’t trade essential liberty for temporary convenience.

                                  1. 10

                                    Unfortunately, “the linux desktop” is also wholly captured by monopolistic proprietary software vendors, who are also not friends of FOSS. The systemd debacle, the dbus debacle, all the renderer debacles, the app store debacles, the various gnome debacles – it’s monopolistic proprietary tasteless foisted broken politically-contrived nonsense all the way down. It’s hardly surprising that people use WSL which at least has the benefit of you being able to run productivity software that consistently works.

                                    1. 4

                                      With big corporations one can never know, but it seems to me that in recent years Microsoft have been a good citizen of the open-source software world. I’m willing to give them the benefit of the doubt.

                                      That being said, I’m also concerned about native Linux desktop losing traction, but I have think this has been happening steadily for a while now. I used to know a ton of people running Linux as their primary OS, and now almost everyone’s on macOS and WSL. Perhaps most people don’t care much about the underlying principles, as long as they get their work done.

                                      1. 22

                                        Well, and from your very post it’s clear where it eventually leads. macOS isn’t the “polished UNIX experience” people thought it always would be. I believe we should make the system we want or else we risk a situation when there will be nowhere to migrate to.

                                        That said, I do get my work done on a Linux desktop and nothing crashes for me on 8th gen Intel NUC hardware.

                                        1. 20

                                          I’m one of them. I’m actually the last person to switch out of… about 20 regulars? in what was once a local LUG. I still run Linux and OpenBSD pretty much everywhere I can except for my main working machine, which runs macOS. I’m not happy about it, but it’s also not a matter of convenience.

                                          It’s not that I’ve sacrificed the principles of open source but, realistically, I do not trust the vision that’s currently prevalent in the world of FOSS desktop. It’s a vision that I don’t understand – seeking to provide software for users who understand and care about the technical aspects of fundamental liberties, who can file bug reports and test fixes from a devel branch, or even submit patches, who are willing to follow HCLs before buying new hardware, but are also confused by too many customisation options, and intimidated by small icons and buttons. It produces software that I find less capable with each release, and which makes it harder and harder for me to make things that others find useful, to work on new things, to learn things about computers and things other than computers, and so on.

                                          Showing up with patches – or heaven forbid, bug reports – that do not demonstrate a sufficient understanding of these principles is met with hostility by many communities, and I honestly have no interest in even trying anymore. I’ve no intention of dying on hills held by designers who asked four interns to perform these actions and rate how hard it was, thus unlocking the secret of how to build the UIs that will finally bring upon us the Year of Linux on the Desktop.

                                          And I honestly have no interest in going the underground route, either – running dwm, i3, or resurrecting my FVWM config from 15 years ago and going back to using mc, pine, bc and emacs under xterm. I like file managers with nice-looking icons and proportional fonts, I like graphical email clients and, generally, all these programs that try (and sometimes fail) to figure out what would be good tomorrow, rather than to get to the essence of what was good thirty years ago.

                                          But I also think that it’s not a good idea to develop professional software – terminal emulators, text editors and IDEs, CAD/CAE tools, system administration tools, whatever – by using UX and UI design principles for novice users and freemium apps. All you get is software designed for people who don’t want to run it in the first place, and deliberately made worse for those who do want to run it.

                                          I wish there was a better way, but if the best way to write things that run under Linux and help others do cool things with it is to get a Macbook or a Surface and SSH into a headless machine, or run WSL, I can do that. (And honestly, I totally get the appeal, too. After a few months with iTerm 2 – which is also open source, and I mean GPL, the commie kind of open source – I don’t want to touch Gnome Terminal ever again in my entire life).

                                          1. 5

                                            That we agree on principle 1. “professionals first” comes as no surprise ;-)

                                            In that vein though, can you think of concrete/specific examples that illustrate the conflict of interests/disconnect? The easiest I have from the OSX brand of face-meets-palm would be the dialog:“Terminal would like to access your Contacts”. Unpacking it speaks lengths about what is going on.

                                            1. 2

                                              My favourite one lately is from Windows, which insists on automatically rebooting in order to install updates in the middle of the night so that you’re greeted by a full-screen ad for Edge, which is now also your default browser a better and more secure Windows experience in the morning. That works out great on paper, but in practice, lots of us who use computers for work just wake up to the Bitlocker screen, and five minutes of please wait, installing updates.

                                              From Linux land… I’ve honestly ragequit a long time ago, I kept poking it for three or four years (by which I mean I mostly ran a bunch of old console apps and a few Qt apps) and eventually gave in and bought a Mac. I mostly have a bunch of bad memories from 2012-2017 or so, after which my interactions with it were largely limited to taking point releases of XFCE, Gnome and KDE for test drives and noping the fsck out back to FVWM. So most of my complaints are probably either out of date, or oddly generic (everything’s huge and moves a lot and that’s not nice to my ageing eyes).

                                              Plus… there’s this whole approach to “improving” things, you know?

                                              When I got this stupid Mac, it was uncanny to see how many of the things I used back in 2006 or 2007, when I last used a mac, are still there and work the same way. Even Finder has the same bugs :). Meanwhile, there are things I liked in KDE 3.5 that literally got rewritten and broken three times since then, like custom folder icons support. Which is now practically useless anyway, since icons from modern themes look pretty much the same below 64x64px – I get looks great in screenshots but guys I have folders with a hundreds of datasheets, there’s no way I can ever find anything in there if I can only see like 12 files at a time.

                                              (Edit: FWIW, I think we pretty much agree on all twelve :P)

                                              1. 2

                                                My favourite one lately is from Windows, which insists on automatically rebooting in order to install updates in the middle of the night so that you’re greeted by a full-screen ad for Edge, which is now also your default browser a better and more secure Windows experience in the morning. That works out great on paper, but in practice, lots of us who use computers for work just wake up to the Bitlocker screen, and five minutes of please wait, installing updates.

                                                Somewhat ironic how denial of service is rephrased as a double-plus good security measure, no? Coming from a SCADA angle, the very idea of an update of any sort borders on the exotic (and erotic), but the value of that contract seems to have been twisted into a means of evaluating change by forcing it on users and study the aftermath. The browsers are perhaps the most obviously, but regardless of source it is quite darn disgusting.

                                                From Linux land… I’ve honestly ragequit a long time ago, I kept poking it for three or four years (by which I mean I mostly ran a bunch of old console apps and a few Qt apps) and eventually gave in and bought a Mac.

                                                So amusingly enough I was a die hard FOSS desktop user from the mid 90ies until the arrival of OSX (though raised in Solaris lands). Most of my bills went to pay the cluster of PPC mini macs I used to do my dirty deeds (the biggest of endians). Come 10.6 it was clear that Apple’s trajectory was “fsck you devs, we’re through” and I took it personally. Left all of it to rot, returned to FOSS and was dismayed by what the powers that be had done to the place. The tools I was working on towards, “solving oscilloscope envy by reshaping the debugger” had to be reused to build a desktop that “didn’t change beneath my feet as I was walking around”.

                                                I get looks great in screenshots but guys I have folders with a hundreds of datasheets, there’s no way I can ever find anything in there if I can only see like 12 files at a time.

                                                I would like to run an experiment on you, but the infrastructure is lacking for the time being – here is one of those things where VR performs interesting tricks. Posit that a few hundreds of datasheets are projected onto a flat surface that is textured onto a sphere. You are inside of that sphere with a head mounted display. How long would it take your cognition to find “that one sheet” among the others, versus scrolling through a listview…

                                                1. 1

                                                  the value of that contract seems to have been twisted into a means of evaluating change by forcing it on users and study the aftermath. The browsers are perhaps the most obviously, but regardless of source it is quite darn disgusting.

                                                  IMHO this trench war of updates, where users are finding new ways to postpone them and companies (in this case, Microsoft) are finding new ways to make sure updates happen, is entirely self-inflicted at the companies’ end.

                                                  Way back when big updates were in the form of service packs, there was generally no question about whether you should update or not. If you were running bleeding-edge hardware you’d maybe postpone it for a week or two, to let the early adopters hit the bad bugs, but that was it. As for the smaller, automatic updates, people loathed them mainly because of the long shutdown times, but it was generally accepted that they at least caused no harm.

                                                  Nowadays, who knows. During lockdown I had to go halfway across the city to my parents’ house twice to get my overly-anxious mother (elementary school teacher who’s one or two years away from retirement, so pretty scared when it comes to tech) past the full-screen ads with no obvious close buttons, restore Firefox as a default browser and so on, while she commandeered my other parental unit’s 15 year-old, crawling laptop to hold the damn classes. No wonder everyone dodges updates for as long as they can.

                                                  I would like to run an experiment on you, but the infrastructure is lacking for the time being – here is one of those things where VR performs interesting tricks. Posit that a few hundreds of datasheets are projected onto a flat surface that is textured onto a sphere. You are inside of that sphere with a head mounted display. How long would it take your cognition to find “that one sheet” among the others, versus scrolling through a listview…

                                                  I’m sure you’ve thought about this for longer than I have, but I suspect there are two things that determine success in this case:

                                                  1. An organisation system that matches the presentation (e.g. alphabetical order for a one-column list view)

                                                  2. Being able to focus on a sufficiently large sample that you can browse the list without moving your eyes back and forth too much

                                                  3. is pretty obvious, I guess. I like to go through listviews because these things are sorted alphabetically, and while many of them have very stupid names like slau056.pdf (which is actually MSP430x4xx Family User’s Guide), I usually sort of know which one I’m looking for, because manufacturers tend to follow different, but pretty stable conventions. As long as they’re laid out in a way that makes it easy to “navigate the list” (in stricter terms, in a way that preserves ordering to some degree, and groups initial navigation options – i.e. sub-directories – separately so they’re easy to reach, wtf GTK…), it’s probably fine.

                                                  4. probably bears some explanation because it’s the reason why large icons suck so much. Imagine you have a directory with 800 items and you’re looking for one that’s in the middle. If you can only see 10-12 at a time, then a) it takes a lot of time to hit the exact window with the file you’re looking for, and the tiniest amount of scroll moves up and down by a whole page of items. So you get to go back and forth between dozens of 10-item pages, and often overshoot, then undershoot the one you’re looking for dozens of times, all while wiggling your eyes all over the window.

                                                  I dunno what to think about the inside of a sphere. My knee-jerk reaction is to say I’d get dizzy and that a curved, but field-of-view-sized surface might be a better fit. But gentleman skeptics once complained that trains would be draughty, too, and it turned out they were, but also that it was more than worth it. Just like flying machines became a thing once we finally figured out imitating birds is just not the right way to go about it, we’re probably going to make real progress in organising and browsing information only at the point where we stop imitating libraries, so I think this meets the essential prerequisites for success ;-).

                                                2. 1

                                                  (everything’s huge and moves a lot and that’s not nice to my ageing eyes).

                                                  For what it’s worth, the Reduce Motion accessibility setting helps with some of that.

                                              2. 7

                                                I think the worst thing that happened to FOSS is UX designers (and that includes things like Flatpak).

                                                It doesn’t matter whether the reason for “software won’t do X anymore” is some evil mega-corp or some arrogant UX designer, the result is the same.

                                                (And before anyone slides in with a “let me mansplain UX to you”: I’m good, thanks.)

                                                1. 8

                                                  The field of UX has massively regressed in the last 15-20 years, everywhere, not just in the FOSS world. It’s a cargo cult at this point. Even many (most?) of the organisations that allegedly practice “metrics-driven” design routinely get it so wrong it’s not even hilarious – they make fancy graphics from heaps of data, but they have no control groups, no population sampling, and interpretations are not checked against subject feedback (which is often impossible to get anyway, since the data comes from telemetry), so it all boils down to squinting at the data until it fits the dogma.

                                              3. 11

                                                It’s a common thing for people to discuss what {company X} thinks about {idea Y}. But if you work for a corporation at least a while, you learn there’s no such sentiment. It’s closer to {high level exec A} thinks that {area B} is a great way to expand and {investing in C} is the way to do it. Soon person on position “A” may change, “B” may have a good replacement, and money pumped into “C” may turn out to not have a good return. Microsoft doesn’t like or dislike anything. Managers with enough power temporarily like some strategies more than others.

                                                We had Balmer on one extreme, now we’ve got Satya who seems like the other extreme. In 5 years we may have either: Balmer++ deciding to sue Valve for Proton, or Satya++ deciding to opensource parts of windows kernel, or someone in the middle who approves something nice for FOSS and destroys something nice for FOSS, because they don’t care about that aspect at all, or anyone in between.

                                                Corporations don’t deserve the benefit of the doubt. Some execs do. Just remember they’ll be out in a few years.

                                                1. 2

                                                  This is very true.

                                                  I think this phenomenon is laid out very well in the book The Dictator’s Handbook. It claims that countries, companies, and other entities don’t have opinions or preferences; people are the top and at every level do. They are looking after themselves.

                                                  Disclaimer: not the author of the book; it’s just one of my favorites, and I recommend it for everyone to read.

                                                2. 4

                                                  With big corporations one can never know, but it seems to me that in recent years Microsoft have been a good citizen of the open-source software world. I’m willing to give them the benefit of the doubt.

                                                  What have they done, or stopped doing, to earn this praise?

                                                  1. 7

                                                    They’ve released a lot of core tech as open source: VS Code, .NET Core, some Windows programs such as Terminal and Calc, etc. They’ve supported a lot of other open source things less directly: NuGet, Python packages, etc. They’ve more or less stopped campaigning, advertising and litigating against open source stuff the way they did up through the mid/late 2000’s – see here for entry points to some good examples.

                                                    I’ll happily give Microsoft the benefit of the doubt, but we’ll see if they continue this strategy of being nice for another 5-10 years, or whether we enter the “extinguish” phase, more or less the same way Google has in the last 5 years. If Microsoft thinks they can make more money being nice than being evil, then that’s what they’ll do; that’s the only decision path that matters to them.

                                                    1. 12

                                                      They won’t abuse their monopoly too much, because they don’t actually have one. It’s no secret that they open-sourced .Net to make it a viable option for web on Linux servers, and even though they dominate desktop, desktop itself has competition from 1) the browser (see: web apps) and 2) phones/tablets (there’s zero difference between a laptop and a tablet with a keyboard). They’re playing nice because they know they’re an underdog and can’t afford to act otherwise.

                                                      Of the three platforms (browser/desktop/touchscreens), the Browser is most controlled by Google, and phones/tablets are most controlled by Google. This conveniently lines up with the “Google is the new Microsoft” meme.

                                                  2. 2

                                                    I also think Microsoft has done way better, and I really think they are likely to continue supporting and embracing open source. It seems like the company culture has shifted in a very fundamental way.

                                                    That being said, we should still operate under the assumption that they won’t. Trust, but verify. Same reason you should be wary of signing a CLA that assigns copyright, even if you really, really trust the company the CLA comes from.

                                                    1. 1

                                                      I used to know a ton of people running Linux as their primary OS, and now almost everyone’s on macOS and WSL. Perhaps most people don’t care much about the underlying principles, as long as they get their work done.

                                                      My main principle was always to get certain things done. For a long time (say, from the year 2000 to 2018) Linux was absolutely number one for that, for me. Now it’s Apple, but by a small margin.

                                                      I’m fairly certain that it will never be Windows except for games. And that’s becoming a small margin as well.

                                                      From the article:

                                                      As you can see there’s nothing fancy about it. I wanted to build a decent workstation, not a gaming rig. Still, I opted to get a decent discrete GPU, as there were a few PC games that I was hoping to eventually play (StarCraft II, Tomb Raider, Diablo III, etc).

                                                      So this is absolutely and obviously a gaming rig :)

                                                    2. 2

                                                      I think that asking people to pay a cost (in time, complexity and frustration) to try and match their desktop workflows from Mac OS or Windows in Linux is a fool’s errand, and in addition, the “real enemy” isn’t Microsoft (Windows) or Apple, but rather Google/Amazon/Microsoft (Azure/Github). I think the real challenges to software freedom are in confronting the big SaaS providers, and I would love to hear what the various freedom advocates think we should do about that.

                                                      1. 1

                                                        big SaaS providers

                                                        What do the big SaaS providers have to do with the state of linux on the desktop?

                                                        1. 2

                                                          Not much! But they have a lot to do with the state of software freedom, or rather, the lack thereof.

                                                    1. 14

                                                      The reason I started using testify is because the error messages were better than the built in messages, and I didn’t like repeating the if statement condition in every error message. I’m not sure if this is still the case though.

                                                      One thing I don’t like about the testify lib is the lack of consistency on parameter order (actual and expected.)

                                                      1. 3

                                                        One thing I don’t like about the testify lib is the lack of consistency on parameter order (actual and expected.)

                                                        assert.Len(t, collection, length, message)

                                                        bothers me a lot

                                                        1. 1

                                                          Isn’t that one correct? Collection is the “actual” and length is the “expected”.

                                                          1. 3

                                                            I don’t know if “correct” is really the appropriate word to use here, but no, it is inconsistent with most other methods. For example:

                                                            1. 2

                                                              Oh haha my bad. I misread the parent comment as claiming “actual, expected” is the prominent order but it’s indeed the reverse.

                                                        2. 2

                                                          I like using libs like testify for the same reason, when a test fails, the output is helpful. Multiline strings are diffed. JSON inconsistencies are highlighted. Nested values with subtle differences are compared. It’s those features that make a huge difference.

                                                          I think testifys has evolved over time in ways it shouldn’t, like inconsistent arguments and functions that bloat the API, but it’s still great imo.

                                                          Out of my own desire to explore ideas I’ve been building my own assertion library, inspired by testify, minimalistic, but useful for the apps I build, I don’t expect to build something better, but to understand the tradeoffs, decisioning process, and how this stuff works.

                                                        1. 15

                                                          Whenever someone wants to send me a file […] they could just send me a magnet link

                                                          Ha. Well. Other than everything in the BitTorrent world being designed for mass sharing and feeling like overkill for one-to-one “beaming”, there are two elephants in the room, none of them having anything to do with the “piracy perception”. First, the possibility of both sides being behind awful cgNAT/NAT444, making p2p connections impossible. Second, the privacy thing: do you even want your recepients to know your home IP address? Probably not always.

                                                          1. 3

                                                            As a particular example of the Inverse DRM Problem, I don’t think that it’s possible to receive a file over an addressed switching network without telling somebody some portion of your address. (Recall that the DRM Problem is that you cannot create a secure computing enclave within somebody else’s machine and keep the inputs and outputs private from them. The Inverse DRM Problem is that you cannot exist as a single small node in a homogenous network without projecting most of your information across your neighbors.) For example, I usually recommend Magic Wormhole for transferring files, but it exposes addressing information to a trusted third-party intermediate server.

                                                            1. 4

                                                              Yeah, I’m not talking about intermediaries though, only specifically the recepients. If you want “Snowden level” security, use Tor, but for casually sending something to someone I only know online, all I need is some intermediary to just “abstract” the content away one step from me.

                                                            2. 2

                                                              Does UDP hole punching not work behind cgNAT?

                                                              edit: I know port forwarding is impossible but I can’t find anything on hole punching not working.

                                                              1. 3

                                                                I recall a network company doing some overlay network so people can communicate easily (like sending files or stuff to each other), no matter their machine. Apparently, a good percentage of their users have to pass through their centralised servers (used as a fallback), because even hole punching didn’t work.

                                                                Besides, whole punching requires a handshake server to begin with. What we would really like is a direct peer-to-peer connection, and that’s just flat out impossible if both sides are under NAT.

                                                                1. 5
                                                                  1. 3

                                                                    Yes, that’s the one:

                                                                    This is a good time to have the awkward part of our chat: what happens when we empty our entire bag of tricks, and we still can’t get through? A lot of NAT traversal code out there gives up and declares connectivity impossible. That’s obviously not acceptable for us; Tailscale is nothing without the connectivity.

                                                                    We could use a relay that both sides can talk to unimpeded, and have it shuffle packets back and forth. But wait, isn’t that terrible?

                                                                    Sort of. It’s certainly not as good as a direct connection, but if the relay is “near enough” to the network path your direct connection would have taken, and has enough bandwidth, the impact on your connection quality isn’t huge. There will be a bit more latency, maybe less bandwidth. That’s still much better than no connection at all, which is where we were heading.

                                                            1. 2

                                                              Wow, some of these are really embarrassing on Apple’s part. Great writeup!

                                                              1. 7

                                                                As of October 6th, 2020, the vast majority of these findings have been fixed and credited. They were typically remediated within 1-2 business days (with some being fixed in as little as 4-6 hours).

                                                                That aspect though was pretty great to read, especially considering how slow apple has been historically for this kind of thing. nice!

                                                              1. 13

                                                                What resources have you found in your research so far? Why have you limited to these two choices? What your goals for your project? What differences are you weighing that you’d like people’s thoughts on? This doesn’t sound like a serious question.

                                                                1. 3

                                                                  I have found two interesting web frameworks that is high performance:

                                                                  • Actix (Rust)
                                                                  • Echo (Go)

                                                                  Actually one of the reasons that I limited these to two choices is I love working with these two languages and both are interesting languages, but also after research AFAIK there is no high-performance Python (I love working with Python too) framework that might be suitable for this.

                                                                  My goals for this project is build a high-performance trading platform that is scalable and high-performance, it is actually not a serious project right now but I want to explore this area, that’s one of the reasons why I am asking about the right framework & websocket server.

                                                                  Sorry for not providing more information on the question, I hope it is helpful. Please let me know if there is still something that I can add.

                                                                  1. 11

                                                                    The framework pattern isn’t as much of a “thing” in Go as it is in many (most?) other languages. (The language deliberately doesn’t provide the expressive power and affordances that frameworks tend to need.) Consequently, frameworks in Go are less necessary, and less useful, versus many other languages. They each might have their slightly different architectures and might be better or worse on various metrics, but in the end they all boil down to sugar over net/http.

                                                                    I write all this to say: consider the language independent of any particular web package or framework or whatever. Also, quantify “high performance” before you start making decisions based on it :) And stay away from fasthttp.

                                                                    1. 2

                                                                      And stay away from fasthttp

                                                                      Other than lack of HTTP/2 support, what reason is there for avoiding fasthttp?

                                                                      1. 6

                                                                        Another big reason is it doesn’t do HTTP/1, either: many corner cases and details are simply not implemented. Also, it doesn’t reuse any of the net/http interfaces, so using it sequesters you in a totally separate ecosystem. But really, the code is just a mess. The structure is emergent, clearly not designed — the documentation is written without care or consistency — it feels like somebody’s hackathon project.

                                                                    2. 4

                                                                      Are you talking about high-frequency trading here, or is this just a general “I’m making a web app” question? There are plenty of easy-search-able results that include benchmarks for popular web frameworks.

                                                                      I’m also interested in how you came to the conclusion that “there is no high-performance Python.”

                                                                      1. 2

                                                                        I believe it actually involves high frequency trading, that is why I felt required to ask. I know Actix is one of the fastest web frameworks, but is Rust true language for high-frequency trading and does it makes sense to use Rust. Also same for Go, Go have very fast web frameworks. That is why I compare these two languages.

                                                                        Python being higher-level language with lots of abstractions was one of the reasons made me think that it is not high-performance, also I have never seen real life usage of Python in HFT area but if Python is actually suitable, I’d like to learn more.

                                                                        1. 7

                                                                          The application serving web requests is (hopefully?) not the same one executing HFT strategies. Any reasonable modern language will be fine for the former.

                                                                          1. 5

                                                                            Most of the already existing high-frequency trading apps are written in C++, however popularity is not a particularly good benchmark when it comes to building something new. I’d say give both a try and benchmark your algorithms’ implementations on both platforms. The results might be surprising; it’s very easy to write something that’s fast on Go and slow on Rust, and vice-versa, due to the different programming styles on both languages.

                                                                            1. 5

                                                                              Do you think you can realistically compete with the FPGA and CUDA-based implementations at the big firms?

                                                                              1. 1

                                                                                What do they use CUDA for?

                                                                                1. 1

                                                                                  This paper has some background:

                                                                          2. 3

                                                                            According to these benchmarks Echo (Go) is not high performance compared to Actix (Rust) - the difference is embarrassing:

                                                                        1. 6

                                                                          If anything, the Linux kernel should consider sourcehut. It’s probably the only platform that works well with open standards and decentralised systems.

                                                                          1. 11

                                                                            I think Sourcehut is an incredible product and speaks very well of Drew Devault’s technical and business acumen, but I’m not convinced that where Linux hosts its source code is its biggest problem to be worrying about.

                                                                            1. 13

                                                                              I don’t think it’s the most pressing matter right now, but it is important to keep in mind that it was once and it was solved. Due to the whole BitKeeper mess, Torvald’s started working on git as a means to perform distributed version control - and mail was the cornerstone. Now, GitHub is lobbying to drift apart from email and use their non-standard and close issue management system. It does look familiar.

                                                                            2. 17

                                                                              Alpine adopted sourcehut and it didn’t end well. Sourcehut’s extreme opinions on mail left package maintainers unable to send mail because Devault didn’t like the settings you sometimes couldn’t even change. After some developers left the project, they instead switched to GitLab.

                                                                              1. 3

                                                                                Source? What opinions?

                                                                                1. 14
                                                                                2. 2

                                                                                  Alpine seems to be a bit all over the place at the moment… Their repos look to be self-hosted cgit, their mailing lists are Sourcehut, their bug tracker is GitLab and their wiki is self-hosted Mediawiki.

                                                                                  Edit: ah, their cgit-hosted repos are a mirror of what’s in GitLab, so I guess that’s an interim measure.

                                                                                  1. 3

                                                                                    This is an extremely uninformed and shitty take on what happened with Alpine Linux which doesn’t map at all onto reality. Keep your slander to yourself.

                                                                                    1. 8

                                                                                      As someone with no stake in this, I went and had a look at the available evidence, and… maybe sourcehut needs to correct the record somewhere?

                                                                                      After looking over the alpine mailing lists, I got the distinct impression that some widely-used mail clients can’t send mail to sourcehut and some alpine contributors have refused to use it on that basis.

                                                                                      1. 20

                                                                                        It never caused any developers to “leave the project”. The arguments were not being made in good faith, either. There was a subversive element with a personal vendetta against me who were pulling strings under the table to crank up the drama to 11. As a matter of fact, it was settled after an IRC discussion and the Alpine mailing lists (still running sourcehut) work fine with those mail clients now. All I pushed for was due consideration of the rationale behind the design decision, and during the discussion, a compromise was reached and implemented.

                                                                                        And they didn’t “switch” to GitLab from sourcehut, either. They were never using more than the mailing list features of sourcehut, and there was never a plan to start using them. The sourcehut mailing lists are still in use today, and the GitLab migration was planned in advance of that and completed for unrelated reasons.

                                                                                        There are persistent, awful comments from a handful of people who spread lies and rumors about what happened as if I attempted a hostile takeover of Alpine Linux. It’s complete horseshit, and incredibly offensive and hurtful, especially in light of how much time and effort I’ve devoted to improving Alpine Linux. I am a maintainer of many packages myself, and a contributor to many of their repositories!

                                                                                        1. 8

                                                                                          Thanks, I appreciate the reply - I can only imagine this has all been quite draining, especially while trying to get a new product off the ground.

                                                                                1. 10

                                                                                  This has been discussed before, but I was wondering if anything has changed.

                                                                                  Several difficulties and problems were raised … have they been addressed?

                                                                                  Does anyone use it?

                                                                                  1. 11

                                                                                    I’ve been there for close to 2 years, and have tried to build my own SSB server from scratch (in a non-JS language). Feel free to ask any questions. For starters:

                                                                                    • The low level protocol (transport encryption, RPC and discovery) is very well documented.

                                                                                    • The application level protocol has almost no documentation, and what’s there is outdated. You really have to resort to reverse engineer behaviour from existing applications, or reading other’s code.

                                                                                    • Replication/gossip mechanism is very inefficient, which leads to clients (especially mobile ones) spending a lot of time during the initial sync. There’s a newer gossip protocol which fixes some of these problems, but has zero documentation, and there’s only one implementation (in JS). There are no plans to port it to other languages since there’s a lot of tricky edge cases in there.

                                                                                    • Yes, the JSON encoding was a mistake. There’s a new format using CBOR, but it’s still a few ways off in terms of mainstream usage in the network.

                                                                                    • There are questionable decisions at the application level. For example, anyone can assign you new profile pictures or visible names, which can–and has–lead to bullying/name-calling.

                                                                                    In terms of community, it’s mostly tech-centric, most discussions are either about SSB itself, or related protocols. The overall spirit is positive, focusing on sustainable living, gardening, off-grid, etc.

                                                                                    However, the community is very small. This can get tiring, considering that most clients will show any replies to threads you follow at the top of your timeline (you will see the same 3 to 5 people all the time).

                                                                                    1. 4

                                                                                      I’ve also built a partial SSB implementation, in C++. I found a C port of the CLI tool (in an obscure Git repo hosted in SSB), which helped immeasurably with the secure handshake and packet codec. I used Couchbase Lite [of which I’m the architect] as the database. This proved a lot faster than the JS data store, but I still found that pulling all the subscribed content from one pub (on which I’m following a handful of accounts) resulted in a 600MB database. It would have helped if the protocol had an option to pull only messages back to a certain date.

                                                                                      I’d love to know more about the new protocol, but not if the only way is to decipher a JS codebase.

                                                                                      It’s a shame they’re so JS-centric. That’s a serious problem for iOS, which has security restrictions that disallow JITs outside of a web browser. Not to mention embedded systems. (And on a personal level I dislike doing serious programming in JS; it feels like sculpting in Jell-O.)

                                                                                      1. 3

                                                                                        There are two C implementations of the low level protocol: one for the secret handshake and one for the boxstream (transport encryption). There’s also some integration tests that you can run against your implementation to validate that everything works.

                                                                                        As for the new replication protocol: the new CBOR format includes “off-chain” contents, which means that the actual log only contains a hash of the post content. This should make initial sync much faster, since clients only fetch the chain of hashes, without downloading anything else.

                                                                                        Messages can also be downloaded out of order, so you only download what you want, if you have the hash for it. As most things, though, the only implementation is in JS.

                                                                                        As for the database, I planned to use SQLite, but never got far enough to test that. I’m unconvinced that the log is a good abstraction for the kind of apps that SSB is used right now (social media). There are future plans to plug more applications on top of the log replication, but that’s in the long term, while the current use-cases are suffering from it.

                                                                                        Edit: wanted to say, though, that for me the biggest block when developing an SSB implementation is the lack of documentation w.r.t. the application-level protocol, and forcing you to develop everything on top of the log abstraction. The JSON formatting can be painful, but solved via forking some json library and doing some changes (hacky, but it works).

                                                                                      2. 1

                                                                                        Yes, the JSON encoding was a mistake.

                                                                                        Was JSON not fast enough?

                                                                                        1. 4

                                                                                          It’s not about JSON per-se, but more about how message signing works. In SSB, every post is represented as a JSON blob signed with your publick key, and it expects other clients to validate this, as well as produce valid JSON messages.

                                                                                          The spec goes over the requirements of a valid message, as well as the steps to compute a valid signature. Unfortunately, it assumes things like key order (which the official JSON spec doesn’t say anything about), indentiation, spacing, etc. (This all goes back to how the V8 engine implements JSON.stringify()). This adds a lot of complexity when implementing SSB in another language, as most JSON libraries won’t care about specific formatting when printing, and specifically the key order requirement makes it quite complicated.

                                                                                          All in all, it’s not the end of the world, but it adds enough friction to make SSB pretty dependend on the “blessed” javascript implementation.

                                                                                      3. 5

                                                                                        I use it regularly. Why..? After being a heavy social media user on the usual platforms, I’ve pretty much removed myself and don’t participate, but Scuttlebutt is the exception because it’s a fun place to be. Nothing more, nothing less.

                                                                                        1. 3

                                                                                          What difficulties and problems were raised?

                                                                                          1. 12

                                                                                            There’s two big obstacles I’ve seen that seem specific to Scuttlebutt:

                                                                                            • There’s no easy way to share identities across devices. If you want to use Scuttlebutt on your computer and your phone, they’ll be separate accounts.

                                                                                            • The protocol is built around an append-only log, which I’m not convinced is a good principle for any social network. Inadvertent mistakes are forever (eg. I post an unboxing photo that has my unredacted invoice visible; I paste an embarrassing link by accident).

                                                                                              It also seems like you could grief pub servers (the Scuttlebutt “hub nodes” that federate content more widely). What happens if someone posts a bunch of illegal content to the pub? As I understand it, all the pub users will pull that content down. You might be able to blacklist certain posts in your client, and you can block users, but their content is still on your device. (Bitcoin has faced similar problems.)

                                                                                            1. 2

                                                                                              Your objection to log storage is valid, but there are ways around it. The data format could have ways to redact posts out of the log while leaving the overall integrity intact; in fact all revisions of a post other than the latest one could be redacted.

                                                                                              Of course the redactions need to be propagated, and there’s no guarantee every copy will be redacted, but that’s an intrinsic problem with most P2P protocols, since distributed caching/replication is so important for availability.

                                                                                              1. 1

                                                                                                Good points.

                                                                                                Also ironically where Facebook could have a chance to differentiate themselves, but chose to go in almost the exact different direction:

                                                                                                • “with federated networking you are trusting each and every host that your host ever federated with to delete a post, with us, once you click delete it is gone. Worldwide. Same with sharing: if you share something with your close friends it stays there. With a federated network it depends on every host around the globe sticking implementing the rules correctly and sticking to the rules.”

                                                                                                • fortunately IMO Facebook messed up massively early on and now everyone in tech now they are fundamentally untrustworthy.

                                                                                              2. 8

                                                                                                The main problem I saw when I looked into it was that it was a single program rather than a well-defined protocol that anyone in the ecosystem could implement.

                                                                                                This might have changed by now, but (for instance) there were aspects baked into the protocol that fundamentally prevented you from building a compatible client unless you used a JSON serializer with the exact same behavior as node.js, because the cryptographic identity of a post was based on a checksum of the output of that particular serializer rather than some inherent property of the post. An easy mistake to make, but one with far-reaching consequences.

                                                                                                1. 6

                                                                                                  That’s my issue as well. It relies heavily on the interaction of a few dozen NodeJS repos. Different frontends all rely on the same backend code, making it not-that-diverse.

                                                                                                  Also, while the protocol itself is well documented and designed, there are some obvious shortcomings. The protocol relies on hashes of pretty-printed JSON. The last time I checked for documentation on HOW to pretty-print that json, it was documented as “like v8 does it”. Tell you - it’s REALLY hard to format JSON like V8 JSON.format(x, true) does. Especially floating point numbers.

                                                                                                  Now this could easily be fixed by changing the protocol from hash-of-pretty-printed-json-subobject to hash-of-blob. ({"data": "{\"foo\" 42}", "hash": ...} vs. {"data": {"foo": 42}, hash: ...} But you can’t do that without breaking compatibility. Worse, relying on hash-chains, you need to implement the old behaviour to be able to verify old hash chains.

                                                                                                  1. 5

                                                                                                    That’s my top issue too. Which is sad, because there’s so much prior art on canonicalizing JSON, going back to 2010 or so.

                                                                                                    The spec is fine, but limited. It doesn’t cover all of the interactions between peers; there are plenty of messages, and properties in the schema, that aren’t documented.

                                                                                                    A lot of the discussion and information about the protocol takes place on Scuttlebutt itself, meaning it’s (AFAICT) invisible to search engines, and accessible over HTTP only through some flaky gateways that often time out.

                                                                                                    The main client is (like everything else) written in JS, so it’s a big Electron app, and in my experience very slow and resource hungry. I only joined two pubs and followed a handful of people and topics, but every time I fire up the app, it starts a frenzy of downloading and database indexing that lasts a long time. (They use a custom log-based DB engine written in JS.)

                                                                                                    Part of the slowness must be because when you join a pub you’re implicitly following every user of that pub, and I believe all of the IDs they’re following. So there’s kind of an explosion of data that it pulls in indiscriminately to replicate the social graph and content.

                                                                                                  2. 4

                                                                                                    Reading the previous discussions shows a degree of frustration and scepticism. I’m sure it’s changed, and many of the questions will have been addressed, but the previous discussions are unconvincing.

                                                                                                    Here are some links to some of them … it’s worth reading them in context:





                                                                                                  3. 2

                                                                                                    I’m one of the core developers. Happy to answer any questions.

                                                                                                    1. 1

                                                                                                      Despite the fact it’s in a ‘junkyard’, I believe this issue remains unresolved, which I believe effectively means that:

                                                                                                      1. scuttlebutt is not cross platform and only works on x86
                                                                                                      2. It’s difficult to implement scuttlebutt libraries and clients in other languages

                                                                                                      Limiting development to people who like nodejs, and limiting usage to x86 devices (when it seems like the sort of concept that should work well with mobile devices) massively reduces its appeal.

                                                                                                      I would be happy to find that I’m wrong.

                                                                                                      1. 3

                                                                                                        You’re off on this one. I’m running SSB on ARM64 just fine, also many pubs are actually just raspberry pis on some closet.

                                                                                                        I is still difficult to implement SSB in other languages mostly because of the amount of work than technical challenges. The mistakes of the past are well understood at this point even if not fixed. At the moment there are 2 Rust based implementations and one based in Go. IIRC there is also implementations in Elixir and Haskell but I am not sure how far they are. I’ve toyed with a client mixing C and Lua (just a toy prototype but it worked).

                                                                                                        1. 2

                                                                                                          It definitely didn’t work on arm last time I tried, so it’s good to hear they’re making some progress. It was the prototype Haskell implementation which pointed to that issue as a blocker: it looks like it hasn’t been updated since, so probably doesn’t work.

                                                                                                          1. 2

                                                                                                            I know it is not what you’re looking for but the JS implementation works fine under Linux/ARM64, which is also how the mobile apps are running.

                                                                                                            My daily driver is a Surface Pro X with an ARM64 CPU. I’ve run go-ssb as native ARM32 binary and the Electron based client apps under win32 x86-32 bits emulation. The reason for it is just that for the love of all that is sacred I can’t find out how to build the native nodejs modules as ARM64 binaries.

                                                                                                          2. 1

                                                                                                            Could you link to the Haskell implementation? I’d be interested in working on that!

                                                                                                      1. 3

                                                                                                        So, and I mean this only slightly facetiously, is there any point in learning any language other than Rust at this point? I feel like all the momentum is with Rust and that languages like Go (which I prefer to Rust), the ML languages, and C++ are all dead in the water and basically just in maintaining-existing-codebases mode…

                                                                                                        In other words, for the vast majority of non-web/non-desktop projects it would seem silly given the current zeitgeist to start them in any language other than Rust/JavaScript.

                                                                                                        (I say this because the Linux kernel is notoriously conservative on language use, and with good reason…obviously other languages are gonna be around a long time but it seems like the only compiled typed language I hear about anymore is Rust.)

                                                                                                        1. 11

                                                                                                          the ML languages … are dead in the water

                                                                                                          Haskell and Ocaml are still moving forward well. Haskell recently merged in Linear Types, which goes a long way to making the type system both richer and more intuitive.

                                                                                                          C++ are all dead in the water

                                                                                                          Take a look at C++20. It has some great features. Some undoubtedly inspired by Rust, but many others that have been working their way into the standard for over a decade. C++20 has lambdas, option types, awkward[1] sum types, designated initializers, and more.

                                                                                                          In other words, for the vast majority of non-web/non-desktop projects it would seem silly given the current zeitgeist to start them in any language other than Rust/JavaScript.

                                                                                                          At the end of the day, Rust is still a low-level language [2], which means the programmer is in charge of keeping track of when data is copied, when references to data are passed around, and when data goes out of scope. While the borrow checker certainly helps in this process, it’s still something that the programmer needs to be aware of while coding, and this has its own cognitive load. Often times, when programmers reach for a low-level language, it is specifically because they want to tackle some aspect of memory management (whether that’s custom allocation, data sharing, real-time performance, or other things) differently from the way a more standard managed runtime would. This can lead to situations where the borrow checker can actually make it more, rather than less, difficult (due to non-standard memory management techniques that the programmer consciously wants to undertake) to develop and reason about code.

                                                                                                          Rust is also difficult to use in the following situations:

                                                                                                          • Interfacing with binary libraries
                                                                                                          • Using an alternate libc
                                                                                                          • Using a custom memory allocator
                                                                                                          • Developing for the Android NDK

                                                                                                          Moreover, Rust is a complicated language. The surface area of the language is large and that makes understanding its nuances both difficult and, at times, time consuming. Other languages like Nim and Zig take alternate decisions that also enable low-level development with greater safety and lower cognitive load than C.

                                                                                                          Then there’s the world of scientific computing. C++, C, and Fortran have long been the stalwarts of scientific computing, but now we have a new crop of dynamic languages like Python, Julia, and R, that are all great. Rust still lacks interfaces to many of the really big scientific-computing frameworks[3], whereas many frameworks (such as Stan or CLASP) are written in C++ and offer C++ libraries out of the box, along with Python, R, and sometimes Julia wrappers. One can certainly wrap the interface for these libraries themselves into Rust, but in C++ or Python, I can just grab the library and start developing.

                                                                                                          I think it’s unfortunate that due to the vocal Rust fanbase on certain parts of the internet that some folks feel pressured to use Rust. Rust is a fantastic language (IMO), but there certainly are interesting, performant, and practical alternatives out there that can often be better for certain usecases.

                                                                                                          [1]: I call them awkward because they aren’t as neatly baked into the language as they are in Rust, but C++ offers the well written any and variant types.

                                                                                                          [2]: It’s funny because literature from a few decades ago largely considered C++ to be a “higher level language”, but here we are.

                                                                                                          [3]: Others often call this machine learning, but the proliferation of scientific computing is not at all restricted to ML. Stan is a Bayesian inference library and CLASP is an Answer-Set Programming language/environment.

                                                                                                          1. 9

                                                                                                            Take a look at C++20. It has some great features.

                                                                                                            I think C++’s problem has never been its lack of features.

                                                                                                            My approach has served me quite well: If I see new software being written in C or C++, I’m going to ask “why?” – and if there is no really really convincing reason – I stay away.

                                                                                                            1. 4

                                                                                                              Developing for the Android NDK

                                                                                                              With my initial contributor of Android support for Rust hat on, I am curious about this. Yes, it always could be better, but what’s the specific problem? Rust for Android is production ready, in the literal sense of the word. It has been years since Firefox for Android (>100M installs) shipped millions of lines of Rust code in production. Cloudflare also shipped WARP for Android, whose WireGuard implementation is written in Rust.

                                                                                                              1. 3

                                                                                                                It’s funny because literature from a few decades ago largely considered C++ to be a “higher level language”, but here we are.

                                                                                                                “High” is a great term for describing languages this way, because it’s inherently context dependant; you can be high on a ladder or high on an aeroplane. That the heights are so different is not a problem; you wouldn’t use an aeroplane to clean your roof.

                                                                                                                1. 2

                                                                                                                  Haskell recently merged in Linear Types

                                                                                                                  I guess you mean GHC, not (standard) Haskell, insofar that even exists.

                                                                                                                2. 4

                                                                                                                  I think for down and dirty systems programming, probably not.

                                                                                                                  For distributed systems, Java is still in the mix at most companies. Rust isn’t really what I would call “web yet” (despite what the web site says) and Go is a much easier lift for that sort of backend work.

                                                                                                                  I think there is still a good number of options on the backend.

                                                                                                                  For the front end, I would only consider TypeScript at this point.

                                                                                                                  1. 2

                                                                                                                    i’m personally perfectly happy using ocaml, and i think the language has a decent bit of momentum these days. it’s a really good statically compiled applications language; it’s not quite a systems language but then neither is go, both of them depend on a gc.

                                                                                                                    for a systems-level language i’m really hoping that D will catch on as cleaner than C++, more capable than go and easier to use than rust, but it doesn’t seem to be gaining popularity as fast as it should. rust is a perfectly fine language too and i keep meaning to do a serious project in it, but it definitely has a higher entry barrier than D does.

                                                                                                                    1. 2

                                                                                                                      I don’t think it’s so black and white. Go has some serious momentum right now. It’s a very pragmatic and practically useful language. Rust is great but has many downsides, e.g. the compiler isn’t anywhere close to as fast and it is far more complex to learn and use.

                                                                                                                      1. 4

                                                                                                                        Go has occupied the “devops tools” niche, but thankfully it didn’t really take off anywhere else.

                                                                                                                        the compiler isn’t anywhere close to as fast

                                                                                                                        It’s gotten really good at being incremental though. Even on my low-power laptop, cargo run after changing a couple source files is pretty quick. Faster than TypeScript for sure. Also I wonder how many Linux users are just losing lots of time to linking with some awfully slow linker like GNU BFD ld instead of LLD.

                                                                                                                        1. 1

                                                                                                                          but thankfully it didn’t really take off anywhere else.

                                                                                                                          Not sure what you mean here. It’s the language of the cloud. Every major cloud provider has an official Go SDK and the Go team at Google is actively involved in supporting this. Many of the CNCF projects are written in Go.

                                                                                                                          e.g. see go-cloud or the support in gcloud functions.

                                                                                                                          For CLI tools as well, it’s become preferred over scripting languages like python and ruby. e.g. fzf is written in Go, the new GitHub CLI is in Go. While rust is used in newer CLIs as well, it’s usually only for performance critical CLIs like ripgrep.

                                                                                                                          It’s gotten really good at being incremental though.

                                                                                                                          Agreed. I still appreciate just how insanely quick Go is, especially when building on a new machine or pulling a dependency. I never have to really wait for things to compile.

                                                                                                                          1. 4

                                                                                                                            Many of the CNCF projects are written in Go

                                                                                                                            That is exactly what I meant by ‘the “devops tools” niche’! Maybe I should’ve used the “cloud” buzzword instead of “devops”, but I like calling it “devops”.

                                                                                                                            e.g. fzf is written in Go

                                                                                                                            fzf is hardly a unique tool, I have a list of these things here :) Personally I use fzy, which is pretty popular and actively developed, is packaged everywhere, has a very good matching algorithm.. and is written in C, haha.

                                                                                                                            the new GitHub CLI is in Go

                                                                                                                            Yep, that kind of thing is what I’m not very happy about, but it’s still a pretty small amount of things still, and I can successfully avoid running any Go code on any of my personal machines.

                                                                                                                            1. 3

                                                                                                                              Yep, that kind of thing is what I’m not very happy about, but it’s still a pretty small amount of things still, and I can successfully avoid running any Go code on any of my personal machines.

                                                                                                                              Why does it matter whether a binary you use on your machine is written in Go or not? If you’re building it from source, presumably your distribution’s build scripts should take care of orchestrating whatever Go code is necessary in order to compile the package; and if you’re just pulling an executable why does it matter what language was used to produce the binary?

                                                                                                                              1. 6

                                                                                                                                I think GP’s answer is great and insightful, just adding my own take:

                                                                                                                                I try to avoid code by people who think LARPing programming-in-the-1960ies is a reasonable choice to build things in 2020, because I’d rather not find out which other “reasonable” choices they have made further down the line.

                                                                                                                                Also, the community’s anti-intellectualism is off-putting for me.

                                                                                                                                1. 6

                                                                                                                                  I’m the kind of person who cares about how the sausage is made :) If I can “boycott” things I just don’t like, I’ll do it. Here’s a great criticism of the language, but what made me hate it was the internals of the implementation:

                                                                                                                                  • the fully static binary thing, not even using libc syscall wrappers
                                                                                                                                    • first obvious practical problem: you can’t hook syscalls with LD_PRELOAD in go binaries! I don’t want any binaries where LD_PRELOAD does nothing.
                                                                                                                                    • they do raw syscalls even on FreeBSD, where libc is the public API and raw syscalls are not. They completely ignored FreeBSD developers’ messages about this and went ahead with the private API. Yeah sure, the syscalls are backwards compatible, it works fine (until someone disables COMPAT_FREEBSDn kernel options but whatever)..
                                                                                                                                    • but porting to a new platform is hell! Yeah, for the most popular ABI (Linux) there’s enough contributors to write custom syscall wrappers for all CPU architectures, but any {less-popular OS + less-popular CPU ISA} combo just won’t be supported for a long time. I’ve started porting Go to FreeBSD/aarch64, abandoned it in frustration, but others have picked it up and finished it. Cool, I guess. But we’ll have the same problem with FreeBSD/powerpc64, FreeBSD/riscv64 etc! (This is the most “practical” consideration here, yes.)
                                                                                                                                  • which brings me to how these syscall wrappers (and other things) have to be written.. Go uses a very custom toolchain with a very custom assembler which is completely alien to normal unix conventions. It’s not easy to write it. Oh, also, it is rather half-assed (this affects applications way more than it affects porting Go itself):
                                                                                                                                    • it doesn’t support SIMD instructions, and if you want to call something fast written in C or normal sane assembly, you’d have to take the overhead of cgo (because the Go calling convention is custom)
                                                                                                                                      • and to not take the overhead, people have written things like c2goasm. Just read that readme and let that sink in!!
                                                                                                                                    • it doesn’t even seem to support all addressing modes of amd64! When writing a binding to SIMD base64, I couldn’t use c2goasm because I couldn’t express mov al, byte ptr [rax + base64_table_enc] in Go assembly. (There were probably ways around it, like only passing the offset into the normal-assembly functions and letting them use that offset to look up in the table.. but I was already fed up with the awfulness and just used cgo.)
                                                                                                                                  1. 2

                                                                                                                                    (I’d like to point out that I too care how the sausage is made and I loved your phrasing on that point.)

                                                                                                                                    Obviously what follows is solely my opinion, but I feel like some of those criticisms are unjustified.

                                                                                                                                    From your comment:

                                                                                                                                    The static binary thing is a huge boon in some situations. We have a remarkably heterogenous mix of Linux versions in the field and yet the same Go binary “just works” on all of them. That’s an enormous burden off our backs.

                                                                                                                                    That Go’s assembler is weird is I suppose problematic, but for the “average user” of Go that’s just an implementation detail that can be safely ignored.

                                                                                                                                    From the linked rant:

                                                                                                                                    Yes the file permissions APIs are weird because the models for Windows and *nix are so different. I feel like that’s a very specific issue that the rant generalizes to the whole language. In general, a large part of the rant focuses on how Go doesn’t do the right thing for Windows which may be true but it doesn’t seem to be quite that bad…I know a lot of people who have written large Go codebases that run on Windows and are very happy with it. In the rant it comes across, IMHO, as nitpicking.

                                                                                                                                    Later on the rant goes on about how a Go project pulls in a bunch of dependencies and then transitively pulls in even more and so on…I feel like comparing that to Rust isn’t quite fair. I’m not super-familiar with Rust but several of the projects I’ve looked at pull in huge numbers of crates which pull in more crates and so on. It seems like it might even be worse in Rust because of the smaller standard library and, given the Actix controversy, it doesn’t seem like you can rely on third-party Rust crates being inherently safe.

                                                                                                                                    Finally, though, and these are again my opinion:

                                                                                                                                    • Rust feels chaotic. There’s no formal specification of the language. The Rustonomicon says this:

                                                                                                                                      Unfortunately, Rust hasn’t actually defined its aliasing model. 🙀 While we wait for the Rust devs to specify the semantics of their language, let’s use the next section to discuss what aliasing is in general, and why it matters.

                                                                                                                                      That does not fill me with warm fuzzies.

                                                                                                                                    • Documentation seems lacking. There are a lot of TODOs everywhere.

                                                                                                                                    • There’s a new release every six weeks so I always feel like I’m playing catch-up. The “Editions” mechanism seems to help some here, but I went to try to build a Rust program (alacritty) and my Rust version was too old…and it wasn’t that old.

                                                                                                                                    • I don’t like the community. I know that this may come as a shock but I have gotten this “superiority complex” feeling from the Rust community. Any criticism of or concerns about the language seems to be met with “no, you’re wrong, Rust is the best! Your language sucks! Why would you use any tool other than Rust?” Sometimes it’s more polite but it’s still there. Any time someone talks about another language (Go, C++, whatever), someone has to come in and say “why don’t you rewrite it in Rust?” There have been times where I’ve seen on SO or Twitter people almost being made to feel foolish for daring to use some language other than Rust.

                                                                                                                                    • Again on the community, one of the core developers made a joke about C being designed by men and breaking constantly and COBOL being designed by women and just working. When a guy on Twitter said that was kinda sexist, she accused him of “mansplaining” and said he was telling her the joke was sexist rather than understanding its “obvious meaning” (which apparently wasn’t so obvious)…which I found kind of bothersome, especially since the community seemed to agree and anyone who took offense was called sexist (and trust me, I’m as leftist/feminist as they come but that doesn’t and shouldn’t mean that I hate men). This again felt like the “Rust community is superior” thing again: the Rust community was correct and those who disagreed with them were wrong, inherently so.

                                                                                                                                    1. 2

                                                                                                                                      We have a remarkably heterogenous mix of Linux versions in the field and yet the same Go binary “just works” on all of them

                                                                                                                                      Sure, do that on Linux where the syscalls are the public API. Zig does that too, but IIRC it has an option to use libc on Linux, so you could make LD_PRELOAD work and everything, say when you’re building distro packages or personal binaries that you don’t want running on ancient distros.

                                                                                                                                      I just really don’t like that a) there’s no choice, only direct usage of syscalls and b) this is forced on platforms that don’t consider raw syscalls as public API.

                                                                                                                                      (Also, something dynamically linked to glibc will work on all glibc distros of the last N years if you use the headers from an N years old version. With a language like Rust that has its own definitions of libc symbols instead of C headers, these definitions usually are as backwards-compatible as possible already.)

                                                                                                                                      for the “average user” of Go that’s just an implementation detail that can be safely ignored

                                                                                                                                      Yes. Well, I’ve seen what’s inside, so it’s my mission to tell everyone how cursed it is there. Feel free to ignore me!

                                                                                                                                      After all, all of computing is cursed on the inside somewhere :D But these particular experiences are what made me scream at computers the most, so I’m sharing them. Some people do care about implementation details of a thing even when they don’t modify that thing specifically.

                                                                                                                                      There’s a new release every six weeks so I always feel like I’m playing catch-up

                                                                                                                                      Use nightly! :P

                                                                                                                                      someone has to come in and say “why don’t you rewrite it in Rust?”

                                                                                                                                      At least on /r/rust this kind of stuff usually downvoted into oblivion.

                                                                                                                                      Twitter is a trash fire that brings the worst out in people. Heck, some people tweeting that might not have written a line of Rust ever, just shitposting memes they’ve heard somewhere.

                                                                                                                                      trust me, I’m as leftist/feminist as they come but that doesn’t and shouldn’t mean that I hate men

                                                                                                                                      Well that should mean hopefully having a (n extremely-online ironic millennial) sense of humor :) I wouldn’t start serious Discourse™ in defense of the joke, but I’m okay with the joke. “sexism against men” isn’t really meaningful because men are the more dominant group in current societies. (oops did I just start a Discourse?) Anyway tbh you don’t have to “hate” anyone to find this funny.

                                                                                                                                      1. 3

                                                                                                                                        trust me, I’m as leftist/feminist as they come but that doesn’t and shouldn’t mean that I hate men

                                                                                                                                        Well that should mean hopefully having a (n extremely-online ironic millennial) sense of humor :) I wouldn’t start serious Discourse™ in defense of the joke, but I’m okay with the joke. “sexism against men” isn’t really meaningful because men are the more dominant group in current societies. (oops did I just start a Discourse?) Anyway tbh you don’t have to “hate” anyone to find this funny.

                                                                                                                                        No doubt. The joke itself was funny. What bothered me was that someone responded with a polite “hey, that seems uncool” and then that person was called “sexist” and a “mansplainer”. I’m all for off-color humor but if someone you’re ostensibly trying to welcome to the community says that it bothered them, the correct response is “sorry, it was a joke but I could see how it could be taken in that manner and I apologize”…not attacking them. That this behavior was from a Rust core team member made it even worse.

                                                                                                                                        But that whole thing was really just a footnote. What pushes me away more is the constant slamming of other tools and languages, which really gets grating. I don’t hang out on Reddit (if I wanted to hear about how COVID-19 is a plot by Obama to make Trump look bad or how same-sex marriage is Satanism I’d just talk to my father-in-law) but the…zealous…behavior of Rustaceans pops up just about everywhere I frequent, including my beloved from time to time..

                                                                                                                                        Long story short (too late!), I like Rust quite a bit, but there are some negatives that I wish would be addressed.

                                                                                                                      1. 4

                                                                                                                        This site has a white div over the content unless CSS is disabled. Awful.

                                                                                                                        1. 1

                                                                                                                          Hmm seems to work fine for me.

                                                                                                                        1. 12

                                                                                                                          i’ve been really digging lwn lately

                                                                                                                          1. 6

                                                                                                                            Same. Just subscribed!

                                                                                                                            1. 6

                                                                                                                              oh you know I just realized all the posts I’ve loved recently have been @benhoyt. Good job, Ben!

                                                                                                                              1. 14

                                                                                                                                You’re welcome. Do subscribe – that’s how the (very small) team makes their living and how the site keeps going.

                                                                                                                          1. 4

                                                                                                                            Hiring at

                                                                                                                            Feel free to hit me up at my email!

                                                                                                                            We love to see open source involvement and our codebase is mainly Go/Typescript/C/Bash

                                                                                                                            1. 1

                                                                                                                              Are you looking for new grads

                                                                                                                              1. 1


                                                                                                                            1. 1

                                                                                                                              Loved this article! I’ve had the exact same thought, HTML really is just structured data, no good reason not to use something like JSON.