1. 34

    I haven’t received an PRs requesting this change but would like to find a polite way to decline that minimizes the bikeshedding.

    Does anyone have success stories for handling this?

    I think it’s pretty easy to start a project with main instead of master, but it seems like a boring housekeeping task on existing stuff.

    1. 46

      Just say no and close the issues as “won’t do”. Nobody is owed an explanation.

      1. 4

        I showed one of my supervisors an article where it talked about github and microsoft and other companies changing it. It ended up turning into company policy about a month later - albeit low priority. Technically you could call it bikeshedding, but a lot of “social justice” issues seem utterly trivial to those not advocating or affected by it - so I don’t know.

        1. 4

          Bikesheds are really important to bikers right? I think the point of bikeshedding is that it’s focusing on not very important things and crowding out important things.

          I think it’s cool if company policy is to use main instead of master as that’s pretty easy to do consistently. And companies can spend money on whatever they want (I remember using VSS and ClearCase).

          I’ve not actually met or even read anything from someone offended by a “master” branch. I’ve read metaoffense, or potential offense, but not anyone affected by it. Of course people advocating have a belief that it is positive, I think. But I’ve yet to see anyone affected.

          Of course, many other issues have great affect and I’m not trying to discount all social justice topics, some that I think are very important (eg, racial equality).

          1. 1

            I have not yet heard of anyone directly affected either. I believe there is a side benefit to this, it could certainly be interpreted as a politically good move internally - but there are a lot of little things that can be considered trivial. But when taken as a group, I feel that it makes other conversations easier (kind of an appeal to the overton window).

            1. 9

              I don’t know, I’ve had bad experiences with making changes based on irrational feedback as it leads to more and more irrational requests.

              Just because people have good intentions, doesn’t mean it will have good outcomes. I think the motivation here is positive and I interpret it as “let’s reduce potentially offensive things.” I think it’s better to put more energy into helping people not care about unimportant things. We choose what offends us and what we focus on and give energy. I think, for sustainability, it’s better to learn to not care than to sanitize everything against what offends anyone.

              Especially for programming, it’s so important to focus on real stuff.

              I remember a while back when people were calling for removing swearwords from Linux and the argument was “why not just make the change?”

              I’ve mostly experienced this with internal changes where people request changes for not well understood reasons and giving in and making them leads to an unproductive cycle of churn and discussion.

      1. 11

        I used to use a lot of util alternatives, but now I just stick to the standard ones: find, make, man, vim, grep, xargs, watch, etc.

        You still have to learn them anyway because they’re installed everywhere by default. You can’t always install random tools on a system.

        1. 14

          Not sure I buy this argument.

          Becoming facile with the standard available everywhere tools is a good investment, to be sure, but some of the tools cited offer a huge value add over the standard tools and can do things they can’t.

          1. 2

            In my own usage, I haven’t really noticed too much of a difference. The things I have noticed are that the non-standard tools are sometimes faster and they tend to use more common --flags. entr is a bit unique, but I’ve also been able to replace entr with watch to achieve basically the same result.

            What are some of the huge value adds that I’m missing out on?

            1. 5

              Utilities like ripgrep and fd natively respecting your .gitignore is a huge feature.

              1. 5

                Oh, man. That’s actually the feature that made me leave ripgrep. I personally prefer to not hide a bunch of information by default. Let me filter it. I didn’t even realize it did that at first. I’ve spent hours debugging something only to realize that ripgrep was lying to me. Definitely user error on my part and I eventually found you can disable that behavior. But that made me realize grep suited my needs just fine.

                1. 3

                  Yep, same here. I think it’s a horrible default and easily the biggest problem I’ve ever encountered with ripgrep.

                  1. 5

                    I find that to be an ok default and for the odd time you don’t want that behaviour there’s --no-ignore-vcs which you can also set in your ripgrep config file

              2. 3

                watch(1) as far as I know doesn’t do anything with inotify(7), so it’s limited to a simple interval instead of being event based. As others have pointed out, you could use inotifywait(1) and some relatively simple scripts to obtain similar results.

                That being said, I still use find, grep and others regularly, these are just a little easier to work with on the daily when they’re available. fd in particular has nicer query syntax for the common cases I have.

                1. 3

                  Yeah, watch doesn’t watch (ha) for file changes, but my goal is to avoid having to rerun commands manually while I work on a file. watch(1) can achieve this goal. Yes, it’s inefficient to rerun the build every -n .1 seconds instead of waiting for file changes, but I end up reaching my goal either way, even if I have to wait 1 second for the build to start.

                  Although, I can definitely see how this could be painful if the command is extremely slow.

                  1. 5

                    I work with Python… so it goes without saying that my tests are already slow, but also many of them are computationally expensive (numpy), so they can take several minutes to complete. Obviously that’s not the same situation for everyone, use what works for you! The whole point of my post is to share tools that others may not be aware of but may find helpful.

                2. 1

                  What are some of the huge value adds that I’m missing out on?

                  Thinking about this I may have over-promised, but one thing that occurs is vastly reduced cognitive load as well as a lot less straight up typing.

                  For example I personally find being able to simply type: ag "foo" as opposed to: find . -name "*" -exec grep "foo" \; -print

              3. 5

                I’m a developer, so it’s super worth it for me to optimize my main workflow. I know the standard tools well enough that I can pretty easily revert back to them if there is one, but some of these don’t have an alternative in the standard *nix toolset. entr(1) in particular has nothing equivalent as far as I know.

                1. 3

                  How about inotifywait?

                  1. 2

                    entr is built on the same os API, inotify(7). You could probably get inotifywait to act in a similar manner with some scripting, but it does not do the same thing.

                    1. 2

                      That’s all I’ve ever used inotifywait for: watch some files or directories, and run scripts when change events occur. But I didn’t even know about entr. It does look relatively streamlined.

                      1. 1

                        Your script(s) have to implement the behavior that entr(1) implements though, so while it’s technically possible it’s far from a ‘single command’ experience, which I find very nice.

                2. 1

                  If you have to work on systems with standard tooling a lot and no option to install stuff, fair. But for me, and I think a lot of other devs, it’s pretty feasible to install non-standard tools. I almost never work on another machine than my own. Most people customize their personal machine anyway.

                  One rule I have is that scripts are always written in plain POSIX compliant shell for portability with the standard tools.

                  1. 1

                    I almost never work on another machine than my own.

                    I wonder if this is a side effect of the trends towards infrastructure as code, “cattle not pets”, containerization, etc etc etc.

                    In general I’ve found the same: the most work I do on a remote box is connecting to a running container and doing some smoke testing.

                    1. 3

                      I feel like in the era where shared UNIX hosts were more common, I would just cart binaries around in my home directory – which was often an NFS directory shared amongst many hosts as well.

                1. 22

                  If you’re the kind of person who’s willing to put up with a learning curve and a smaller ecosystem of plugins to gain access to a powerful editing model, also consider Kakoune. It’s like Vim, but moreso.

                  1. 8

                    I simply can’t go back to Vim after using Kakoune. Feels like a step back to me.

                    1. 8

                      I am in the same boat. I think by and large Kakoune is a step up from Vim. That said, it is not emulated in very many other places (it is in vs-code now via dance) – so you do lose the ability to pop into IDE of your choice and have a good experience.

                      1. 1

                        Dance is cool, but there are a lot of little things that does not work the same way and it’s annoying.

                        When i’m at the beginning of a line and press x, it doesn’t select the line. It selects the line below. If i go forward one character before pressing x, it works.

                        It’s good enough…

                      2. 1

                        I wish emacs had a kakoune mode like the evil mode. It would help me pick up emacs finally. Each time I have tried evil, I got stuck in small differences with vim.

                        1. 1

                          Unfortunately every emulation of another editor is not the same thing.

                          I use kakoune to write small programs and scripts, but i have vscode as well. Vscode has a plugin called “dance”, it is a kakoune simulation plugin. It works, but not very much…

                          The problem is the little things… there is always something that doesn’t really work the same and becomes annoying.

                      3. 6

                        How would you say the transition to Kakoune from someone who’s been using vim for awhile is like? I took it for a spin and am very confused, but I can already see things that I like.

                        1. 6

                          I switched from vim to kakoune about 6 months ago. I think the majority of the users, including the author himself, came from vim. My strategy was to switch entirely for a week then decide if it was worth committing fully or not. I never went back to vim. Once you get over the initial hurdle of unlearning your vim muscle memory, kakoune is very intuitive to learn, more so than vim in my opinion.

                          1. 4

                            Seconding ifreund’s experience, I came to Kakoune after maybe 20 years using Vim and it took me maybe a month for Kakoune to feel comfortable. The biggest hurdles for me were a few commonly-used key-bindings that changed (x in Kakoune is “select the current line”, not “delete the character under the cursor), and that Kakoune is a little bit like that “Snake” game on old Nokia phones: as you’re moving around, you need to be conscious of where the “tail” of your selection is as well as the “head”.

                            The thing I love most about Kakoune is global search-and-replace. In Vim, I’d often wind up in a cycle of “write a replacement regex, try it, check the document, find a mistake, undo, try again”, which was particularly frustrating when the thing I wanted to select was easy to describe in Vim’s normal mode (like % to select a matched pair of brackets) but difficult to describe in regex. Meanwhile, in Kakoune, I can match a regex across the entire document, producing multiple selections, cycle through them all to check I’ve selected the right things, manually adjust them with normal-mode commands or even drop false-positives, and then do the replacement. It’s more work than the best-case Vim equivalent, but far smoother and more pleasant than the worst-case Vim equivalent.

                            1. 2

                              I know you don’t use Vim anymore but for anyone else who has the problem described in the second paragraph: traces.vim offers a live preview that makes searching and replacing easier, if not quite as easy as it seems to be in Kakoune. As you’re typing a :s command, the plugin will highlight the parts of the file or line that would be matched, and it will also show you what they would be replaced with. It’s pretty magical.

                          2. 3

                            I really want to, but the hidden benefit of Vim keybindings is they translate to other programs too (I assume Vim people are just very militant and force devs to support them ;) ) so I can use IntelliJ or Emacs or even a web-based IDE and have access to those bindings. If I changed muscle memory to Kakoune, I’m going to be in trouble for hopping between programs.

                            1. 3

                              powerful editing model

                              Can someone pitch to me, the established emacs user, what the benefits of Kakoune are? I have multiple cursors package enabled, plus helm and swoop (intra file fuzzy matching), but I presume Kakoune presents more benefits.

                              1. 6

                                EDIT: This explains it better https://kakoune.org/why-kakoune/why-kakoune.html


                                Disclaimer: I’ve used Emacs for fewer than 10 hours in my life. I remember very little.

                                The last time I looked into it, the big difference that Kakoune brings to the table is that nouns come before verbs. This feels minor but in practice it makes discoverability so much easier because you’re deciding which text to act upon before doing an action.

                                For example, in Vim if you want to delete three words you type d3w, and if you realize that you meant to delete 2 words then you have to undo and try again. Kakoune lets you make your selection first, highlighting as you go, and makes it easy to change your selection before taking an action. It’s also constantly auto-completing with all of the possible commands you might want to make, which is much simpler than reading through a manual.

                                1. 2

                                  Not having used the Emacs multiple cursors package (or Emacs at all, really) it’s hard for me to say what the advantages of Kakoune’s editing model might be. If I had to guess, though, I suspect the biggest difference would be that since the Emacs core is only built for single selections, most Emacs functionality (including third-party extensions, etc.) only works with single selections, except for the things that the multiple cursors package specifically modifies. Meanwhile, all of Kakoune’s standard features and third-party extensions deal with multiple selections, so you don’t need a mental model of how single-selection features interact with multiple-selection data.

                                  I don’t know how complete Emacs’ multiple cursors package is, but I expect it has all the same kinds of cursor interactions as Kakoune, like selecting substrings of the selection that match a regex, splitting the selection on a regex, dropping selections that do/do not match a regex, dropping selections interactively, rotating content through selections, etc. If not, that might be another reason to try Kakoune!

                              1. 8

                                I did at one point consider alacritty, but the latency turned me away. How mb/s of text it can render is quite irrelevant when latency is bad.

                                A bug existed about this, but it was closed without fixing.

                                1. 6
                                  1. 1

                                    No. Furthermore, I did not miss that when looking at the bug I linked.

                                    1. 4

                                      Yeah but your comment sounded a bit like the devs had no intentions of improving the latency issue.

                                      Also I hoped that the PR would help you, and all of us with those issues.

                                      1. 2

                                        I’m not as optimistic.

                                        When I see the bug tracker, what I see is that throughput, not latency, was their priority from the start. They didn’t care about latency in 2017 when the bug was opened. This hasn’t changed, and thus it’s unlikely it ever will.

                                        1. 1

                                          The wayland backend is driven by frame-callbacks since https://github.com/alacritty/alacritty/commit/04f0bcaf54ed373128ca0f84ee8fcdd8e52bce23 - any adaptive timing logic and sleeping barber like scheduling should be approached compositor side.

                                  2. 2

                                    How does the latency compare to gnome-terminal? I’ve been thinking that alacritty feels a lot faster.

                                    1. 1

                                      I wouldn’t know. I’ve never even used gnome-terminal.

                                    2. 1

                                      What do you use instead?

                                      1. 1

                                        Plain mlterm.

                                    1. -1

                                      Being it a lot simpler than vim, most people (including myself) use vim.

                                      1. 8

                                        I like how the commands are listed at the bottom. I always use it on servers.

                                        1. 6

                                          I disagree with the notion that most people use vim. Even among sysadmins I know, few of them prefer vim.

                                          1. 4

                                            Being a programmer that works with sysadmins on a daily basis, I can confirm that almost none of them use vim.

                                            EDIT: Not that, that’s a problem. I could care less. Just an interesting tidbit.

                                            1. 2

                                              In your experience what editor is the most used?

                                              1. 6

                                                notepad.exe

                                                :(

                                                1. 1

                                                  Don’t take it the wrong way, but perhaps you need to widen your connections. The vast majority of sysadmins/devops folks I know are vim enthusiasts.

                                                  1. 1

                                                    Huh, that’s very different from my experience. I definitely wouldn’t say that the sysadmins that I’ve worked with are vim enthusiasts, but they all use vi since they can trust that it’ll be installed on a system. Many have had familiarity with ed as well.

                                                    1. 1

                                                      I would say a bit under half know how to use vi/vim. Most probably know the very basics (insert mode, :wq), but in general I don’t know many people who use vim as their editor of choice. There’s a couple emacs guys, but the most common editor I see used on Linux servers is Nano. Most of my current and former coworkers are <30, which might have something to do with it.

                                                1. 2

                                                  I just bought an entry-level mirrorless camera, so I think I’m going to walk around downtown and take some photos. I may also have to help my sister and her boyfriend move.

                                                  1. 7

                                                    https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines but seriously, I don’t see this taking off. Open source OSs can take on Microsoft with enough coders because it’s just software - hardware is a very different business. I wish it could happen, but it’s very doubtful IMHO.

                                                    1. 31

                                                      Depends on what you mean by ‘taking off’. RISC-V has successfully killed a load of in-house ISAs (and good riddance!). For small control-plane processors, you don’t care about performance or anything else much, you just want a cheap Turing-complete processor with a reasonably competent C compiler. If you don’t have to implement the C compiler, that’s a big cost saving. RISC-V makes a lot of sense for things like the nVidia control cores (which exist to set up the GPU cores and do management things that aren’t on the critical path for performance). It makes a lot of sense for WD to use instead of ARM for the controllers on their SSDs: the ARM license costs matter in a market with razor-thin margins, power and performance are dominated by the flash chips, and they don’t need any ecosystem support beyond a bare-metal C toolchain.

                                                      The important lesson for RISC-V is why MIPS died. MIPS was not intended as an open ISA, but it was a de-facto one. Aside from LWL / LWR, everything in the ISA was out of patent. Anyone could implement an almost-MIPS core (and GCC could target MIPS-without-those-two-instructions) and many people did. Three things killed it in the market:

                                                      First, fragmentation. This also terrifies ARM. Back in the PDA days, ARM handed out licenses that allowed people to extend the ISA. Intel’s XScale series added a floating-point extension called Wireless MMX that was incompatible with the ARM floating point extension. This cost a huge amount for software maintenance. Linux, GCC, and so on had to have different code paths for Intel vs non-Intel ARM cores. It doesn’t actually matter which one was better, the fact both existed prevented Linux from moving to a hard-float ABI for userland for a long time: the calling convention passed floating-point values in integer registers, so code could either call a soft-float library or be compiled for one or the other floating-point extensions and still interop with other libraries that were portable across both. There are a few other examples, but that’s the most painful one for ARM. In contrast, every MIPS vendor extended the ISA in incompatible ways. The baseline for 64-bit MIPS is still often MIPS III (circa 1991) because it’s the only ISA that all modern 64-bit MIPS processors can be expected to handle. Vendor extensions only get used in embedded products. RISC-V has some very exciting fragmentation already, with both a weak memory model and TSO: the theory is that TSO will be used for systems that want x86 compatibility, the weak model for things that don’t, but code compiled for the TSO cores is not correct on weak cores. There are ELF header flags reserved to indicate which is which, but it’s easy to compile code for the weak model, test it on a TSO core, see it work, and have it fail in subtle ways on a weak core. That’s going to cause massive headaches in the future, unless all vendors shipping cores that run a general-purpose OS go with TSO.

                                                      Second, a modern ISA is big. Vector instructions, bit-manipulation instructions, virtualisation extensions, two-pointer atomic operations (needed for efficient RCU and a few other lockless data structures) and so on. Dense encoding is really important for performance (i-cache usage). RISC-V burned almost all of their 32-bit instruction space in the core ISA. It’s quite astonishing how much encoding space they’ve managed to consume with so few instructions. The C extension consumes all of the 16-bit encoding space and is severely over-fitted to the output of an unoptimised GCC on a small corpus of C code. At the moment, every vendor is trampling over all of the other vendors in the last remaining bits of the 32-bit encoding space. RISC-V really should have had a 48-bit load-64-bit-immediate instruction in the core spec to force everyone to implement support for 48-bit instructions, but at the moment no one uses the 48-bit space and infrequently used instructions are still consuming expensive 32-bit real-estate.

                                                      Third, the ISA is not the end of the story. There’s a load of other stuff (interrupt controllers, DMA engines, management interfaces, and so on) that need to be standardised before you can have a general-purpose compute platform. Porting an OS to a new ARM SoC used to be a huge amount of effort because of this. It’s now a lot easier because ARM has standardised a lot of this. x86 had some major benefits from Compaq copying IBM: every PC had a compatible bootloader that provided device enumeration and some basic device interfaces. You could write an OS that would access a disk, read from a keyboard, and write text to a display for a PC that would run on any PC (except the weird PC98 machines from Japan). After early boot, you’d typically stop doing BIOS thunks and do proper PCI device numeration and load real drivers, but that baseline made it easy to produce boot images that ran on all hardware. The RISC-V project is starting to standardise this stuff but it hasn’t been a priority. MIPS never standardised any of it.

                                                      The RISC-V project has had a weird mix from the start of explicitly saying that it’s not a research project and wants to be simple and also depending on research ideas. The core ISA is a fairly mediocre mid-90s ISA. Its fine, but turning it into something that’s competitive with modern x86 or AArch64 is a huge amount of work. Some of those early design decisions are going to need to either be revisited (breaking compatibility) or are going to incur technical debt. The first RISC-V spec was frozen far too early, with timelines largely driven by PhD students needing to graduate rather than the specs actually being in a good state. Krste is a very strong believer in micro-op fusion as a solution to a great many problems, but if every RISC-V core needs to be able to identify 2-3 instruction patterns and fuse them into a single micro-op to do operations that are a single instruction on other ISAs, that’s a lot of power and i-cache being consumed just to reach parity. There’s a lot of premature optimisation (e.g. instruction layouts that simplify decoding on an in-order core) that hurt other things (e.g. use more encoding space than necessary), where the saving is small and the cost will become increasingly large as the ISA matures.

                                                      AArch64 is a pretty well-designed instruction set that learns a lot of lessons from AArch32 and other competing ISAs. RISC-V is very close to MIPS III at the core. The extensions are somewhat better, but they’re squeezed into the tiny amount of left-over encoding space. The value of an ecosystem with no fragmentation is huge. For RISC-V to succeed, it needs to get a load of the important extensions standardised quickly, define and standardise the platform specs (underway, but slow, and without enough of the people who actually understand the problem space contributing, not helped by the fact that the RISC-V Foundation is set up to discourage contributions), and get software vendors to agree on those baselines. The problem is that, for a silicon vendor, one big reason to pick RISC-V over ARM is the ability to differentiate your cores by adding custom instructions. Every RISC-V vendor’s incentives are therefore diametrically opposed to the goals of the ecosystem as a whole.

                                                      1. 3

                                                        Thanks for this well laid out response.

                                                        The problem is that, for a silicon vendor, one big reason to pick RISC-V over ARM is the ability to differentiate your cores by adding custom instructions. Every RISC-V vendor’s incentives are therefore diametrically opposed to the goals of the ecosystem as a whole.

                                                        This is part of what makes me skidish, as well. I almost prefer the ARM model to keep a lid on fragmentation than RISC-V’s “linux distro” model. But also, deep down, if we manage to create the tooling for binaries to adapt to something like this and have a form of Universal Binary that progressively enhances with present CPUIDs, that would make for an exciting space.

                                                        1. 6

                                                          But also, deep down, if we manage to create the tooling for binaries to adapt to something like this and have a form of Universal Binary that progressively enhances with present CPUIDs, that would make for an exciting space.

                                                          Apple has been pretty successful at this, encouraging developers to distribute LLVM IR so that they can do whatever microarchitectural tweaks they want for any given device. Linux distros could do something similar if they weren’t so wedded to GCC and FreeBSD could if they had more contributors.

                                                          You can’t do it with one-time compilation very efficiently because each vendor has a different set of extensions, so it’s a combinatorial problem. The x86 world is simpler because Intel and AMD almost monotonically add features. Generation N+1 of Intel CPUs typically supports a superset of generation N’s features (unless they completely drop something and are never bringing it back, such as MPX) and AMD is the same. Both also tend to adopt popular features from the other, so you have a baseline that moves forwards. That may eventually happen with RISC-V but the scarcity of efficient encoding space makes it difficult.

                                                          On the other hand, if we enter Google’s dystopia, the only AoT-compiled code will be Chrome and everything else will be JavaScript and WebAssembly, so your JIT can tailor execution for whatever combination of features your CPU happens to have.

                                                          1. 1

                                                            Ultimately, vendor extensions are just extensions. Suppose a CPU is RV64GC+proprietary extensions, what this means is that RV64GC code would still work on it.

                                                            This is much, much better than the alternative (vendor-specific instructions implemented without extensions).

                                                          2. 2

                                                            Vendor extensions only get used in embedded products. RISC-V has some very exciting fragmentation already, with both a weak memory model and TSO: the theory is that TSO will be used for systems that want x86 compatibility, the weak model for things that don’t, but code compiled for the TSO cores is not correct on weak cores. There are ELF header flags reserved to indicate which is which, but it’s easy to compile code for the weak model, test it on a TSO core, see it work, and have it fail in subtle ways on a weak core. That’s going to cause massive headaches in the future, unless all vendors shipping cores that run a general-purpose OS go with TSO.

                                                            I don’t understand why they added TSO in the first place.

                                                            Third, the ISA is not the end of the story. There’s a load of other stuff (interrupt controllers, DMA engines, management interfaces, and so on) that need to be standardised before you can have a general-purpose compute platform. Porting an OS to a new ARM SoC used to be a huge amount of effort because of this. It’s now a lot easier because ARM has standardised a lot of this. x86 had some major benefits from Compaq copying IBM: every PC had a compatible bootloader that provided device enumeration and some basic device interfaces. You could write an OS that would access a disk, read from a keyboard, and write text to a display for a PC that would run on any PC (except the weird PC98 machines from Japan). After early boot, you’d typically stop doing BIOS thunks and do proper PCI device numeration and load real drivers, but that baseline made it easy to produce boot images that ran on all hardware. The RISC-V project is starting to standardise this stuff but it hasn’t been a priority. MIPS never standardised any of it.

                                                            Yeah this part bothers me a lot. It looks like a lot of the standardization effort is just whatever OpenRocket does, but almost every RISC-V cpu on the market right now has completely different peripherals outside of interrupt controllers. Further, there’s no standard way to query the hardware, so creating generic kernels like what is done for x86 is effectively impossible. I hear there’s some work on ACPI which could help.

                                                            1. 7

                                                              I don’t understand why they added TSO in the first place.

                                                              Emulating x86 on weakly ordered hardware is really hard. Several companies have x86-on-ARM emulators. They either only work with a single core, insert far more fences than are actually required, or fail subtly on concurrent data structures. It turns out that after 20+ years of people trying to implement TSO efficiently, there are some pretty good techniques that don’t sacrifice much performance relative to software that correctly inserts the fences and perform a lot better on the software a lot of people write where they defensively insert too many fences because it’s easier than understanding the C++11 memory model.

                                                              Yeah this part bothers me a lot. It looks like a lot of the standardization effort is just whatever OpenRocket does, but almost every RISC-V cpu on the market right now has completely different peripherals outside of interrupt controllers. Further, there’s no standard way to query the hardware, so creating generic kernels like what is done for x86 is effectively impossible. I hear there’s some work on ACPI which could help.

                                                              Initially they proposed their own thing that was kind-of like FDT but different, because Berkeley. Eventually they were persuaded to use FDT for embedded things and something else (probably ACPI) for more general-purpose systems.

                                                              The weird thing is that Krste really understands the value of an interoperable ecosystem. He estimates the cost of building it at around $1bn (ARM thinks he’s off by a factor of two, but either way it’s an amount that the big four tech companies could easily spend if it were worthwhile). Unfortunately, the people involved with the project early were far more interested in getting VC money than in trying to build an open ecosystem (and none of them really had any experience with building open source communities and refused help from people who did).

                                                              1. 2

                                                                Are the Apple and Microsoft emulators on the “far more fences than are actually required” side? They don’t seem to have many failures..

                                                                1. 2

                                                                  I don’t know anything about the Apple emulator and since it runs only on Apple hardware, it’s entirely possible that either Apple’s ARM cores are TSO or have a TSO mode (TSO is strictly more strongly ordered than the ARM memory model, so it’s entirely conformant to be TSO). I can’t share details of the Microsoft one but you can probably dump its output and look.

                                                              2. 2

                                                                there’s no standard way to query the hardware, so creating generic kernels like what is done for x86 is effectively impossible

                                                                Well, device trees (FDT) solve the “generic kernel” problem specifically, but it all still sucks. Everything is so much better when everyone has standardized most peripherals.

                                                                1. 1

                                                                  That’s the best solution, but you still have to have the bootloader pass in a device tree, and that device tree won’t get updated at the same cadence as the kernel does (so it may take a while if someone finds a bug in a device tree).

                                                                  1. 2

                                                                    For most devices it’s the kernel that maintains the device tree. FDT is not really designed for a stable description, it changes with the kernel’s interface.

                                                                    1. 2

                                                                      FDT is not specific to a kernel. The same FDT blobs work with FreeBSD and Linux, typically. It’s just a description of the devices and their locations in memory. It doesn’t need to change unless the hardware changes and if you’re on anything that’s not deeply embedded it’s often shipped with U-Boot or similar and provided to the kernel. The kernel then uses it to find any devices it needs in early boot or which are attached to the core via interface that don’t support dynamic enumeration (e.g. you would put the PCIe root complex in FDT but everything on the bus is enumerated via the bus).

                                                                      The reason for a lot of churn recently has been the addition of overlays to the FDT spec. These allow things that are equivalent to option roms to patch the root platform’s FDT so you can use FDT for expansions connected via ad-hoc non-enumerable interfaces.

                                                                      1. 2

                                                                        It doesn’t need to change.. but Linux developers sometimes like to find “better” ways of describing everything, renaming stuff, etc. To be fair in 5.x this didn’t really happen all that much.

                                                                        And of course it’s much worse if non-mainline kernels are introduced. If there’s been an FDT for a vendor kernel that shipped with the device, and later drivers got mainlined, the mainlined drivers often expect different properties completely because Linux reviewers don’t like vendor ways of doing things, and now you need very different FDT..

                                                                        The reason for a lot of churn recently has been the addition of overlays to the FDT spec

                                                                        That’s not that recent?? Overlays are from like 2017..

                                                                2. 1

                                                                  Further, there’s no standard way to query the hardware, so creating generic kernels like what is done for x86 is effectively impossible. I hear there’s some work on ACPI which could help.

                                                                  There’s apparently serious effort put into UEFI.

                                                                  With rpi4 uefi boot, FDT isn’t used. I suppose UEFI itself has facilities to make FDT redundant.

                                                                  1. 2

                                                                    With RPi4-UEFI, you have a choice between ACPI and FDT in the setup menu.

                                                                    It’s pretty clever what they did with ACPI: the firmware fully configures the PCIe controller by itself and presents a generic XHCI device in the DSDT as if it was just a directly embedded non-PCIe memory-mapped XHCI.

                                                                    1. 1

                                                                      I have to ask, what is the benefit of special casing the usb3 controller?

                                                                      1. 2

                                                                        The OS does not need to have a driver for the special Broadcom PCIe host controller.

                                                                        1. 1

                                                                          How is the Ethernet handled?

                                                                          1. 2

                                                                            Just as a custom device, how else? :)

                                                                            Actually it’s kinda sad that there’s no standardized Ethernet “host controller interface” still… (other than some USB things)

                                                                            1. 1

                                                                              Oh. So Ethernet it’s not on PCIe to begin with, then. Only XHCI. I see.

                                                                3. 1

                                                                  This doesn’t paint a very good picture of RISC-V, IMHO. It’s like some parody of worse-is-better design philosophy, combined with basically ignoring all research in CPU design since 1991 for a core that’s easy to make an educational implementation for that makes the job of compiler authors and implementers harder. Of course, it’s being peddled by GNU zealots and RISC revanchists, but it won’t benefit the things they want; instead, it’ll benefit vanity nationalist CPU designs (that no one will use except the GNU zealots; see Loongson) and deeply fragmented deep embedded (where software freedom and ISA doesn’t matter other than shaving licensing fees off).

                                                                  1. 3

                                                                    Ignoring the parent and focusing on hard data instead, RV64GC has higher code density than ARM, x86 and even MIPS16, so the encoding they chose isn’t exactly bad, objectively speaking.

                                                                    1. 8

                                                                      Note that Andrew’s dissertation is using integer-heavy, single-threaded, C code as the evaluation and even then, RISC-V does worse than Thumb-2 (see Figure 8 of the linked dissertation). Once you add atomics, higher-level languages, or vector instructions, you see a different story. For example, RISC-V made an early decision to make the offset of loads and stores scaled with the size of the memory value. Unfortunately, a lot of dynamic languages set one of the low bits to differentiate between a pointer and a boxed value. They then use a complex addressing mode to combine the subtraction of one with the addition of the field offset for field addressing. With RISC-V, this requires two instructions. You won’t see that pattern in pure C code anywhere but you’ll see it all over the place in dynamic language interpreters and JITs.

                                                                      1. 1

                                                                        I think there was another example of something far more basic that takes two instructions on RISC-V for no good reason, just because of their obsession with minimal instructions. Something return related?? Of course I lost the link to that post >_<

                                                                        1. 1

                                                                          Interesting. There’s work on an extension to help interpreters, JITs, which might or might not help mitigate this.

                                                                          In any event, it is far from ready.

                                                                          1. 6

                                                                            I was the chair of that working group but I stepped down because I was unhappy with the way the Foundation was being run.

                                                                            The others involved are producing some interesting proposals though a depressing amount of it is trying to fix fundamentally bad design decisions in the core spec. For example, the i-cache is not coherent with respect to the d-cache on RISC-V. That means you need explicit sync instructions after every modification to a code page. The hardware cost of making them coherent is small (i-cache lines need to participate in cache coherency, but they can only ever be in shared state, so the cache doesn’t have to do much. If you have an inclusive L2, then the logic can all live in L2) but the overheads from not doing it are surprisingly high. SPARC changed this choice because the overhead on process creating from the run-time linker having to do i-cache invalidates on every mapped page were huge. Worse, RISC-V’s i-cache invalidate instruction is local to the current core. That means that you actually need to do a syscall, which does an IPI to all cores, which then invalidates the i-cache. That’s insanely expensive but the initial measurements were from C code on a port of Linux that didn’t do the invalidates (and didn’t break because the i-cache was so small you were never seeing the stale entries).

                                                                            1. 1

                                                                              L1$ not coherent

                                                                              Christ. How did that go anywhere?

                                                                              1. 4

                                                                                No one who had worked on an non-toy OS or compiler was involved in any of the design work until all of the big announcements had been made and the spec was close to final. The Foundation was set up so that it was difficult for any individuals to contribute (that’s slowly changing) - you had to pay $99 or ask for the fee to be waived to give feedback on the specs as an individual. You had to pay more to provide feedback as a corporation and no corporation was going to pay thousands of dollars membership and the salaries of their contributors to provide feedback unless they were pretty confident that they were going to use RISC-V.

                                                                                It probably shouldn’t come as a surprise that saying to people ‘we need your expertise, please pay us money so that you can provide it’ didn’t lead to a huge influx of expert contributors. There were a few, but not enough.

                                                                  2. 7

                                                                    Keep in mind an ISA isn’t hardware, it’s just a specification.

                                                                    1. 6

                                                                      That ties into my point - RISC-V is kinda useless without fabbing potential. And that’s insanely expensive, which means the risk involved is too high to take on established players.

                                                                      1. 9

                                                                        According to the article, it seems that Samsung, Western Digital, NVIDIA, and Qualcomm don’t think the risk is too high, since they plan to use RISC-V. They have plenty of money to throw at any problems, such as inadequate fabbing potential. Hobbyists may benefit from RISC-V, but (like Linux) it’s not just for hobbyists.

                                                                        1. 8

                                                                          According to the article, it seems that Samsung, Western Digital, NVIDIA, and Qualcomm don’t think the risk is too high, since they plan to use RISC-V.

                                                                          I think it is more accurate they plan to use the threat of RISC-V to improve negotiating position, use it in some corner cases and as a last ditch hedge. Tizen is a prime example of such a product.

                                                                          1. 2

                                                                            I think it is more accurate they plan to use the threat of RISC-V to improve negotiating position, use it in some corner cases and as a last ditch hedge.

                                                                            Yet WD and NVIDIA designed their own RISC-V cores. Isn’t it a bit too much for “insurance”?

                                                                            The fact here is that they do custom silicon and need CPUs in them for a variety of purposes. Until now, they paid the ARM tax. From now on, they don’t have to, because they can and do just use RISC-V.

                                                                            I’m appalled at how grossly the impact of RISC-V is being underestimated.

                                                                            1. 4

                                                                              Yet WD and NVIDIA designed their own RISC-V cores. Isn’t it a bit too much for “insurance”?

                                                                              I don’t think so – it isn’t purely insurance, it is negotiating power. The power can be worth tens (even hundreds) of millions for companies at the scale of WD and NVIDIA. Furthermore they didn’t have to develop FABs for the first time, both have existing manufacturing prowess and locations. I think it is a rather straightforward ROI based business decision.

                                                                              The fact here is that they do custom silicon and need CPUs in them for a variety of purposes. Until now, they paid the ARM tax. From now on, they don’t have to, because they can and do just use RISC-V.

                                                                              They will use this to lower the ARM tax without actually pulling the trigger on going with something as different as RISC-V (except on a few low yield products to prove they can do it, see Tizen and Samsung’s strategy).

                                                                              I’m appalled at how grossly the impact of RISC-V is being underestimated.

                                                                              Time will tell, but I think that RISC-V only become viable if Apple buys and snuffs out new customers of ARM, only maintaining existing contracts.

                                                                              1. 1

                                                                                I don’t think so – it isn’t purely insurance, it is negotiating power.

                                                                                Do you think they have any reason left to license ARM, when they clearly can do without?

                                                                                Time will tell, but I think that RISC-V only become viable if Apple buys and snuffs out new customers of ARM, only maintaining existing contracts.

                                                                                I see too much industry support behind RISC-V at this point. V extension will be quite the spark, so we’ll see how it plays out after that. All it’ll take is one successful high performance commercial implementation.

                                                                                1. 2

                                                                                  Do you think they have any reason left to license ARM, when they clearly can do without?

                                                                                  I think you are underestimating the cost of rebuilding an entire ecosystem. I have run in production ThunderX arm64 servers – and ARM has massive support behind it and we still fell into weird issues, niches and problems. Our task was fantastic fit (large-scale OCR) and it still was tough setup and in the end due to poor optimizations and other support issues – it probably wasn’t worth it.

                                                                                  I see too much industry support behind RISC-V at this point. V extension will be quite the spark, so we’ll see how it plays out after that. All it’ll take is one successful high performance commercial implementation.

                                                                                  Well – I think it actually takes a marketplace of commercial implementations so that selecting RISK-V isn’t single-vendor lockin forever, but I take your meaning.

                                                                          2. 3

                                                                            As I said up top, I hope this really happens. But I’m not super confident it’ll ever be something we can use to replace our AMD/Intel CPUs. If it just wipes out the current microcontroller and small CPU space that’s good too, since those companies don’t usually have good tooling anyway.

                                                                            I just think features-wise it’ll be hard to beat the current players.

                                                                            1. 1

                                                                              I just think features-wise it’ll be hard to beat the current players.

                                                                              Can you elaborate on this point?

                                                                              What are the features? Who are the current players?

                                                                              1. 4

                                                                                Current players are AMD64 and ARM64. Features lacking in RV64 include vector extension.

                                                                                1. 4

                                                                                  I notice you’re not the author of the parent post. Still,

                                                                                  Features lacking in RV64 include vector extension.

                                                                                  V extension is due to be active standard by September if all is well. This is practically like saying “tomorrow”, from a ISA timeline perspective. To put it into context, RISC-V was introduced in year 2010.

                                                                                  Bit manipulation (B) is also close to active standard, and also pretty important.

                                                                                  With these extensions out of the way, and software support where it is today, I see no features stopping low power, high performance implementations appearing and getting into smartphones and such.

                                                                                  AMD64 and ARM64.

                                                                                  The amd64 ISA is CISC legacy. Popular or not, it’s long overdue replacement.

                                                                                  ARM64 isn’t a thing. You might have meant aarch64 or armv8.

                                                                                  I’m particularly interested whether the parent meant ISAs or some company names regarding current players.

                                                                                  1. 4

                                                                                    ARM64 isn’t a thing. You might have meant aarch64 or armv8.

                                                                                    The naming is a disaster :/ armv8 doesn’t specifically mean 64-bit because there’s technically an armv8 aarch32, and aarch64/32 is just an awful name that most people don’t want to say out loud. So even ARM employees are okay with the unofficial “arm64” name.


                                                                                    Another player is IBM with OpenPOWER.. Relatively fringe compared to ARM64 (which the Bezos “Cloud” Empire is all-in on, yay) but hey, there is a supercomputer and some very expensive workstations for open source and privacy enthusiasts :) and all the businesses buying IBM’s machines that we don’t know much about. That’s much more than desktop/server-class RISC-V… and they made the POWER ISA royalty-free too now I think.

                                                                                    1. 4

                                                                                      SPARC is also completely open. Yes, POWER is open now, but I don’t see why it would fare better than SPARC.

                                                                                      1. 1

                                                                                        In terms of diversity of core designers and chip makers, maybe not. But POWER generally just as an ISA is doing much better. IBM clearly cares about making new powerful chips and is cultivating a community around open firmware.

                                                                                        Who cares about SPARC anymore? Seems like for Oracle it’s kind of a liability. And Fujitsu, probably the most serious SPARC company as of late, is on ARM now.

                                                                                      2. 3

                                                                                        The naming is a disaster :/ armv8 doesn’t specifically mean 64-bit because there’s technically an armv8 aarch32

                                                                                        Amusingly, the first version of the ARMv8 spec made both AArch32 and AArch64 optional. I implemented a complete 100% standards-compliant soft core based on that version of the spec. They later clarified it so that you had to implement at least one out of AArch32 and AArch64.

                                                                                    2. 1

                                                                                      Current players are AMD64 and ARM64

                                                                                      And ARM32/MIPS/AVR/SuperH/pick your favorite embedded ISA. The first real disruption brought by RISC-V will be in microcontrollers and in ASICs. With RISC-V you aren’t tied to one’s board/chip to a single company (Like ARM Holdings, MIPS Technologies, Renesas, etc.). If they go under/decide to get out of the uC market/slash their engineering budgets/start charging double you can always license from another vendor (or roll your own core). In addition, the tooling for RISC-V is getting good fairly fast and is mostly open source. You don’t have to use the vendor’s closed-source C compiler, or be locked into their RTOS.

                                                                                      1. 1

                                                                                        The first real disruption

                                                                                        Indeed. The second wave is going to start soon, triggered by stable status of the V and B extensions.

                                                                                        This will enable Qualcomm and friends to release smartphone-tier SoCs with RISC-V CPUs in them.

                                                                              2. 6

                                                                                Yes fab is expensive, but SiFive is a startup, it still managed to fab RISC-V chips that can run Linux desktop. I don’t think there is need to be too pessimistic.

                                                                                1. 4

                                                                                  The economics are quite interesting here. Fabs are quite cheap if you are a brand-new startup or a large established player. They give big discounts to small companies that have the potential to grow into large customers (because if you get them early then they end up at least weakly tied into a particular cell library and you have a big long-term revenue stream). They give good prices to big customers, because they amortise the setup costs over a large number of wafers. For companies in the middle, the prices are pretty bad. SiFive is currently getting the steeply discounted rate. It will be interesting to see what happens as they grow.

                                                                                2. 5

                                                                                  RISC-V is kinda useless without fabbing potential.

                                                                                  RISC-V foundation have no interest on fabbing themselves.

                                                                                  And that’s insanely expensive, which means the risk involved is too high to take on established players.

                                                                                  Several chips with CPU in them based on RISC-V have been fabricated. Some are already shipped as components in other products. Some of these chips are available on sale.

                                                                                  RISC-V’s got significant industry backing.

                                                                                  Refer to: https://riscv.org/membership/

                                                                                  1. 3

                                                                                    There’s a number of companies that provide design and fabbing services or at least help you realize that.

                                                                                    The model is similar to e.g. SOLR, where the core is an open source implementation, but enterprise services are actually provided by a number of companies.

                                                                                    1. 2

                                                                                      With ARM on the market, RISC-V has to be on a lot of people’s minds; specifically, those folks that are already licensing ARM’s ISA, and producing chips…

                                                                                  2. 3

                                                                                    Open source OSs can take on Microsoft with enough coders because it’s just software

                                                                                    Yet we haven’t seen that happening either. In general, creating a product that people love require a bit more than opensource software. It requires vision, deep understanding of humans and rock solid implementation. This usually means the cathedral approach that is exactly the opposite of FOSS approach.

                                                                                    1. 5

                                                                                      Maybe not for everyone on the market, but I’ve been using Linux exclusively for over 10 years now, and I’m not the only one. Also, for some purposes (smartphones, servers, SBCs, a few others) Linux is almost the only choice.

                                                                                      1. 3

                                                                                        You are absolutely in the minority though in terms of desktop computing. The vast majority of people can barely get their hand held through Mac OS, much less figuring out wtf is wrong with their graphics drivers or figuring out why XOrg has shit out on them for the 3rd time that week, or any number of problems that can (and do) crop up when using Linux on the desktop. Android, while technically Linux, doesn’t really count IMO because it’s almost entirely driven by the vision, money, and engineers of a single company that uses it as an ancillary to their products.

                                                                                        1. 6

                                                                                          That’s a bit of a stereotype - I haven’t edited an Xorg conf file in a very long time. It’s my daily driver so stability is a precondition. My grandma runs Ubuntu and it’s fine for what she needs.

                                                                                          1. 3

                                                                                            Not XOrg files anymore, maybe monitors.xml, maybe it’s xrandr, whatever. I personally just spent 4+ hours trying to get my monitor + graphics setup to behave properly with my laptop just last week. Once it works, it tends to keep working (though not always, it’s already broken on me once this week for seemingly no reason) unless you change monitor configuration or it forgets your config for some reason, but getting it to work in the first place is a massive headache depending on the hardware. Per-monitor DPI scaling is virtually impossible on XOrg, and Wayland is still a buggy mess with limited support. Things get considerably more complex with a HiDPI, multi-head setup, which are things that just work on Windows or Mac OS.

                                                                                            The graphics ecosystem for Linux is an absolute joke too. That being said, my own mother runs Ubuntu on a computer that I built and set up, it’s been relatively stable since I put in the time to get it working in the first place.

                                                                                      2. 2

                                                                                        Not on th desktop, for sure. Server side however, GNU Linux is a no brainer, the default choice.

                                                                                    1. 39

                                                                                      Wanna quickly figure out how you got to a particular point in a foreign codebase? Throw an exception there.

                                                                                      1. 14

                                                                                        I use import ipdb; ipdb.set_trace() in python almost pathologically. It’s 1 step up from an exception cause it drops you to a REPL, which is super convenient.

                                                                                        1. 11

                                                                                          With 3.7 we now have breakpoint() as a built-in.

                                                                                          1. 2

                                                                                            I’ve found the interactivity of the IPython debugger to be slightly better than the built-in one, but nearly all of my codebase is pinned to Python 3.5 for now.

                                                                                            1. 6

                                                                                              Use export PYTHONBREAKPOINT=ipdb.set_trace where ipdb is available and you don’t have to change your code for different environments. (at least once you’re on 3.7)

                                                                                          2. 5

                                                                                            I like to use IPython’s embed to achieve a similar goal.

                                                                                            Add in a call to embed() as a breakpoint and find yourself in a full-blown IPython session with your objects in scope.

                                                                                            1. 2

                                                                                              also, ptpython

                                                                                            2. 5

                                                                                              I use binding.pry the same way in Ruby. Requires the pry gem, but it’s well worth it.

                                                                                              1. 4

                                                                                                I’ve been using pudb in the same manner. It’s got a TUI and it’s simple to jump into the REPL from there.

                                                                                                1. 1

                                                                                                  Ooh, pudb looks really cool! Might switch to using that

                                                                                              2. 3

                                                                                                Yeah, unless there’s a try catch higher in the callstack consuming everything.

                                                                                                1. 3

                                                                                                  Or unless you are in a minified React component with no source map.

                                                                                                2. 2

                                                                                                  Long time before I learned the debugger; statement in JavaScript. Still seems unusual.

                                                                                                  1. 1

                                                                                                    I use this all the time but always feel a little dirty and imposterish. I’m glad my method has been validated by a stranger on an internet board!

                                                                                                    1. 1

                                                                                                      For big monolithic web apps: if finding the backend code for an operation is less obvious than it should be, find some method that will almost always be called (in our app it’s a particular method that gets the user context for a request, but it could be various things) and breakpoint it.

                                                                                                    1. 11

                                                                                                      Like the author I think having your own mail server isn’t worth it in most cases. But I do think it’s useful to have your own domain for email (and using an email service that supports custom domains). This way you’re never locked in to a particular email service because you can easily point your MX record to another one.

                                                                                                      1. 8

                                                                                                        Using your own domain has its risks too. If you miss a renewal payment, perhaps due to an errant email filter or an unusually long illness, you might lose control of it. Not only is it a major hassle—the new owner gains access to every account that can be reset by email without 2FA (i.e., most of them).

                                                                                                        That doesn’t necessarily mean using your own domain is a bad idea, but after many years doing that I’ve been slowly transferring some eggs out of that basket.

                                                                                                        1. 3

                                                                                                          I have crucial things (such as domains, but also water, electricity, …, phone service) set up with automatic direct withdrawal on a bank account that won’t run dry anytime soon. Not worth the hassle to check every invoice on those before the fact, especially since, given that they maintain crucial things, I took some care to choose providers I think I can trust with that as much as I can trust with them providing a reasonable service.

                                                                                                          1. 2

                                                                                                            That’s a good point. I have a yearly reminder in my calendar and have auto-renewal enabled, for me that’s good enough. I wonder how mail providers handle this actually. If someone stops paying for their account and it gets deleted, can someone else register using that same email?

                                                                                                            1. 2

                                                                                                              Most registrars will give you a reminder (or several) a few weeks before your domain expires. Assuming you keep up on that inbox (you should), it’s not too difficult. Many will park the domain for a time period after it expires too to prevent scalpers. Obviously if you’re super out of commission for a month, you probably have other things to worry about than your email.

                                                                                                          1. 37

                                                                                                            Having done sysadmin/SRE/whatever work for >10 years at this point, I’ve come to greatly appreciate paying for hosted services.

                                                                                                            I know that I’m perfectly capable of hosting my own email, as I’ve done it before. I’m also confident that I could host my own chat service, wiki, pastebin, k8s cluster, or whatever.

                                                                                                            But all of those things are, to one extent or other, a pain. I ran them because I had a business need to do it myself, and generally supported some number of users who were happy they didn’t have to do it themselves. I was also happy they weren’t doing it themselves, because they got to focus on other things that paid the bills, and I didn’t have to worry about them running into the sharp edges I had already cut myself on.

                                                                                                            I still self-host some things myself, even for personal use. But they’re either things that (a) there isn’t a convenient and trustworthy provider for, or (b) I personally find entertaining to operate. There are plenty of email providers out there, they aren’t that expensive, and email servers do not entertain me. So I pay for email. ;-)

                                                                                                            1. 7

                                                                                                              I’m a (significantly younger) software engineer, and I came to the same conclusion. I’ve hosted my own email. I’ve hosted my own chat, wiki, pastebin, etc. The only ones that I actually like hosting are Gitea and a private IRC server. As a software engineer, knowing how to set up and maintain an email server is really tangential to my normal job expectations, and the cost for paying a decent mail provider (say, Migadu) is probably cheaper if you factor in the time spent * (hourly rate of some kind) + headaches + hosting costs.

                                                                                                              That being said, if you enjoy hosting email, go for it. It’s just not worth it for me.

                                                                                                            1. 36

                                                                                                              On the interwebs the only articles I find are biased

                                                                                                              Everyone is biased. I don’t think it’s useful to seek an “objective opinion” - opinions are by their nature subjective.

                                                                                                              As someone who has only used Kubernetes professionally, my thoughts basically boil down to:

                                                                                                              • YAML is not a very good configuration format, and is the norm, leading to a lot of unnecessary friction
                                                                                                              • The system itself is very complex, even in its most simple incarnations, but it is also very powerful
                                                                                                              • If you need the kind of scale, reliability, and automation Kubernetes provides, you need Kubernetes or something else like it - at that point it’s down to preference and what integrates well with your stack
                                                                                                              1. 11

                                                                                                                YAML is not a very good configuration format, and is the norm, leading to a lot of unnecessary friction

                                                                                                                And aside from YAML as a configuration format, templated YAML (for example with Helm Charts, but many methods exist) is a really hard to deal with, and IMO a strong sign that this kind of declarative configuration is not a good fit for a tool of k8s’ complexity.

                                                                                                                “Lets do declarative configuration so you don’t need to program and it’s easier, but then let’s programmatically generate the declarative configuration” … hmm, okay…

                                                                                                                I’m not a huge fan of YAML as such, but I think the misapplication of a “one size fits all” declarative configuration format is probably the biggest issue.

                                                                                                                1. 2

                                                                                                                  Ehh, I think templating is fine in moderation. I don’t particularly like Helm, you can get 90% of the functionality with plain POSIX sh and heredocs. Having spent just shy of a year designing a company-wide OpenShift (RH K8s) deployment setup, I can say that Yaml is terrible format for configuration, templates are great in moderation, and jesus christ build your own images nightly. K8s is insanely complex, it’s effectively a distributed operating system, and so it has all the complexities of one plus the complexities of being distributed.

                                                                                                                  I also agree, YAML isn’t the worst in small doses, but when literally everything is a .yaml, it’s nothing but spaghetti garbage using strings/ids to tie together different disconnected things potentially across many different files.

                                                                                                                  1. 1

                                                                                                                    And aside from YAML as a configuration format, templated YAML (for example with Helm Charts, but many methods exist) is a really hard to deal with, and IMO a strong sign that this kind of declarative configuration is not a good fit for a tool of k8s’ complexity.

                                                                                                                    In my opinion templating yaml has been approached in the wrong way, or at least in an incomplete way.

                                                                                                                    Referring to the helm charts example: as yaml (which is json) are a textual representation of object, often times what would be better is not templating the text representation, but attaching a generated subtree to an otherwise leaf node.

                                                                                                                    Example: when including a yaml file into a chart template you have to “manually” call the indent macro to make sure the yaml parser is happy. But if we were reasoning in terms of trees of object, we could just say: this placeholder object here shall be replaced in its entirety with this other tree there (that you get by parsing that file) which will become a subtree of the initial tree.

                                                                                                                    Indentation would then be “automatic” because the next logical step would be to re-serialize the object and feed it to the kubernetes apiserver.

                                                                                                                  2. 4

                                                                                                                    I agree there is no escaping subjectivity, but would add: Not everyone is equally biased or honest with regard to any particular issue. Some opinions are well-informed, some aren’t. Some people’s advice is worth hearing on some particular topic, because they have overcome at least some of their biases through diligent, open-minded work, or because their biases are similar to one’s own (yes, that can be very useful - “I’d like to know if people who, like me, for some reason like X also would recommend Y”). Some people’s advice is irrelevant on a given topic. Getting advice from a range of people on here is likely to be more valuable than getting marketing disguised as technical blogging, for example.

                                                                                                                  1. 9

                                                                                                                    This is a great idea for a post that I’ve wanted to write myself. Leaving aside trees, hash tables, and stacks, I probably used less than one “interesting” data structure/algorithm PER YEAR of programming. Some notable counterexamples are two kinds of reservoir sampling, regular languages/automata, random number generation, some crypto, and some info retrieval algorithms.

                                                                                                                    One thing that sparked it is a obscure but long-running debate over whether dynamic programming interview questions are valid.

                                                                                                                    I don’t think they’re valid. It’s mostly a proxy for “I got a CS degree at a certain school” (which I did, I learned dynamic programming in my algorithms class and never used it again in ~20 years.)

                                                                                                                    I challenge anyone to name ANY piece of open source software that uses dynamic programming. Or to name an example in your own work – open source or not.

                                                                                                                    I’ve tried this in the past and nobody has been able to point to a concrete instance. I think the best I’ve heard is someone heard about a professor who heard about some proprietary software once that used it.


                                                                                                                    Related: algorithms used in real software (although this is certainly not representative, since compiler engineering is a subfield with its own body of knowledge):

                                                                                                                    https://old.reddit.com/r/ProgrammingLanguages/comments/b22tw6/papers_and_algorithms_in_llvms_source_code/

                                                                                                                    https://github.com/oilshell/blog-code/blob/master/grep-for-papers/llvm.txt

                                                                                                                    Linux kernel algorithms:

                                                                                                                    https://github.com/oilshell/blog-code/blob/master/grep-for-papers/linux.txt

                                                                                                                    1. 10

                                                                                                                      I challenge anyone to name ANY piece of open source software that uses dynamic programming.

                                                                                                                      Git, or most reasonable implementations of “diff”, will contain an implementation of the Myers Algorithm for longest-common-subsequence, which is very dynamic-programmy.

                                                                                                                      No concrete example for this one, but I know that bioinformatics code is full of dynamic programming algorithms for the task of sequence alignment, which is similar to diff — identifying a way to align two or more base sequences so that they coincide with the minimal number of changes/additions/deletions required to make them identical.

                                                                                                                      1. 1

                                                                                                                        Hm I’m familiar with that algorithm but I never thought of it as dynamic programming.

                                                                                                                        Wikipedia does say it’s an instance of dynamic programming. Although when I click on the paper, it appears to contrast itself with “the basic O(MN) dynamic programming algorithm” (section 4).

                                                                                                                      2. 8

                                                                                                                        Since you mentioned dynamic programming, it’s worth pointing out that the name “dynamic programming” was chosen for political reasons, as pointed out in the history section of the Wikipedia article on dynamic programming. So I think it’s a really bad name.

                                                                                                                        1. 1

                                                                                                                          That’s funny, I remembered xkcd’s “dynamic entropy” comic, and it quotes the same passage:

                                                                                                                          https://xkcd.com/2318/

                                                                                                                          It also has a very interesting property as an adjective, and that is it’s impossible to use the word dynamic in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning. It’s impossible.

                                                                                                                          LOL

                                                                                                                          Agree it’s a terrible name… I would say it was chosen for “marketing” reasons

                                                                                                                        2. 7

                                                                                                                          I have thought about whether dynamic programming questions are fair to ask, and I ended up where you are: they are not.

                                                                                                                          Dynamic programming was the one I struggled most in understanding and implementing correctly. And while there are semi-practical examples (like the backpack problem), I have not found any practical, day-to-day use cases on this.

                                                                                                                          I had an argument with my colleague who asked this kind of problem, saying it’s basic knowledge. Turns out he did competitive programming and there, it is table stakes. But in practice, it just filtered for anyone who has learned and practiced this approach.

                                                                                                                          I stay away from asking this, problems that need dynamic programming to solve.

                                                                                                                          1. 4

                                                                                                                            I’m familiar with dynamic programming mostly from high-school competitive programming as well. Otherwise I can’t say I’ve encountered real-life problems where it occurred to me to use the technique.

                                                                                                                          2. 8

                                                                                                                            I challenge anyone to name ANY piece of open source software that uses dynamic programming. Or to name an example in your own work – open source or not.

                                                                                                                            I’m literally about to implement something that could be classified as dynamic programming at work, which can be sumarrized as “computing a few simple statistics such as number of bytes changed for each directory in a large filesystem”. Dynamic programming is such a general term that it applies regularly if you want to use it.

                                                                                                                            1. 4

                                                                                                                              I’d like to see it. I don’t think dynamic programming is a general term.

                                                                                                                              In fact one time I participated in this argument (which was years ago so my memory is fuzzy), a former CS professor I worked with explained the difference between memoization and dynamic programming. A bunch of 10-20+ year programmers like myself went “ah OK”. Unfortunately I don’t even remember the difference, but the point is that most programmers don’t, because dynamic programming is very uncommon.

                                                                                                                              What you’re describing sounds like an obvious algorithm that anyone could implement, which is not the same as dynamic programming interview questions, or dynamic programming in competitive programing.

                                                                                                                              As other posters mentioned, competitive programming is the main place you see it outside of a CS class.

                                                                                                                              1. 2

                                                                                                                                It’s absolutely an obvious algorithm, so is most dynamic programming. That was sort of my point.

                                                                                                                                Can’t share the code unfortunately, but it’s just iterate over sorted list of file changes in reverse order and collect statistics as we go. Dynamic part comes from the fact that we can just look at the subdirectories of a dir (that we already have numbers for) instead of recursing into it.

                                                                                                                                1. 2

                                                                                                                                  What you’re talking about could be called memoization, or it probably doesn’t even deserve that name. It’s just what a “normal” programmer would come up with.

                                                                                                                                  That’s not the type of thing that’s asked in interview questions or competitive programming. The wikipedia page gives some examples.

                                                                                                                                  Dynamic programming usually changes the computational complexity of an algorithm in some non-obvious way. There’s very little you can do recursing over a directory tree that doesn’t have a clear O(n) way to code it (e.g. computing statistics).

                                                                                                                                  1. 7

                                                                                                                                    I like Erik Demaine’s explanation, that problems where dynamic programming can be applied are ones where their subproblems and their dependencies can be modeled as a directed acyclic graph [1]. Up to you if you’d like to tackle that with a top down approach where you look at a node and calculate its solution based on the solutions of its ancestors, or a bottom up approach starting from the nodes in the DAG with no dependencies and propagate the solutions in topological order.

                                                                                                                                    My colleague and I used it for a generalization of matrix chain multiplication (for tensors) [2].

                                                                                                                                    [1] https://youtu.be/OQ5jsbhAv_M?t=2736

                                                                                                                                    [2] https://github.com/TensorCon/ratcon/blob/release/opt/gencon/pathopt.py#L198

                                                                                                                                    edit: by the definition above, even naive memoization can count as DP, if you’re caching solutions to subproblems of that structure. Doesn’t have to be at the difficulty level of competition to count as DP in that case.

                                                                                                                                    1. 1

                                                                                                                                      Hm interesting, searching through my personal wiki, I found a related definition which is different. I haven’t really thought about it enough to form an opinion.

                                                                                                                                      Either way, it doesn’t change my overall point: that there are certain kinds of algorithms problems that appear on coding interviews, and in competititve programming, that do not show up in 99% of programming jobs. They are easy to pose and have cute solutions, but aren’t testing very much.

                                                                                                                                      I think the “DP tables” part is key but again I haven’t thought about it enough …

                                                                                                                                      https://blog.racket-lang.org/2012/08/dynamic-programming-versus-memoization.html

                                                                                                                                      Memoization is fundamentally a top-down computation and DP is fundamentally bottom-up. In memoization, we observe that a computational tree can actually be represented as a computational DAG

                                                                                                                                      In DP, we make the same observation, but construct the DAG from the bottom-up. That means we have to rewrite the computation to express the delta from each computational tree/DAG node to its parents. We also need a means for addressing/naming those parents (which we did not need in the top-down case, since this was implicit in the recursive call stack). This leads to inventions like DP tables, but people often fail to understand why they exist: it’s primarily as a naming mechanism (and while we’re at it, why not make it efficient to find a named element, ergo arrays and matrices).

                                                                                                                                      This bottom-up / top-down distinction might have been the same as what the aforementioned professor said 5+ years ago, but I don’t remember exactly.

                                                                                                                                      1. 1

                                                                                                                                        So, is memorization of factorial top-down, or bottom-up?

                                                                                                                                        1. 1

                                                                                                                                          I would probably say neither… Either factorial or Fibonacci is so trivial that it doesn’t help to be thinking about it that way.

                                                                                                                                          Though I think the quote hints at a clear test for whether it’s top-down or bottom-up: if you need extra state outside the call stack. I’m getting curious enough to try this out, but right now all I can do is quote what other people say.

                                                                                                                                          In any case it’s clear to me that there’s some controversy over what dynamic programming really is. I think the issue is that a lot of algorithms could be framed that way but were not discovered that way, and not taught and learned that way.

                                                                                                                                          1. 1

                                                                                                                                            I would probably say neither… Either factorial or Fibonacci is so trivial that it doesn’t help to be thinking about it that way.

                                                                                                                                            I think that the triviality is actually helpful here. If it’s actually true that memoization and dynamic programming are different (and there’s clearly debate on this), can 2 implementations of a trivial function, that everyone can understand highlight the differences?

                                                                                                                                    2. 1

                                                                                                                                      On the contrary the stupidly naive way (recurse on every directory) is O(n^2).

                                                                                                                                      Dynamic programming is just solving a series of problems while using the answers of shared subproblems multiple times. Memoization is a common way to implement this.

                                                                                                                                      Yes there are some very clever algorithms that use dynamic programming, this doesn’t make obvious algorithms that use dynamic programming not also fit under the definition.

                                                                                                                                      1. 3

                                                                                                                                        Why would recursing into every directory be O(n^2)? You’re still only visiting every directory/file once? It seems like something missing?

                                                                                                                                        1. 1

                                                                                                                                          Say you have a directory structure with a single file in it, /a/b/c/d/e

                                                                                                                                          To get the number of bytes changed in e you need to visit e, then to get the number of bytes changed in d you need to visit d and then e, then for c you need to visit c, d, and e, and so on.

                                                                                                                                          Like I said, it takes a really naive solution, but if you don’t remember the values you calculate anywhere for some reason it’s sum over inodes (depth of inode)… which is O(n^2) (for bad directory structures).

                                                                                                                                          Note that I need these values for every directory, not just for one particular directory.

                                                                                                                                          1. 2

                                                                                                                                            That’s still linear complexity space. Unless you’re hardlinking directories (which you then have to deal with potential recursion), it’s still O(n). If you add a file at /a/b/c/file you only visit 1 more file and no more dirs, not an exponential. O(n + n + n) or O(n + 3) still simplifies to O(n).

                                                                                                                                            1. 1

                                                                                                                                              If you add /a/b/c/file you add 4 more visits, not 1. Going from n= 3 /a/b/file to n=4 /a/b/c/file adds 4 more visits. In other words this worst case example takes time O(sum from 1 to n of i) = O(n(n-1)) = O(n^2).

                                                                                                                                              N is the number of inodes in a arbitrary tree not a number of files in a fixed tree.

                                                                                                                                              1. 1

                                                                                                                                                That’s still adding a linear number of operations for each file, the depth could technically be considered a different variable, say m. So for each file (n+1) you add, you also add the number of directory traversals (m) resulting in O(m+n), which simplifies again to O(n), but in reality folders are files too, so are part of n in the first place, so again O(n). Ultimately your n space is the total number of inodes, which both files and folders have.

                                                                                                                                                Abstractly, you’re just traversing a tree structure (or a directed graph if using links), which is well understood to be O(n) (maybe O(n^2) worst case if all folders are links, resulting in a fully connected graph), because you only visit each node once. If it were O(n^2), you would visit each node n times.

                                                                                                                                                Remember, Big O notation is about scaling, not the actual concrete number of operations, which is why you drop any constants or variables other than n.

                                                                                                                                                1. 1

                                                                                                                                                  It’s O(mn) not O(m+n) (in the insanely naive algorithm that recalculate things every time).

                                                                                                                                                  It’s not a single tree traversal but #internal nodes tree traversals.

                                                                                                                                                  1. 1

                                                                                                                                                    Even if it was O(mn) (it’s not), that still simplifies to O(n). An ‘internal nodes’ tree traversal is still O(n), n is just smaller, but again, your problem is not an internal nodes traversal, it’s a full traversal because you have to look at the blocks attached to the file (leaf) inodes, which means you need to read all inodes of all files and of all folders one time each. n = # of files + # of folders = O(n)

                                                                                                                                                    1. 1

                                                                                                                                                      I supposed an extremely naive solution could be to fully traverse each sub tree for every folder visited, which would be… O(log n)? But even that isn’t O(n^2), as the total repeated space shrinks the deeper you get.

                                                                                                                                                      1. 1

                                                                                                                                                        You’re assuming a balanced tree, which is not guaranteed. Depth of tree is O(n) in pathological cases (and average case O(sqrt(n)) is typical for randomly generated trees)

                                                                                                                                                        1. 1

                                                                                                                                                          Ah yeah, I think it would be O(n log n) not O(log n), because you traverse the tree once for each node, and a subset of of the tree for almost every n (except leafs), at least in the worst case. Still not O(n^2), and the solution for a O(n) is almost easier to conceptualize than the completely naive solution :)

                                                                                                                                                          1. 1

                                                                                                                                                            and the solution for a O(n) is almost easier to conceptualize than the completely naive solution :)

                                                                                                                                                            No argument here…

                                                                                                                                                            I think it would be O(n log n)

                                                                                                                                                            We agree it’s O(n) * O(time tree search) now right? And you’re trying to call the tree search time log(n)? Because trees are height log(n)? Then see the post you replied to, that’s true in a balanced tree, it’s not true in a random tree (where it is sqrt(n)) and it’s definitely not tree in a pathological worst case (where a tree is just a n length linked list).

                                                                                                                                                            1. 2

                                                                                                                                                              Yeah, the part I was hung up on before was that you’re naive solution traverses the entire subtree below a node for each node visit, I was stuck in the simple optimal solution. For the pathological case, basically just a bunch of folders in folders with a single file at the bottom, the depth of the tree is n, and the file inode at the bottom would be accessed n times, so O(n^2). For the common case it would be about O(n log n) where you can skip traversing larger and larger parts of the tree the deeper you get on each ‘path.’ Thanks for the discussion, I enjoyed it :)

                                                                                                                                    3. 1

                                                                                                                                      I think comparing memoization to dynamic programming is a category mistake: they are different kinds of things.

                                                                                                                                      ‘Dynamic programming’ is a description of a style of algorithm. It’s divide-and-conquer, usually with overlapping subproblems, making it possible to reuse intermediate results.

                                                                                                                                      Memoization is a programming technique to remember intermediate results, by remembering the results of function calls. You can e.g. also store the intermediate results somewhere explicitly, usually in a matrix, in which case you don’t memoize the result ‘transparently inside the function’, but use a lookup table ‘external to the function that computed the result’.

                                                                                                                                      1. 1

                                                                                                                                        I dunno I find that in addition to the Racket language resource I gave elsewhere in the thread, lots of people compare them:

                                                                                                                                        https://medium.com/@afshanf/cracking-the-coding-interview-questions-part-7-a7c8f834d63d

                                                                                                                                        A note on terminology: Some people call top-down dynamic programming “memoization” and only use “dynamic programming” to refer to bottom-up work. We do not make such a distinction here. We call both dynamic programming.

                                                                                                                                        There does seem to be disagreement on what dynamic programming is. And many algorithms that were not derived with dynamic programming techniques could be described as dynamic programming.

                                                                                                                                        But it seems that most people agree it’s related to memoization.

                                                                                                                                  2. 4

                                                                                                                                    GCC uses dynamic programming to split IA-64 instructions into bundles.

                                                                                                                                    1. 2

                                                                                                                                      Thanks, nice example and link! Still I would say it’s a niche skill, especially to come up with from scratch in an interview.

                                                                                                                                    2. 4

                                                                                                                                      I challenge anyone to name ANY piece of open source software that uses dynamic programming. Or to name an example in your own work – open source or not.

                                                                                                                                      Ever do video encoding or transcoding with anything built on FFmpeg or x264? Encode images with MozJPEG? Encode an AV1 video or AVIF image with libaom? Trellis quantization in advanced lossy compression encoders is a dynamic programming algorithm.

                                                                                                                                      1. 3

                                                                                                                                        Hm very interesting! I was not aware of that algorithm. Paper I found:

                                                                                                                                        https://www.mp3-tech.org/programmer/docs/e00_9.pdf

                                                                                                                                        I would still say it’s a bad interview topic, but it’s cool to see real world usages of it.

                                                                                                                                        1. 2

                                                                                                                                          Oh, no disagreement there! Even after coding it up myself, I’d hate to have someone ask me to whiteboard a working implementation of trellis quantization in 40 minutes or whatever (though I’m pretty sure I could sketch out an explanation now).

                                                                                                                                          In general I’m not a fan of whiteboard coding exercises at all. Whenever I’ve interviewed candidates I’ve always preferred the old-fashioned method of just reading their resume well ahead of time, looking up what ever piques my interest on it, and then having a friendly conversation about that. Usually that provides plenty of esoteric material for me to quiz them on and it also lets them show me their strengths and enthusiasm.

                                                                                                                                          1. 1

                                                                                                                                            My current company doesn’t do a whiteboard exercise, but my previous one did… but the thing is, the usual task was to implement a basic “grep”. That is, read a file and print all of the lines that contain a user-specified string, in a language of your choice, with whatever libraries make you happy (it’s not a trick, you’re not supposed to implement Boyer-Moore on your own). Assuming you succeeded at that, we would ask you to implement a few more features, like a -v flag (only print lines that don’t match), and -A and -B flags (print context lines before and after the matching line), until you got stuck or the time for that segment was up. It wasn’t graded on minutiae like correct semicolon placement, it was just an evaluation of whether a candidate could do a trivial task, how they handled additional requirements, whether they asked sensible questions and got clarification when needed, etc. I found it pretty reasonable.

                                                                                                                                      2. 4

                                                                                                                                        I challenge anyone to name ANY piece of open source software that uses dynamic programming. Or to name an example in your own work – open source or not.

                                                                                                                                        I used Warshall’s algorithm (which is dynamic programming) to compute the transitive closure of a graph for a typechecker. This is, in my experience, a very common algorithm.

                                                                                                                                        In high school, I wrote a program for my professor that places students into groups of 4 such that their meyers-briggs personalities are as different as possible. This used dynamic programming.

                                                                                                                                        A professor of mine (who taught the subject to me) used dynamic programming for some kind of RNA sequencing problem in a paper he published. One of our homework assignments had us arrive at a watered down version of his (and his co-authors’) algorithm.

                                                                                                                                        I’m fairly certain that at least some fuzzy string matching algorithms use string distance, which is also solved using dynamic programming.

                                                                                                                                        These are all diverse applications of DP. In my personal, subjective experience, the idea that DP is in any way obscure or dated is absurd.

                                                                                                                                        Edit:

                                                                                                                                        To be more concrete, the “transitive closure of a graph” is for the graph of dependencies, computing the set of all functions that a particular function depends on. This is as described in the Haskell Report.

                                                                                                                                        For fuzzy string matching, I have in mind something like fzf, though I cannot say with certainty that it uses string distance (I’m unfamiliar with its implementation).

                                                                                                                                        Here’s the paper that I think I’m referencing: Statistical Mechanics of Helix Bundles using a Dynamic Programming Approach

                                                                                                                                        1. 2

                                                                                                                                          Thanks for the examples. The claim is definitely not that it’s outdated or obscure; the claim is that it’s not a good interview question because it doesn’t show up much at work. Although there were lots of people here who pointed out interesting uses of dynamic programming, that’s not incompatible with the idea that you could have a 10 or 20 year programming career and never use it.

                                                                                                                                          Side note: I’m familiar with the Floyd Warshall algorithm but I never thought of it as dynamic programming. I think part of the issue is that I may have a more narrow definition of it than others. (I think people even say the linear time fibonacci is an example of dynamic programming, which I find silly. I just call that the obvious algorithm. I guess it can be used to illustrate a principle.)

                                                                                                                                          Even so, I definitely think it’s more popular in universities, and certain domains like bioinformatics. In contrast to what people on this site typically do “for work”.

                                                                                                                                        2. 3

                                                                                                                                          I challenge anyone to name ANY piece of open source software that uses dynamic programming. Or to name an example in your own work – open source or not.

                                                                                                                                          I do a lot of work with text processing – computing the edit distance between two strings is something I do very often. That’s a classic dynamic programming algorithm. There are probably hundreds of open source packages that do this or some variation thereof.

                                                                                                                                          1. 3

                                                                                                                                            Just to add to the list of responses clearly demonstrating Cunningham’s Law:

                                                                                                                                            I believe the Knuth-Plass line-breaking algorithm used in LaTeX to lay out text “optimally” uses dynamic programming. This was done for efficiency, as opposed to using some kind of general global optimization routine. It’s also the reason why LaTeX doesn’t support “backtracking”.

                                                                                                                                            1. 2

                                                                                                                                              It’s also the reason why LaTeX doesn’t support “backtracking”.

                                                                                                                                              Sile uses a variant of the same dynamic programming algorithm to lay out paragraphs on a page. The original paper describing the algorithm says that TeX wanted to use it like that, but it would require more than one entire megabyte of state for a large document, which was infeasible.

                                                                                                                                              1. 1

                                                                                                                                                Definitely an instance of Cunningham’s law at work :) I should make another go for my pet problems:

                                                                                                                                                • it’s impossible to make a zsh-like interactive interface on top of GNU readline
                                                                                                                                                • you can’t make a constant-space linear-time model of computation that’s more powerful than regular languages, and that can parse shell/JS/Python/C++
                                                                                                                                                • you can’t make an extended glob to regex translator in less than a week (https://github.com/oilshell/oil/issues/192)

                                                                                                                                                Thanks for the example. If there were more specific links I would make a blog post out of this :)

                                                                                                                                                And although my comment was a tangent, it did motivate me to get out the “Algorithm Design Manual” and flip to the part on dynamic programming. Though I remember the applications in that book being interesting but seemingly far removed from what programmers do day-to-day. It seemed to be by a professor who consulted on algorithms for various companies, which is an interesting job!

                                                                                                                                              2. 1

                                                                                                                                                The Grasshopper routing library uses contraction hierarchies, which are implemented using Dijkstra’s shortest path algorithm and A* search, which are special cases of dynamic programming.

                                                                                                                                                I have to agree it’s not something most people will use every day, but it never hurts to have a general idea how it works.

                                                                                                                                                1. 1

                                                                                                                                                  Here is a concrete example of Dynamic Programming that you use every day: Word Wrap. Knuth has an algorithm that is often used for maximizing the number of words per line.

                                                                                                                                                  Also the field of Bioinformatics often uses the Levenshtein distance when matching two dna strands.

                                                                                                                                                  Also I would like to mention the single most important thing I learned from Dynamic Progrmming: Start at the end case, and figure out what constraints can work from there. For example, think about the last recursion call, and what constraints it needs, and go backwards from there.

                                                                                                                                                1. 5

                                                                                                                                                  Do we have a lot of Lobsters on Windows? I keep debating doing a “Windows is Not Unix” guide for people who are moving to Windows + WSL2, but I keep convincing myself there’s no interest. The fact this is so high on the front page makes me wonder if I’m wrong.

                                                                                                                                                  1. 4

                                                                                                                                                    I use a Surface Book pretty regularly and run WSL2 on it. Although I also have a Linux desktop and a work-issued Macbook Pro.

                                                                                                                                                    1. 2

                                                                                                                                                      I use Windows at work.

                                                                                                                                                      1. 2

                                                                                                                                                        Ditto.

                                                                                                                                                      2. 2

                                                                                                                                                        I switched full-time to Windows about a year ago. I avoid WSL as much as possible, even going so far as to send PRs to open source projects that assume a Linux environment. I actually find it quite rewarding!

                                                                                                                                                        1. 2

                                                                                                                                                          I do that but for BSD instead of Windows ;)

                                                                                                                                                        2. 2

                                                                                                                                                          I’m interested. I use Linux as my primary desktop computing environment, but some things I do some of the time (e.g. building games for Unity) go better on Windows. And every time I try to adopt Windows for a few days, I’m a bit stymied. I feel like so many people use this, there must be something wrong with my workflow that’s making it feel so wrong to me.

                                                                                                                                                          This article is helpful and kind of sniffs around the edges. But I’d really be interested to learn more about what a really “Windows Native” workflow looks like and what people who use that all the time consider appealing about it. So if you post it, I hope it gets some attention here. Because even if I won’t ever be a “Windows person” I think I’m making myself suffer needlessly when I do use it, and I’d like to be educated about how to be more effective on it.

                                                                                                                                                          1. 1

                                                                                                                                                            For what it’s worth, I think your experience is very common (which is why ‘Windows’ gets followed immediately by ‘WSL’.)

                                                                                                                                                            As far as I can tell - and for no good reason - Linux users have a good command line and tend to use it, and Windows developers end up with a good IDE and tend to use it, and in both cases the thing not being used is neglected. So switching platforms is really about switching cultures and embracing a very different workflow. People going in both directions see the new platform as “inferior” because they’re judging the quality of tools used to implement their previous workflow.

                                                                                                                                                            As someone who really loves the command line, Windows has often felt like it required me to “suffer needlessly.” Personally I ended up writing my own command processor to incorporate a few things that the Linux command line environment “got right” and it ended up being extended with many interactive things. I haven’t felt like I’m suffering or missing out by using Windows after that, although clearly there’s a personal bias since those tools are designed for my workflow. If you don’t like mine, then it pays to learn Powershell, which is also a big cultural adjustment since the basic idioms are very different to textual Posix shells.

                                                                                                                                                            To me the most “appealing” part of being a Windows developer is WinDbg, which also has a steep learning curve but it’s far better than anything I’ve used on Linux and is much more capable than Visual Studio for systems level debugging. It’s one of those “best kept secret” type things that’s easy to get - once you know it exists and that it makes serious development much easier.

                                                                                                                                                          2. 1

                                                                                                                                                            I use Windows at work (though work is Microsoft Research, so that’s not very surprising). I use WSL1 and a FreeBSD VM. I don’t think WSL2 yet has the nice PTY and pipe integration in WSL1. With WSL, I can run cmd.exe in a *NIX shell (including something like Konsole in VcXsrv) and it works. More usefully, I can run the Visual Studio Command Prompt batch file to get a Windows build environment from my WSL environment. If a Windows program creates a named pipe in the WSL-owned part of the filesystem namespace, it is a Linux named pipe.

                                                                                                                                                            1. 1

                                                                                                                                                              I reluctantly use Windows, but I try to avoid it. There’s rarely anything I want to use that is only available on Windows with no viable alternative. Still, there are times I have to use it. I haven’t bothered with WSL2 but I would still read something like this if it existed.

                                                                                                                                                              1. 1

                                                                                                                                                                My work/play machine at home is Windows. Always has been. (Because games, obviously, and I don’t like dual boot.)

                                                                                                                                                                I detest working on Windows though and WSL hasn’t improved anything for me yet, that’s why I still do most of my work via PuTTY on a linux box if possible. (work means any ops/programming work I do in my free time).

                                                                                                                                                                Work machine is Linux and I’m actually glad I couldn’t work on Windows so no one can push me :P

                                                                                                                                                                1. 1

                                                                                                                                                                  I’ve recently committed Linux apostasy :) partly on account of WSL2, and I’d definitely be interested in that. I haven’t used Windows, except to run putty to log in to a remote server, in a very, very long time (in fact, save for a brief period between 2011 and 2012, when I worked on a cross-platform tool, I haven’t really used it in 15+ years, the last version I used regularly was Windows 2000). I’m slowly discovering what’s changed since then but 15 years is a lot of time…

                                                                                                                                                                  1. 1

                                                                                                                                                                    I’m on Windows and routinely browse lobsters by looking at the ‘windows’ tag.

                                                                                                                                                                    That said, I really don’t understand the point of moving to Windows to then run WSL2. Using WSL2 is using a Linux VM, and for the most part, the issues there are not about Windows. If you want a Linux VM, you can run one on any platform…

                                                                                                                                                                    1. 4

                                                                                                                                                                      Well, it’s a damn convenient VM :). I’ve tried it as an experiment for a month or so – I’m now on an “extended trial” of sorts, I guess, for 6 months (backups over the network, all Linux-readable – if I ever want to go back I just have to install Linux again). I use it for two reasons:

                                                                                                                                                                      • It’s a good crutch – I’m sure everything I could do under Linux can be just as easily be done under Windows but 20 years of muscle memory don’t get erased overnight. It lets me do things I haven’t had time to figure out how to efficiently do under Windows in a manner that I’m familiar with. I could do all that with a VM, too, but it’s way better when it works out of the box and boots basically instantaneously.
                                                                                                                                                                      • It gets me all the good parts of Linux – from mutt to a bunch of network/security tools and from ksh to my bag of scripts – without any of the bad parts. Of course, that can also be done with a VM – but like I said, this one works out of the box and boots instantaneously :).

                                                                                                                                                                      I don’t use it too much – in the last two weeks I don’t think it’s seen six hours of use. I fire it up to read some mailing lists (haven’t yet found anything that handles those as well as mutt – even Pegasus Mail isn’t as cool as I remember it…), and I fiddle with one project inside it, mostly because it’s all new to me, and I want to work on it in a familiar environment (learning two things at a time never goes well).

                                                                                                                                                                      I still haven’t made up my mind on keeping it, we’ll see about that in six months, but I’m pretty sure I’d have never even considered the experiment without WSL2.

                                                                                                                                                                    2. 1

                                                                                                                                                                      My monitor + video card setup is aimed at gaming and is almost entirely unusable on Linux period. I’m looking at replacing the graphics card in the future, but until that happens, Windows is the only tolerable OS.

                                                                                                                                                                      1. 1

                                                                                                                                                                        I like to play video games, which almost always means “you need windows”.

                                                                                                                                                                        I used to dual boot Debian but nowadays I tend to just ssh into a cloudish instance thing that I use as a kind of remote workstation. That combined with vscode’s wonderful remote support means outside of work most of my personal-hacking-stuff still takes place through the window of… well, windows.

                                                                                                                                                                      1. 3

                                                                                                                                                                        I recently purchased an @.dev domain for email and started referring to my blog on my CV (I mainly talk about math). I don’t know if it’s coincidence, but I landed a high-profile job shortly after that. I suspect it might have helped, not because my blog is especially high quality, but just because it shows that you like to spend time investigating.

                                                                                                                                                                        1. 4

                                                                                                                                                                          that’s really cool and I would love to read your blog

                                                                                                                                                                          1. 2

                                                                                                                                                                            Ooh, please link your blog!

                                                                                                                                                                            1. 1

                                                                                                                                                                              It’s rubenvannieuwpoort.nl (the site is hosted on rubenvannieuwpoort.github.io) :) The “I mainly talk about math” remark was directed at my blog, not at my CV. BTW I am currently rewriting the article about Cardano’s formula (I think it’s actually called “Cardano’s method”) and I have a lot of ideas for articles that I haven’t found the time for to write.

                                                                                                                                                                          1. 24

                                                                                                                                                                            Don’t. They’re loud, heavy, inconvenient, and expensive.

                                                                                                                                                                            1. 3

                                                                                                                                                                              To add to this: unless you specifically want to learn about enterprise technologies (RAID, SFPs, etc.), go with consumer hardware. You’ll save money on parts, electricity, mounting, etc. and won’t have to deal with the noise as much. NUCs are great if you want something small and powerful(-ish), or Mini-ITX, mATX for small form-factor custom builds. The consumer space has a lot more flexibility than enterprise hardware, which lets you fine-tune your build for whatever usecase you have.

                                                                                                                                                                              1. 1

                                                                                                                                                                                I second the NUC comment. I have two set up at home and they’re awesome.

                                                                                                                                                                              2. 2

                                                                                                                                                                                Agreed.

                                                                                                                                                                                I had a rack-mount 2U Compaq DL380 with all the bells and whistles that I got for free during the dot-com bust. It was a pain to live with:

                                                                                                                                                                                • Weird boot process that only worked with Windows and Red Hat Linux
                                                                                                                                                                                • So many fans that even though it was in the basement I could hear it on the ground floor
                                                                                                                                                                                • It cost $10/mo in electricity, even after I removed one of the two power supplies
                                                                                                                                                                                • Not that fast, except for the disks which were 15k RPM

                                                                                                                                                                                Do the planet and yourself a favor and go with consumer-grade hardware.

                                                                                                                                                                                1. 1

                                                                                                                                                                                  Do the planet and yourself a favor and go with consumer-grade hardware.

                                                                                                                                                                                  I’m not an environmentalist, but the prospective impact to my power bill has me concerned. $10 / month to run a fast machine would be okay, though. I’ll have to do some more research into TCO I think.

                                                                                                                                                                                2. 2

                                                                                                                                                                                  They also eat power. See vermaden’s posts (e.g. https://vermaden.wordpress.com/2019/04/03/silent-fanless-freebsd-server-redundant-backup/ ) on running a simple server if you really need something.

                                                                                                                                                                                  (bias: I’ve done the freenas thing and paid into the myths of server hardware/ecc ram/etc. While it’s come in handy at times (ipmi is convenient) it’s also been a time burden when it comes to doing maintenance or trying to do any slight customization in setup from what freenas wants. If your needs are largely just storage, consider just leveraging your fast internet connection and paying a cloud provider. Remember even with a server-grade NAS you’re arguably still going to be paying someone for off-site backup.)

                                                                                                                                                                                1. 2

                                                                                                                                                                                  Much better now that I’ve been going into the office. I moved from an apartment to a duplex last weekend and am still recovering from the processes (bruises everywhere, sore limbs, etc.). Work is heating up a bit too, we have a milestone that we’re pushing for at the end of the month, I haven’t gotten a chance to rest much for awhile now.

                                                                                                                                                                                  1. 2

                                                                                                                                                                                    Do any commercial VPN providers offer WireGuard support yet? I remember reading that ExpressVPN was working on it, but haven’t heard any news in a while.

                                                                                                                                                                                    1. 17

                                                                                                                                                                                      mullvad.net has had it for quite some time, not sure about any others (as in I’ve not tried to check it with others)

                                                                                                                                                                                      1. 3

                                                                                                                                                                                        I also use mullvad. WireGuard works great especially as VPN on my iPhone.

                                                                                                                                                                                        1. 1

                                                                                                                                                                                          I’ve been using Mullvad for several months not, it’s wonderful

                                                                                                                                                                                        2. 7
                                                                                                                                                                                          1. 3

                                                                                                                                                                                            NordVPN

                                                                                                                                                                                            1. 0

                                                                                                                                                                                              Though I would want to use a provider who is known for hiding their ownership information and their leadership behind some Panamanian shell company.

                                                                                                                                                                                            2. 1

                                                                                                                                                                                              ExpressVPN was working on it

                                                                                                                                                                                              They built their own protocol on DTLS.

                                                                                                                                                                                            1. 14

                                                                                                                                                                                              Destroying the notion of “fighting the compiler” was one of my biggest steps forward. I started to view the type checker as a pair programming buddy. One that doesn’t get tired, only states facts based on information it has, and will always update its assumptions based on new information, unlike most of us. ;)

                                                                                                                                                                                              So basically “type systems”, but only after a necessary perspective shift. Well, that and being exposed to what modern type systems can actually do. Instead of the limited assumption that they only catch errors like: “That’s not an Int!”.

                                                                                                                                                                                              1. 2

                                                                                                                                                                                                As someone building extremely complex, scientific systems in Pythonthis. Seriously, type systems are so undervalued.

                                                                                                                                                                                                1. 2

                                                                                                                                                                                                  Python 3.x doesn’t have type hints/annotations?

                                                                                                                                                                                                  1. 1

                                                                                                                                                                                                    It does! But it’s not a full replacement for a static/strong typing system, but it does help. We’re also pinned to an older version of Python (3.5) which doesn’t have full support for type hinting.

                                                                                                                                                                                                    1. 2

                                                                                                                                                                                                      I’m a Java developer working with Python to automate some things. It seems like when the program is more than a few lines of code (1K) and needs to be maintained by a team, the desire for type systems grows strong.

                                                                                                                                                                                                      1. 2

                                                                                                                                                                                                        I’ve been porting a large Python 2.x code-base to Python 3. Adding type annotations (https://mypy.readthedocs.io/en/stable/cheat_sheet.html) has been hugely helpful. The type system is quite expressive and mypy’s reporting is pretty good. After spending some time with Rust – which i really like – I appreciate the flexibility of Python’s approach.

                                                                                                                                                                                                1. 2

                                                                                                                                                                                                  Check out the original social network with finger kirch@tilde.town and if your server is running the fingerd daemon, try creating .forward, .plan, .project, and .pubkey or .pgpkey files in your home directory (check man finger) and see what happens when you finger yourself.

                                                                                                                                                                                                  1. 1

                                                                                                                                                                                                    see what happens when you finger yourself

                                                                                                                                                                                                    I’m going to wait to get home from work before I do this