1.  

    Someone correct me if I’m wrong but it seems that in the absence of a hardware RNG, getrandom() can never be expected to work well. Collecting randomness based on interrupts is never guaranteed to unblock in any sort of reasonable time also it’s just a software heuristic that even then may not meet the application’s criteria for randomness. Soap box here, but IMO ideally getrandom() should return EINVAL simply in the absence of a reliable hardware RNG and user-space can then take that as a signal to employ their own heuristics.

    1.  

      I believe you can employ two tricks to achieve sufficient entropy in the situation where you do not have a hardware RNG:

      1. You can save the entropy pool at shutdown and restore it at boot.
      2. For the first boot of a device, I think you can pre-seed it with entropy from somewhere else.
      1.  

        I think in a case where user-space seeded the kernel with randomness like that, I would support that also being a situation where getrandom() didn’t return EINVAL. At the same time, it may not even be necessary for the kernel to be involved unless it needed randomness itself (which I guess it often does).

    1.  

      Anyone remember that post Linus made about getting help for his anger issues and focusing hard on being compassionate towards other maintainers? Looks like he’s really taken that to heart. Seems like his old tone is basically gone here - very refreshing.

      1.  

        TBH, Linus’s release emails always had this sort of tone even before he vowed to get help on his anger issues.

      1. 2

        Too bad the modulus operation via truncation isn’t faster than the native divide instruction.

        1. 2

          I think the thing that’s mentioned up the top but missing from the discussion further down is that these operations are likely to be a lot faster on architectures which don’t provide good native divide instructions … ARM for example.

          1. 1

            The way it’s mentioned at the top is a bit misleading but I see what you mean. Makes a lot more sense for resource-constrained hardware.

          2. 1

            The problem is that there is only one divisor per code (at least in x86-64) and it is not fully pipelined, so instead the compiler does the magic for you. It also uses slightly different way of doing that than the article proposes.

            1. 1

              I think you mean “per core.” But I see, the compiler uses integer multiply + some fixups to generate the remainder, not the divide instruction. I remember looking this up once, but I wonder if there is a primer on that method.

              1. 1

                Yes

          1. 1

            Would be cool if you also got it to absorb power from the atmosphere somehow, kind of like those old desk calculators.

            1. 2

              Or the Atmos clock.

              1. 1

                What old desk calculators? How were they powered? I guess you aren’t talking about solar-power.

              1. 2

                Interesting concept but pragmatically, I can’t use it if it’s not POSIX compliant

                1. 4

                  FWIW Oil is POSIX compliant (and it doesn’t feel too far from exceeding the POSIX compliance of bash with recent help by the Smoosh project )

                  I have designs for structured data, but they aren’t implemented.

                  https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil

                  Oil is supposed to do most of what I see so far in nushell (probably minus the cd into a data structure), but I probably bit off more than I can chew. Help is appreciated :)

                  I did just start implementing the Oil language though, and it’s not too far from dictionaries / vectors to data frames.

                  1. 1

                    Oil looks interesting. I would use any alternative shell that adds expressiveness as long as it were POSIX compliant. I never use bashisms except for the auto-complete scripts that come with my OS.

                  2. 2

                    I’m genuinely curious, why?

                    1. 2

                      For me it’s not a matter of using bash OR powershell / bash OR nushell. It’s a matter of using “just bash” or “bash + powershell”. They augment bash rather than replace it.

                      It’s hard to switch because there’s a lot of other stuff that depends on bash.

                      For example

                      • git has thousands of lines of bash completion logic
                      • there are thousands of lines completion logic for coreutils, etc.
                      • Python’s virtualenv has a bunch of bash
                      • nix-env is bash
                      • asdf is bash
                      • opam switch is bash (OCaml’s package manager)

                      I started the Oil shell with the idea of bash compatibility, but even I underestimated how much bash code there is out there!


                      Basically I want to use one shell and not two. Although honestly I can see why that might be too idealistic… plenty of people use two shells – zsh/fish interactively and bash for scripting / system shell seems to be a very common combo. But still I’m going to pursue replacing bash and then augmenting it :)

                      1. 1

                        Well if it’s not my login shell (selected via chsh), then I guess it wouldn’t really matter. It would be the equivalent of dropping into a Python shell. To act as a login shell, it would need to be POSIX compatible because everything in a POSIX ecosystem expects that to be the case. It also would be tiresome to have to disregard all my previous knowledge to begin to even start using the shell.

                        If it’s not my login shell, I probably wouldn’t use it much and would opt for more general purpose languages like Python for the situations where I need a more expressive language.

                      2. 1

                        You can’t use Software if it’s POSIX compliant?

                        1. 1

                          If it’s not POSIX compliant.

                          1. 1

                            Oops, you’re right.

                      1. 19

                        Their reasoning makes sense. They lack the manpower to provide a quality big-endian MIPS port. I used to run a big endian PPC on gentoo and I often had to write and maintain custom patches to mainstream packages that were broken. It’s a lot of work. I eventually sold my machine and got x86.

                        They still provide a little endian port that covers the pareto majority of MIPS hardware. I don’t see what the big issue is. IMO Debian was never “universal” in the sense that it ran on all hardware because it never did, it isn’t NetBSD. It’s universal in the sense that it’s free software and available for everyone.

                        1. 2

                          it’s free software and available for everyone

                          ..for everyone to modify :)

                          But yeah, in the end it’s all about man power. I wonder if the sheer size of debian hurts its long term survivability vs something small like netbsd or openbsd?

                          1. 2

                            I wonder if the sheer size of debian hurts its long term survivability vs something small like netbsd or openbsd?

                            In what ways do you see Debian’s size hurting it’s survivability in a way to which NetBSD and OpenBSD are immune? I can see an analogue in large dinosaurs vs small mammals but can’t exactly imagine it here.

                            1. 2

                              First we need to define what makes it “large” (if it even is). To me large is supporting / maintaining lots of packages, many workflows for particular system administration (ifconfig vs ip), difficult to setup to develop, etc.

                              We know popularity contributes significantly to an OS’s survival, but look at 9front, netbsd, openbsd and haiku: they still live because they can be maintained by such a small group of people - and to me that’s powerful.

                              At this point in time I believe NetBSD has the best chance of being around in 100 years. It has adapted to run on top of other OSs.

                        1. 1

                          Really interesting project. I would jump from Debian if/when this project is ready for daily use. The ability to install multiple versions of a package is crucial for experimentation and local package development. Atomic updates are another feature that would give me less anxiety before upgrading.

                          One question, how is this different/better than NixOS? Answered elsewhere, thanks!

                          1. -7

                            It’s an awesome project; my personal blocker is the fact that it’s written in Go. Otherwise I wouldn’t care whether it’s ready for daily use.

                            1. 1

                              What would you prefer over Go?

                              1. -1

                                Probably Rust, given the use-case.

                                1. 2

                                  Why is Go a blocker? It’s not my favorite language, but it is a safe language, produces static binaries, and is fast enough.

                                  1. -5

                                    A mixture of the code that gets written in it, the community around it and the anti-intellectualism that seeps through it.

                                    1. 3

                                      I would classify it as a conservative language, not an anti-intellectual language.

                                      1. -5

                                        It’s an anti-intellectual one.

                                        1. 0

                                          This thread is a perfect example of people downvoting because they dislike something someone said. Shameful.

                                  2. 2

                                    Rust has a terrible bootstrap from source story if you care about minimizing binary blobs you have to download from the internet.

                                    Package managers care a lot about that sort of thing, and Go has a pretty good bootstrap story.

                                    1. 1

                                      Go is a slight turn off to me as well, but that objectively probably doesn’t mean much since I would prefer if it were written in C. Mostly for consistency with the rest of Linux infrastructure.

                                      1. 3

                                        I disagree — while we have to retain C for backwards compatibility (probably for decades to come, maybe even a century), it’s well past time to use a language with fewer footguns. I’d prefer Lisp (of course), you prefer Rust, but honestly Go is a decent language in which one can get work done without excessive boilerplate and pain.

                                        IMHO Go is probably the ideal language for tooling like this in the real world (as opposed to my ideal world, in which everything is a Lisp machine).

                                        1. 1

                                          With what do you disagree? That’s my preference, I’ve made no claim. Write your software in Go or Lisp or whatever you want, I just won’t use it.

                              1. 6

                                This is quite similar in mechanisms used by Nix - immutable paths, path patching, storing at well known places via symlinks. Where would be the main differences?

                                1. 6

                                  There is indeed some overlap with Nix et al. (separate hierarchies key idea, and using a package store).

                                  I have used NixOS for about half a year, and the user experience felt distinctly different than what was possible to achieve in distri (which I have been using for a few months on my laptop now).

                                  Notable differences are package installation speed, how packages are built (declaratively specified, not in a functional language), and how well it works with third-party software.

                                  Does that answer your question?

                                  1. 2

                                    Nice, thanks for such a prompt answer! The installation speed sounds pretty interesting! Did you do any scalability measurements, how many images could be supported under /ro? (Just curious if the approach could help nix, which has a lot under /nix/store)

                                    1. 3

                                      I currently have 677 packages in my store on my laptop. There are only 425 different packages in distri, so there are some duplicates which I have not yet deleted.

                                      The only slow-down I could find thus far is when the exchange directories are traversed when booting. This could be done lazily and/or cached, I just haven’t gotten around to it. I wanted to do the release first, and implementation details can always be polished later :)

                                    2. 2

                                      Just to make sure I’m understanding you correctly, are you saying that Distri is sort of an unopinionated and more compromising version of NixOS?

                                      Also just curious, aren’t you trading off package installation efficiency for efficiency during operation of the package? I have to imagine that it’s more expensive to access data through a FUSE program parsing squashfs images at runtime, than natively through the file system.

                                      1. 2

                                        Absolutely, that’s a good observation!

                                        To productionize this idea, it would probably make sense to implement this file system as a kernel module, not via FUSE.

                                        That said, in my day-to-day, programs are typically loaded into memory once, and I don’t notice that being slower than on the non-distri computers I use. So I don’t feel any pain to rush from user space to kernel space :)

                                        1. 3

                                          While that is a component of my premise, user/kernel space slow down isn’t primarily to what I was referring. From a high-level perspective, it would seem to me that traversing squashfs images isn’t as efficient as, for example, the ext4 on-disk format.

                                          If you haven’t noticed a slowdown maybe it’s not a cause for concern, though I could see FS competitive performance becoming a bigger issue that may block adoption when cold-booting larger programs like Firefox / LibreOffice, etc.

                                          1. 2

                                            it would seem to me that traversing squashfs images isn’t as efficient as, for example, the ext4 on-disk format.

                                            Why is that? I haven’t looked into ext4’s on-disk format as much as I have into SquashFS’s, but it seems pretty efficient to me.

                                            If you haven’t noticed a slowdown maybe it’s not a cause for concern, though I could see FS competitive performance becoming a bigger issue that may block adoption when cold-booting larger programs like Firefox / LibreOffice, etc.

                                            I’m using google-chrome regularly. Reading the few files it contains is not a big deal, even through my FUSE file system :)

                                            1. 2

                                              it would seem to me that traversing squashfs images isn’t as efficient as, for example, the ext4 on-disk format.

                                              Why is that? I haven’t looked into ext4’s on-disk format as much as I have into SquashFS’s, but it seems pretty efficient to me.

                                              You would know more, but I see two situations:

                                              1. squashfs on-disk format is meant to be mapped directly into memory, and traversed directly. In that case, I would imagine that there is a tradeoff between the size of the squashfs image file, and the efficiency of traversing it once it is in-memory. E.g. redundancy of data usually allows faster in-memory data structures but not good for image file size.

                                              2. squashfs on-disk format requires some pre-processing before it can be loaded into memory and traversed. This is different from ext4, which basically requires no pre-processing before being traversed.

                                              According to those two situations, squashfs is either not efficient disk-space wise, or not efficient performance-wise (relative to e.g. ext4). Of course, it’s possible that squashfs is not as efficient as ext4, but close enough that it doesn’t matter. As long as cold-boot performance is comparable (e.g. not more than 1-5% slower), it probably will never bubble up as an issue.

                                              1. 2

                                                SquashFS is actually pretty flexible. By default, it optimizes for size, but my implementation optimizes for easy access. There are still some wins like producing SquashFS’s directory index data structures.

                                                1. 2

                                                  Gotcha, thanks for entertaining my reasoning :)

                                                  Just curious, any reason you didn’t choose to use a combination of overlayfs and squashfs, e.g. mounting squashfs images natively, and using overlayfs to create the exchange directories?

                                                  1. 6

                                                    I actually implemented it like that before implementing my FUSE file system! Turns out that adding new kernel mounts gets really slow really quickly. One of my kernel developer friends told me that mounts are linked lists that degrade pretty quickly.

                                                    Setting up all the mountpoints required to build moderately sized programs took many seconds. With the FUSE file system that can lazily mount these images (and mounts them more quickly), this setup is now done in the fraction of a second, which is a massive developer experience improvement :)

                                                    Additionally, overlayfs were pretty complicated to correctly manage programmatically, especially changing the composition of overlays, like when installing a new package.

                                  1. 5

                                    If, at the end of your life, you end up only wasting a single year of your life, that would be pretty good.

                                    If you can code, there is a very good chance there is someone out there ready to hire you for that, regardless of your past experience.

                                    1. 1

                                      This would be great for efficiency and consistency, but on the other hand I can see this stifling application innovation. Framebuffers allow application developers the flexibility to do whatever they want.

                                      1. 1

                                        Output

                                        1. 4

                                          I have an idea for a blog post: How to Rewrite Oil in C++, Rust, or D. If you’re interested in that, leave a comment.

                                          that sounds extremely interesting! a python/c -> d conversion in particular would be fantastic because d seems like it would naturally accommodate straightforward translation of code from both languages, and possibly even incremental migration.

                                          1. 5

                                            Yeah I found D interesting because it has garbage collected data structures, unlike C++ or Rust. That makes it the most natural target of translation for a typed Python codebase, which doesn’t have any notion of deallocation.

                                            (Although I’m personally working on C++, because I’m familiar with it, build dependencies matter for a shell, and because shell depends a lot on libc, and libc bindings are trivial in C++.)

                                            But the basic idea is that I wanted to encourage other people to try to rewrite Oil in a different language, in case I fail :)

                                            I believe in the power of parallel development in open source. Since there are at least 5 major, independent JS engines, there could probably be 1 or 2 more shell implementations.

                                            You would get a “leg up” in the following sense:

                                            • The Oil lexer is specified with regexes, which can be ported to any language easily. It does a huge amount of work, so you don’t have to “grovel through backslashes and braces one-by-one”, as I like to characterize the implementation strategy of other shells.
                                            • The Oil ADSL schemas, e.g. syntax.asdl. They are also code generators that are easy to port, and represent a huge amount of “logic”.
                                            • The parsing algorithms documented on the blog

                                            So it would be significantly less work than writing a shell from scratch.

                                            I have a few blog posts in front of that, but if people are interested, let me know and that will motivate me to write it.


                                            Random: I’ve also had the idle thought that since Fabrice Bellard just released a huge JS interpreter written from scratch, and has done a C compiler, maybe he will want do also do a shell :) The goal of OSH was really to “discover a spec” and I think that has already succeeded.

                                            That said, I’m also going to try to do it myself. But it is a huge amount of work, and as mentioned I’m also working on the Oil language, which is significantly larger than OSH. And everything needs documentation, etc.

                                            1. 5

                                              Do consider Nim as well. Probably a better alternative to D in this case, since it also has Pythonic syntax, which might aid in the translation process.

                                              1. 3

                                                I honestly don’t think that matters very much. Aside from whitespace for indentation, the syntax isn’t especially more pythonic, and switching out indentation for braces is far from the most interesting piece of the puzzle.

                                                Of course, I might be a bit biased, since I prefer d to nim :P

                                                1. 3

                                                  I will try to make the title more language-agnostic. The idea is to encourage with people with expertise in a particular language to think about how they would do it, and maybe try it. The content of the post itself is mostly language agnostic.

                                                2. 2

                                                  I’m definitely interested. I enjoy rewriting code, both refactoring and porting it to different languages, and it would be neat to join in an effort to do that for oil.

                                                  I probably can’t do it singlehanded due to the sheer amount of work involved, but it would be a fun project to chip away at, and I bet a blog post with a roadmap of sorts would spark tons of interest.

                                                  1. 3

                                                    OK awesome! Yeah I enjoy rewriting code too, especially when there are tests, which Oil has a lot of. I liked adding all the type annotations, which feels like a form of refactoring. Only about half of it is type annotated, and I’m looking forward to annotating the rest.

                                                    I will bump that up on the list… First I have to publish the draft of The Interactive Shell needs a Principled Parser, and then I’ll try to do this one.

                                                    It’s partly “defensive” in that I get a lot of uninformed “Why Don’t you rewrite it in X?” comments :) But it’s also 100% serious – I do believe that the diversity of ideas in open source moves things forward faster. That includes diversity in languages.

                                                  2. 1

                                                    Since Oil is already type annotated, why not just use mypyc to compile to native code?

                                                    https://github.com/mypyc/mypyc

                                                    1. 2

                                                      Yeah good question. Oil isn’t fully annotated yet (only about half of it is), but I may try it.

                                                      The main issue is that I want to completely remove the dependence on the Python interpreter, while mypyc generates Python extensions. So after mypyc you still need a Python interpreter.

                                                      In March I visited Guido van Rossum at Dropbox, and we talked about MyPy and mypyc (and Oil). This “mycpp” experiment is heavily inspired by mypyc and Shed Skin:

                                                      https://github.com/oilshell/oil/tree/master/mycpp

                                                      I have a bunch of test cases working, but there are still a few more things to get working before I could apply it to Oil.

                                                      1. 1

                                                        Dropping the Python interpreter requirement makes sense. It’s a lot of code. I wonder how easy it would be to build a stripped-down version of the Python interpreter for “embedded” use cases (no parser / compiler / interpreter, just enough runtime to run Python extensions). Going further, I wonder how easy it would be to statically bundle all the required extensions. I imagine many mypyc-based projects would benefit from a mode like this.

                                                        1. 1

                                                          Yes, I did that for Oil. The whole thing is statically linked and it doesn’t include Python’s parser or compiler.

                                                          http://www.oilshell.org/blog/2017/05/05.html

                                                          http://www.oilshell.org/blog/2018/11/15.html

                                                          It requires a lot of hacking and it’s difficult to generalize, due to the way CPython is written. (Though there is a Rust project trying to do it, I think PyOxidizer). You really need some cooperation from upstream, and upstream there have been many different half-measures over the years.

                                                          But yeah the next step is to go even further and get rid of the whole thing!

                                                          1. 1

                                                            I don’t think you removed the interpreter too, but that would require adding type annotations to all standard library code on which you rely. In any case it looks like you’ve thoroughly explored this route!

                                                1. 3

                                                  Sad that Gamma never gets to understand the bubble in which he lives.

                                                  1. 15

                                                    Thought experiment

                                                    What if this signed firmware was not stored in P/ROM on the chip, but was instead equivalently implemented in actual logic gates (as the rest of the chip is).

                                                    Does that make things better (no firmware blob), do nothing at all (equivalent problems), or make things worse (can’t be replaced ever)?

                                                    Now think the other way. What if we take some chip functionality that is currently implemented as logic gates, but turn it into signed firmware that other parts of the chip interpret. Is that better, the same, or worse?

                                                    Big chips (modern MCUs & processors) have a lot of logic that is just as secret (or more so) as firmware.

                                                    Context: I’m an electronics enthusiast. I always find it interesting to see a focus on non-free firmware, perhaps because it’s an easier target. It’s cheaper to implement a lot of things as rewritable firmware because it’s easier to fix if you make mistakes, but the exact same results can always be had without needing something called firmware. This tradeoff is considered by the chip designers for a lot of parts and features.

                                                    1. 7

                                                      I think people largely object to signed firmware on ‘method’ grounds (it would be cheaper to build without requiring signing; the hardware was made more expensive in order to make it harder to use).

                                                      1. 3

                                                        I think so too.

                                                        My other thought is that ‘method’ works backwards too: if a vendor wanted to make their devices more open then the easiest first step might be to open/unlock their firmware. ie firmware is the lower-hanging fruit, so it’s what privacy-minded people focus on.

                                                        Firmware is a good start, but having free firmware does not make the hardware free. We need free hardware too.

                                                      2. 5

                                                        Yes! Very good point. The conceptual boundary between hardware and software is rather fuzzier than most software people realize. At one extreme we have completely free RISC-V processor RTL that can be run on a big-enough FPGA… but the big FPGA makers encrypt their formats to protect the IP that they also sell. You can contribute to a reverse-engineering effort if you have the right FPGA sitting around.

                                                        1. 1

                                                          Fuzzy is a good description.

                                                          Running free software/designs (like a RISC-V core) on an opaque non-free system seems ironic, but I guess that’s what we already do on our desktop machines :)

                                                          Edit addendum: it’s also worth mentioning that CPUs are only one turing-complete logic family too. FPGAs (as you mention) and DSPs are also good examples. All of these are capable of performing each other’s tasks and emulating each other (with varying degrees of performance). Firmware is just another feature that can be used in this mix.

                                                        2. 3

                                                          Great point. I’d never thought about that before; I really appreciate you raising it. My first thought is that opaque logic is as much of a problem as opaque firmware, but I don’t know where I’ll land after digesting this. I suppose the complexity of the task being accomplished is a factor to consider, but of course since there’s no visibility into what the task is, it’s hard to actually consider that.

                                                          1. 3

                                                            Let’s say a manufacturer produces hardware that does a particular thing, and all the functionality is implemented in actual logic gates rather than firmware. They can’t afford to do anything too user-hostile, or nobody would buy the thing in the first place. If a bug is discovered, and a new version of the hardware is released that fixes it, the bug may be a hassle, but upgrading is also a hassle, so the new version still can’t afford to do anything too user-hostile lest the user switch to a different brand.

                                                            Let’s say a manufacturer produces hardware that does a particular thing, and requires an unsigned, freely modifiable firmware blob. They can’t afford to do anything too user hostile, or users will just patch it out. If a bug is discovered, and a firmware update is released that fixes it, applying the update is much less of a hassle than living with the bug or migrating to a whole other system, and if the firmware update adds anything user-hostile it will be patched out pretty quickly.

                                                            Let’s say a manufacturer produces hardware that does a particular thing, and requires a signed, proprietary firmware blob. The manufacture can release the first version with very pro-user features, and then when a bug is found, release a new version of the firmware that fixes the bug but removes the pro-user features, or adds user-hostile features. Living with the bug is annoying right now, switching to a different manufacturer is annoying right now, but applying the firmware update is easy right now, even if it’s a bad deal in the long term, so almost every human will do that.

                                                            Firmware, as a technology, has a lot of useful and interesting engineering trade-offs, but it also can change the power-dynamic between vendors and consumers. For thousands of years, it was impractical (or even illegal) for a vendor to mess with a thing you bought after you bought it, but now firmware and wireless Internet makes it practical and even easy, for good and for ill. It’ll be a while before society’s expectations of “fairness” catch up to technology, and it’ll be interesting to see where it settles.

                                                            1. 2

                                                              It’s much harder to implement complex functionality in hardware, and much easier to do so in firmware, so it’s not very likely there will be malicious behavior encoded in hardware. Your machine ceases to be under your control if you permit opaque firmware to run without restriction.

                                                              1. 8

                                                                so it’s not very likely there will be malicious behavior encoded in hardware

                                                                Not at all. Malicious behavior does not need to be complicated, and in fact I’d argue it is far easier to hide if it is kept simple.

                                                                Take for example a gigabit NIC: let’s assume that received bits are shifted into 128 bits of shift register on their way in. If a certain (hard-chosen) unique pattern appears in these 128 bits then trigger a payload. If there is no big input shift-register then you could piggback the CRC checksum circuitry instead.

                                                                The payload could be something simple, like shorting the NIC’s input lines to something related to the memory bus (or even PCIE bus), whatever is most convenient. The NIC would then start transmitting a (non-compliant) raw bitstream. Depending on the NIC implementation it might be easy to ‘accidentally’ frame this data up into proper ethernet-compliant signalling too, but this wouldn’t strictly be necessary.

                                                                Result: a way of stealing disk encryption keys from memory. All you need to do is compromise the local router (these tend tend to have garbage updates & security models across various/most vendors), send 128 magic bits and sift through the massive reply.

                                                                Alternatively the payload could be something that exploits existing firmware in ways that people (even with access to the firmware’s source code) could never predict. Eg trigger an execution of the LAN boot rom (switch processor into real mode, jump?) whilst the OS is still running. Now you can deliver any payload over the network. A small amount of the running computer’s memory will get trashed and you’ll probably have to fight the interrupt handler, but if it works then you again get complete remote control.

                                                                Alternative: Wifi chipsets instead of gigabit NICs

                                                                The attack method is a bit different here. Visit a corporate environment with lots of bandwidth & a single brand of routers to make things easier (eg companies, universities, McDonalds).

                                                                Scanning the massive memory dump being transmitted for key strings (informant names, email addresses, website URLs, etc) would be easier than hunting for passwords; but still potentially just as damaging if you are a journalist.

                                                                Your machine ceases to be under your control if you permit opaque firmware to run without restriction.

                                                                Does a machine cease to be under my control if I permit opaque silicon to run without restriction? What’s the difference?

                                                                If anything firmware can be changed, silicon cannot.

                                                                1. 1

                                                                  Since you’re already close, my idea was to do similar pattern matching in PCI or DDR subsystems. With IOMMU’s, it might need to be in memory unit. Local escalation through Javascript or email via a pattern the backdoored unit sees moment it’s sent to the processor. Immediately execute what follows as kernel mode.

                                                                  1. 2

                                                                    The ios security papers assume the baseband can be compromised. A group published a successful hack of one awhile back, but it took a team and 6 months VS an afternoon with DMA. Not perfect, but certainly raising the cost of attacks.

                                                            1. 9

                                                              I’m glad they mentioned that the filesystem operations will be faster. I wish they would explain further as to how they achieved that. This was my biggest concern using WSL last year while I was working on a Magento 2 site. Serving pages was slow, DI compilation was slow, JS build scripts were slow, everything was slow.

                                                              1. 16

                                                                I’m guessing it’s probably the obvious route given the description they use – Linux gets its own block backing store like a regular VM and manages its own cache just like a regular disk, and all the ugly coherence issues are completely dodged. Nobody ever needed a 1:1 filesystem translation in the first place, and they’re too hard and/or impossible to build without suffering the woeful perf issues WSL1 had

                                                                Really sad they’re giving up on the WSL1 approach – an independent reimplementation of Linux had immense value for all kinds of reasons. Must be heartbreaking for a lot of people who worked so hard to get it to where it was

                                                                Anyone care to place wagers on how long it’ll take after release before it’s hacked to run FreeBSD? :)

                                                                1. 15

                                                                  Must be heartbreaking for a lot of people who worked so hard to get it to where it was

                                                                  I thought the same thing! I’m also a bit bummed because the current implementation makes use of the “personalities” feature of the NT kernel, whereas the new one is a VM.

                                                                  1. 10

                                                                    It is a bit of a shame indeed. Also a good example of Agile development. I suppose the initial developers thought it would be a nice way to go, so they tried it out and shipped it. I suppose they couldn’t be sure in advance that the file system would be that much slower, or that people would be that concerned about it. Now that they know that, they had to switch over to a VM to get the performance level that people demanded. Too bad, but hard to predict ahead of time.

                                                                    1. 6

                                                                      Nobody ever needed a 1:1 filesystem translation in the first place

                                                                      Honestly, that was the one reason I would find WSL interesting: seamlessly sharing data between Linux and Windows programs.

                                                                      1. 3

                                                                        an independent reimplementation of Linux had immense value for all kinds of reasons

                                                                        There’s nothing really stopping someone else from doing it (I encourage them to call it ENIW).

                                                                        For what it’s worth, I believe FreeBSD still has a Linux emulation layer.

                                                                        1. 2

                                                                          I believe SmartOS does too.

                                                                        2. 3

                                                                          Yeah this was the logical conclusion of the WSL team initially targeting ABI compatibility with Linux. Diversity of implementation would probably have had a better chance of surviving if the WSL team targeted POSIX source compatibility, like macOS. That would have given them more wiggle-room.

                                                                          That’s not to say the WSL team made the wrong decision to target ABI compatibility, they likely have different goals for their corporate customers.

                                                                          1. 6

                                                                            Their goal wasn’t an OS though, they want to use the old Mac OS X argument: you can run all your Windows programs on your Mac, so your only machine should be a Mac.

                                                                            Just swap Mac and Windows.

                                                                            1. 2

                                                                              If they just wanted POSIX source compatibility, they could have continued development of the POSIX subsystem / SFU. Apparently there’s something to be gained from being able to run unmodified Linux binaries (otherwise we’d just have Windows Ports). My guess is that the goal is to grow Azure – it’s likely to be a bigger revenue source than Windows in the future.

                                                                            2. 3

                                                                              Really sad they’re giving up on the WSL1 approach – an independent reimplementation of Linux had immense value for all kinds of reasons. Must be heartbreaking for a lot of people who worked so hard to get it to where it was

                                                                              It seemed like it involved a lot of kludges onto NT though - to the point it seemed the easier approach to me was refurbishing the old POSIX subsystem and making it feel modern. There was a lot of (wasted) potential there.

                                                                              1. 1

                                                                                Anyone care to place wagers on how long it’ll take after release before it’s hacked to run FreeBSD? :)

                                                                                It sounds like it’s a modified Linux kernel, so it would probably also have to be a modified FreeBSD kernel.

                                                                            1. 17

                                                                              It goes without saying that the vast majority of people involved in creating open source software are moral, fair, and hard working people. Unfortunately there does exist a minority of “celebrity-status” malicious actors, I’ve had the unfortunate experience of directly interacting with a few of them.

                                                                              It’s hard for good people to write exposes like this because they don’t want to be associated with petty drama or the optics of being someone that attacks others online. At the same time articles like this can be important as a warning to others, and to root out bad behavior in our communities.

                                                                              Obviously we should hear the other side of the story, no one should be condemned without being a given a chance to defend themselves. Would like to see others who have also been maligned to share their stories.

                                                                              1. 21

                                                                                There also exists a sort of person who seeks that “celebrity-status”, and I suspect most celebrities – even in “open source” are that sort of person. I’ve worked in media for a while (usually in a technical capacity, but at a high enough level that I’ve gotten to go to some parties), and bigger celebrities I’ve met seem to have some wiring that allows them to quickly evaluate their interactions and determine if this new person they meet produces potable praise or access to new channels, and shut down – what they perceive as – the wastes of time. I predict any response will be to malign njs as just that – a waste of time.

                                                                                However I think the real problem is the rest of us: Looking for heroes and champions to celebrate their life- to follow their amazing stories, rather than having our own. Why do we even care?

                                                                                When we sit around the campfire and share with our closest friends our craziest and funniest stories, what are we enjoying first, a friend who would help us move? give us the best advice when we need it most? or a fun laugh with a cool bro?

                                                                                I think a lot of people get the latter, and maybe for luck or for a devolved and increasingly critical community, we have far few of the former. To some, maybe it sounds like that voice in their head asking, how can we have only one or two friends, when others have dozens? Yet surely, we’d rather one close friend to help in a time of need, rather than a dozen people to laugh at us when we’re in trouble?

                                                                                Understanding ourselves the kind of human interaction we ourselves crave may make us more immune to the wiles and guiles of those who crave something else… something wrong.

                                                                                1. 3

                                                                                  There also exists a sort of person who seeks that “celebrity-status”

                                                                                  Overt status-seeking behavior is a red flag in my book (maybe me projecting?), for whatever that is worth. Like you said, it leads to people being heavily transactional because they’re so intent on continuing to win. Life is not about winning. (This is not a statement on Kenneth).

                                                                                  However I think the real problem is the rest of us: Looking for heroes and champions to celebrate their life- to follow their amazing stories, rather than having our own. Why do we even care?

                                                                                  Precisely. Celebrity culture is stupid, and tech’s doubly so.

                                                                                  1. 1

                                                                                    So what can we do?

                                                                                    1. 1

                                                                                      I was wondering the same thing. In one sense, celebrity culture in technology is stupid, in another, it’s not at all.

                                                                                      Take for instance the fact that Guido Van Rossum is a celebrity in the Python community. He invented the language, and constantly puts effort into both its substance and its community. What precisely is bad about his celebrity status?

                                                                                      Are we maybe trying to come down on people who become celebrities through ceaseless self promotion as opposed to those who get there because of the sheer volume of adoration they receive from their community?

                                                                                      1. 4

                                                                                        I think it’s the difference between stewardship and self-promotion. There’s nothing wrong with respecting the hard work of open source contributors and appreciating the time and effort that goes in.

                                                                                        However, I think that is wholly different from becoming a brand. Perhaps it is the leftover hangups of the old guard hacker culture that distrusts brands, or social media’s pervasive live and die by the sword mentality, but once you cross over into the territory of brand then it seems like people think anything is fair game, unfortunately.

                                                                                        1. 2

                                                                                          Agreed!

                                                                                          Maybe all we need is “good” celebrities to call out the bad ones…

                                                                                          1. 3

                                                                                            I think celebrity culture is dangerous because it ends up being this pseudo-meritocracy of people who are most visible are disproportionately rewarded while others who may be contributing in significant ways get ignored because they didn’t happen to get lucky on HN. It’s a sort of centralization of influence. Like many forms of centralization, it ends up being a collusion of power that everyone tries to get in on, rather than fix.

                                                                                            The only real idea I have towards solving it is a community that actively prizes decentralizing influence. I perceive this as a hard sell to the many devs who won’t touch a GitHub repo unless it gets its first 100 stars, because there is significant value in outsourcing trust and influence: namely, the individual is absolved of having to do it. But what ends up happening is the people who are actively driven to seek such things are able to lap it up while most other people are busy just doing things.

                                                                                            This is related to the rising corporate influence on OSS as well.

                                                                                            1. 2

                                                                                              The only real idea I have towards solving it is a community that actively prizes decentralizing influence.

                                                                                              How do we do this?

                                                                                              A fully anonymous codebase seems impractical – most people can’t read code (even those people who call themselves “programmer”), so is there anything else we can do?

                                                                                              This is related to the rising corporate influence on OSS as well.

                                                                                              It happens everywhere: When we measure “only results” we’re just creating another KPI to be gamed, and people will game it.

                                                                                              How do we create a game without points?

                                                                                              1. 2

                                                                                                Become people who want to contribute without winning :)

                                                                                            2. 2

                                                                                              They wouldn’t. Why should they bother engaging in such petty drama when they can write more awesome code? :)

                                                                                    2. 4

                                                                                      Yeah. It’s not clear to me that in this case Kenneth was actually malicious as opposed to scatter brained and disconnected from the effect his actions were having on others.

                                                                                      On the other hand, reading what others have written about how his mental health issues mightn’t be enough of a reason to explain away such behavior, maybe it’s a foregone conclusion.

                                                                                      1. 5

                                                                                        Sorry, my comment about “malicious actors” wasn’t intended to confirm what the article implies about Reitz. It was more a meta-comment about the unfortunate and perhaps counter-intuitive reality that malicious actors do indeed exist in the philanthropic world of open source, and also, in my experience, tend to be disproportionately represented by the public celebrity types.

                                                                                        I agree that it’s not clear that Reitz’s behavior as described in the post was based in malicious intent. The claims laid out against him also don’t necessarily seem to be ethically questionable either.

                                                                                    1. 1

                                                                                      Language-wise, why would one use Zig over C? From this page, it doesn’t seem like Zig is different from C outside of minor syntactic sugar.

                                                                                      Toolchain-wise, it makes the case for why the zig command is useful, which makes sense to me, since this is an area where C is deficient.

                                                                                      1. 15

                                                                                        Huh? That’s that the whole page is about. C doesn’t have: generic data structures and functions, compile-time reflection and code execution, order independent top-level declarations, non-nullable pointer types, allocator types, etc.

                                                                                        None of those things are “minor syntactic sugar”.

                                                                                        If you don’t understand what those things mean, then it would be better to frame your comment as an honest question. Otherwise, what you are asking is explained very well by the article.

                                                                                        1. -5

                                                                                          Outside of compile-time reflection, I consider those other features minor syntactic sugar. Sugar aside, for what it’s worth, I’ve never felt that the C language was lacking for not providing those features. As an experienced C programmer, pet features aren’t enough of an incentive for me to migrate to Zig, and I don’t think Zig can compete with C on the basis of pet features.

                                                                                          1. 14

                                                                                            Generics and non-nullable pointers are certainly not minor syntactic sugar, and to me they make Zig potentially more expressive and safer than C. Doesn’t hurt code re-usability either to have generic data structures.

                                                                                            1. -4

                                                                                              Would you stop writing C and exclusively use Zig in all previous situations where you would have used C just because it has generics and non-nullable pointers?

                                                                                              1. 4

                                                                                                When it reaches 1.0, yes, sure, anything to avoid C’s army of footguns.

                                                                                                1. 1

                                                                                                  What features in 1.0 are you anticipating?

                                                                                                  1. 3

                                                                                                    the same features it already has, mostly, but with a self hosted compiler and better stdlib and tooling (like editor plugins). I’m not the person to ask. The biggest 1.0 feature is language stability, which is required for adoption.

                                                                                                    1. 1

                                                                                                      What makes you think Zig is not stable enough today? For what have you used it?

                                                                                            2. 10

                                                                                              Did you read this section? https://ziglang.org/#Performance-and-Safety-Choose-Two

                                                                                              Based on the fact that you’re saying “pet features” I think you either didn’t see this, or didn’t grasp the implications of this.

                                                                                              1. -3

                                                                                                Putting the reading comprehension abilities of your target audience into question when they aren’t convinced by your arguments is probably not the best way of increasing the user-base of your project.

                                                                                                As far as detecting integer overflows at runtime goes, clang and gcc have that ability with UBSAN. https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html

                                                                                                1. 3

                                                                                                  Have you ever tried to turn on multiple of the sanitizers at once?

                                                                                                  1. -4

                                                                                                    No, what is your point? I only need UBSAN to replicate the behavior of Zig.

                                                                                                    1. 6

                                                                                                      Incorrect. UBSAN does not catch:

                                                                                                      • invalid union field access
                                                                                                      • unsigned integer overflow

                                                                                                      More capabilities are planned. Clang does not support more than one sanitizer at once; if you want UBSAN and ASAN you’re out of luck. incorrect, see below

                                                                                                      It also does not allow you to selectively disable checks in the bottlenecks of your code. also incorrect

                                                                                                      1. 1

                                                                                                        UBSAN provides -fsanitize=unsigned-integer-overflow, which does catch unsigned integer overflow. I’m not sure what you exactly mean by invalid union field access, but UBSAN also provides -fsanitize=alignment which catches misaligned structure deferencing.

                                                                                                        ASAN only catches memory errors, I don’t think it catches what you are referring to as invalid union field access, so still not sure what your point is about running ASAN and UBSAN simultaneous. As I said before, UBSAN is enough to replicate Zig’s features and much more.

                                                                                                        Clang does not support more than one sanitizer at once; if you want UBSAN and ASAN you’re out of luck.

                                                                                                        This claim is simply false, this works fine:

                                                                                                        clang -fsanitize=address -fsanitize=undefined foo.c
                                                                                                        

                                                                                                        It also does not allow you to selectively disable checks in the bottlenecks of your code.

                                                                                                        That’s also false, see __attribute__((no_sanitize("undefined"))) https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#id10

                                                                                                        1. 4

                                                                                                          I stand corrected. I have corrected the misinformation in my posts, but you have not corrected the misinformation in yours.

                                                                                                          UBSAN provides -fsanitize=unsigned-integer-overflow, which does catch unsigned integer overflow.

                                                                                                          This is inherently flawed since unsigned integer arithmetic in C is defined to be twos complement wraparound. There will be false positives.

                                                                                                          I don’t need to win this argument. I was misinformed on a couple points which make my position less of an open and shut case, fine. There are plenty of other talking points. See you in the thread for 0.5.0 release notes, I look forward to reading your comments then.

                                                                                                          1. 0

                                                                                                            I stand corrected. I have corrected the misinformation in my posts, but you have not corrected the misinformation in yours.

                                                                                                            What misinformation have I put forth?

                                                                                                            This is inherently flawed since unsigned integer arithmetic in C is defined to be twos complement wraparound. There will be false positives.

                                                                                                            Just because it’s defined doesn’t mean it’s intentional. In 80% of cases (read: pareto majority) unsigned integer overflow is not intended in C.

                                                                                                            See you in the thread for 0.5.0 release notes, I look forward to reading your comments then.

                                                                                                            Believe it or not, it’s not my mission in life to trash your language. I believe I gave fair feedback. Many of the people here may pretend to support you, but it doesn’t seem like many of them will actually use the language when asked directly. If you want to eventually appeal to the majority of real C programmers, and not a minority of hobbyist enthusiasts, it would be wise to not take negative feedback defensively.

                                                                                                            1. 5

                                                                                                              I believe I gave fair feedback.

                                                                                                              To me it seems more like, for whatever reason/motivation, that you just came to this thread to do some trolling and to generally be combative with folks.

                                                                                                              1. 2

                                                                                                                He just said in his opinion the features are not worth making the switch. That is just an opinion - an honest one at that. I don’t know if I would call it trolling…

                                                                                                                1. 1

                                                                                                                  Hmm trolling is defined as feigning a position in bad faith. I’m not feigning any position, I genuinely believe the feedback I am providing, and I’m providing this feedback in good faith. That’s actually the opposite of trolling. What statement specifically made it seem to you like I was trolling?

                                                                                                                  1. 4
                                                                                                                    1. -3

                                                                                                                      Ugh an accusation of sealioning requires an assumption of bad faith, which is an act of bad faith itself. You have no rational basis to assume bad faith on my part, and criticism itself isn’t an indication of bad faith.

                                                                                                2. 6

                                                                                                  I somewhat agree that if you look from a big picture perspective, many of the features are just syntactic sugar. There are not significantly more things you can do with zig that you can’t do with C. I also think that the sum is greater than the parts, and all the ergonomic features will make a large difference in how pleasant it is to write code.

                                                                                                  Things like defer statements and automatic error code types may be enough to push bug counts down for example.

                                                                                                  Generics and a compelling dependency manager probably would be a fundamental shift in how people use the language vs C too.

                                                                                                  1. 4

                                                                                                    Just try to imagine what kind of changes would be required for C compilers to implement this minor syntactic sugar. On top of all the things that were already mentioned, there’s some support for error handling and it seems to have some sort of type-interference.

                                                                                                    It’s perfectly fine you don’t necessarily consider these features important enough to use Zig in favor of C but technically speaking, these differences are quite major.

                                                                                                    From Wikipedia:

                                                                                                    Language processors, including compilers and static analyzers, often expand sugared constructs into more fundamental constructs before processing, a process sometimes called “desugaring”. […] This is, in fact, the usual mathematical practice of building up from primitives.

                                                                                                    Transcompilation from Zig to C would be far beyond a simple expansion (assuming you don’t consider C to be a syntactic sugar for assembly in which case yes, it would).

                                                                                                    1. 3

                                                                                                      C++ competed with C on the basis of having extra features like that. There’s a chance for Zig to as well, esp if maintaining simplicity.

                                                                                                      Far as syntactic sugar, I think anything that increases productivity and maintainability is a net benefit. You might continue writing and rewriting more code in C for nothing to perform the same tasks. At same time, there might be other people who don’t want to work with 1970’s limitations if a language is available without them in 2019. They’ll gladly take a chance to reduce code required, make less breaking changes, and catch more errors at compile time.

                                                                                                      1. -1

                                                                                                        My argument isn’t that language features aren’t good. If you want an advanced language, use Rust or C++. But if you want something simple like C, is using Zig really worth it?

                                                                                                        Be honest here, is Zig’s language feature set compelling enough to you such that you will abandon C, and exclusively use Zig in all places you would have previously used C?

                                                                                                        1. 9

                                                                                                          But if you want something simple like C

                                                                                                          The C standard isn’t simple. A subset of Rust might even be simpler than analyzing all that and the compiler behaviors. We’re talking a complicated language that looks simple vs another language that might be as or less complicated. Could be an advantage.

                                                                                                          “and exclusively use Zig in all places you would have previously used C?”

                                                                                                          You keep saying that but it’s nonsense. I always watch out for people saying a proposal should “always” or “never” be (property here). They’re often setting something up to fail rather than trying to understand real-world pro’s and con’s. The reason is that there’s a large variety of contexts that a solution might apply to… or not. There might be uses for C that a C replacement isn’t as appropriate for. Especially while the replacement is less mature without decades of ecosystem investment. More like there will be “some places” or “most places” to use Zig with others using C. Over time, some people might hit the “all places” point. Not a good starting point for evaluation, though.

                                                                                                          1. -5

                                                                                                            There might be uses for C that a C replacement isn’t as appropriate for.

                                                                                                            Interesting logic. The fact that you have avoided answering whether or not Zig replaces C for you, when that is the explicit intended purpose of Zig, shows that your defense of Zig is sort of phony so really not worth acknowledging.

                                                                                                            1. 6

                                                                                                              Zig is an alpha language in development. C took some time to get where it is. You’re fine with that but Zig has to, in version 0.4, immediately replace a language with decades of development in all situations. You should ditch C, too, on that basis since people with high-level languages and efficient assembly were saying similar things about it early on. Alternatively, allow Zig a chance to develop and attempt to prove itself incrementally like C did.

                                                                                                              1. -3

                                                                                                                You’re arguing against a point I never made. I’m criticizing Zig on the features it’s offering today, not what it may provide tomorrow. Could Zig be a viable replacement for C in the future? Sure, we agree there. Am I saying that development of Zig should halt? No.

                                                                                                                Now be honest here, is Zig’s current language feature set compelling enough to reasonably consider it a replacement for C today? Doesn’t have to be in all cases, let’s say most cases.

                                                                                                                1. 5

                                                                                                                  Now be honest here, is Zig’s current language feature set compelling enough to reasonably consider it a replacement for C today?

                                                                                                                  Depends on the use case and compiler quality. I think one should assume answer is no by default until proven otherwise. It’s currently at a stage where hobbyists with an interest might apply it to various problems C is used for. That will produce the field data needed to evaluate what it can do now and how it might be improved.

                                                                                                                  You moved the conversation pretty far, though. You originally accused it of just being syntactic sugar vs C. That’s like saying C is just syntactic sugar for assembly knowing there’d be a lot of work making a compiler for that “syntactic sugar” to work vs doing all the same things as easily in assembly. Zig is doing a mix of syntax, semantics, and analysis to attempt to bring benefits over what C and its compilers do. The compile-time programming alone looks like a huge improvement over C’s preprocessor.

                                                                                                                  1. 1

                                                                                                                    Depends on the use case and compiler quality. I think one should assume answer is no by default until proven otherwise.

                                                                                                                    Thanks for agreeing with me. By default I wouldn’t use Zig over C either. It might be useful if you let the creators know what it would take for you to generally default to using Zig over C.

                                                                                                                    That’s like saying C is just syntactic sugar for assembly

                                                                                                                    Nope, there is a specific definition of syntactic sugar, and C is most definitely not syntactic sugar for assembly. I would consider Zig’s small set of language features syntactic sugar for C.

                                                                                                                    1. 3

                                                                                                                      If it is syntactic sugar, you should be able to throw it together in an afternoon or so integrating it with C. Nothing semantic or deep that poses challenges. Whereas, he’s been working on this full time for a language that already does things C cant without going far from the language.

                                                                                                                      1. -2

                                                                                                                        If it is syntactic sugar, you should be able to throw it together in an afternoon or so integrating it with C.

                                                                                                                        Yes, with the exception of compile-time reflection, I can implement all of Zig’s language features in C using a combination of macros and a small support library.

                                                                                                                        1. 3

                                                                                                                          with the exception of compile-time reflection, I can implement all of Zig’s language features in C using a combination of macros and a small support library

                                                                                                                          I don’t care about Zig one way or the other, but this is a gross misrepresentation. In Zig, quote, “A generic data structure is simply a function that returns a type”. You cannot implement that with a combination of macros. You could implement a set of predefined parameterised types (lists, sets, so on), but that is not the same thing as supporting declaration of generic types directly in the language, particularly with the flexibility that Zig does provide.

                                                                                                1. 21

                                                                                                  Strange world, I just use a case statement in a shell script

                                                                                                  #!/bin/sh
                                                                                                  case $1 in
                                                                                                      foo) echo hello ;;
                                                                                                      bar) echo world ;;
                                                                                                  esac
                                                                                                  
                                                                                                  1. 6

                                                                                                    Me too - crazy this exists. I don’t know how we got to this point.

                                                                                                    sh -x and you also get perfect debugging output.

                                                                                                    1. 3

                                                                                                      And with https://github.com/koalaman/shellcheck you get an extremely valuable linter as well.

                                                                                                  1. 17

                                                                                                    However, it also defines operator* and operator->. These methods also provide access to the underlying T, however they do not check if the optional actually contains a value or not.

                                                                                                    Wat?

                                                                                                    1. 15

                                                                                                      Using operator* or operator-> on an optional that doesn’t contain a value is undefined behavior, see: spec, cppreference

                                                                                                      I’ve been about to migrate a bunch of APIs from std::unique_ptr<T> to std::optional<T> and now I’m really not feeling as great about that plan.

                                                                                                      1. 19

                                                                                                        On the face of it this seems like an insanely bad API design decision by the committee.

                                                                                                        1. 5

                                                                                                          What should it do? Throwing exceptions would make the API useless to many large consumers. Default-constructing the value type would probably be worse. The C++ type system just isn’t very good.

                                                                                                          1. 25

                                                                                                            What should it do?

                                                                                                            not offering those operators at all and instead forcing the usage of map, flatmap or get_or_default like most languages do.

                                                                                                            We opensourced a library that implements a more functional API here

                                                                                                            1. 2

                                                                                                              What should it do?

                                                                                                              not offering those operators at all and instead forcing the usage of map, flatmap or get_or_default like most languages do.

                                                                                                              That’s unacceptable when you know the optional has a value and performance is critical. Nothing is stopping you from implementing “get_or_default” on top of the operations provided, but you cannot implement operator* as it currently is on top of “get_or_default”

                                                                                                              1. 4

                                                                                                                That’s unacceptable when you know the optional has a value and performance is critical.

                                                                                                                C++ is a zero cost abstraction language and lambdas get inlined. Unless the optional is implemented via a vtable, you should not see any overhead.

                                                                                                                1. you might be right today but what about tomorrow? - will your assumption still be true if someone else will modify some (distant) part of the codebase?

                                                                                                                2. If you know that an optional will always contain a value, why is it in an optional in the first place?

                                                                                                                Nothing is stopping you from implementing “get_or_default” on top of the operations provided, but you cannot implement operator* as it currently is on top of “get_or_default”

                                                                                                                Well the argument is operator* you should not exist as it eventually leads to unsafe code and bugs. You may always know when you can get away of using operator* without prior checks and may not need such construct but most people get it wrong at least once.

                                                                                                                If you work in a team you will realize those unsafe usages become a liability as they are prone to break due to distant changes. As a result this is not very social: this practice pushes the responsibility of debugging or verifying your assumptions to everyone who is contributing to the codebase. Every Time someone works on a feature and coincidentally breaks the unsafe code will also need to fix it up; it becomes his responsibility.

                                                                                                                Lastly, just for the sake of argument: In case your optional implementation does not include a footgun, this is how you could build one:

                                                                                                                T const& optional<T>::operator*()const {
                                                                                                                   return this->get_or_default(*(T*)nullptr);  // that's fine it's just undefined behaviour 
                                                                                                                }
                                                                                                                
                                                                                                                1. 2

                                                                                                                  You shouldn’t be forced to check if the option is filled each time you access it. One check is enough, then you can assert that the option is filled in all following code paths. If you don’t want to do that, force your team to always use “get_or_default” but don’t force the world to work like your team, not everyone has your performance constraints.

                                                                                                                  1. 1

                                                                                                                    You shouldn’t be forced to check if the option is filled each time you access it.

                                                                                                                    You only test once. If you are worried that running map or get_or_default test for the content multiple times, don’t worry, the compiler will optimize it away: https://godbolt.org/z/7RGUCf

                                                                                                                    1. 1

                                                                                                                      That’s not true in all cases.

                                                                                                                2. 2

                                                                                                                  If you know a value exists and performance is critical …why are you using std::optional in the first place?

                                                                                                                  By this logic, you might as well demand that std::map should have implemented operator[] not to take const Key& key but instead an already hashed bucket identifier, and if there’s nothing in that bucket, or not what you were expecting, oh well, have some undefined behaviour?

                                                                                                                  It’s silly. If you don’t want the overhead of a particular container, don’t use that container.

                                                                                                                  Insisting that std::optional container has to have a hole in it … either you want an Option or you want a raw pointer. “I want an Option but it has to behave exactly like a bare pointer” is an incoherent position for an API trying to satisfy everyone and ending with something useful to no one. “It’s safe as long as you always 100% without fail manually check it before dereferencing, otherwise undefined behaviour” adds nothing that wasn’t already available.

                                                                                                                  1. 2

                                                                                                                    If you know a value exists and performance is critical …why are you using std::optional in the first place?

                                                                                                                    What if you already checked it once? Why should you be forced to check each time you access the contained value?

                                                                                                                    Insisting that std::optional container has to have a hole in it … either you want an Option or you want a raw pointer.

                                                                                                                    It’s not a raw pointer, it’s a boolean paired with a value. There’s no necessary indirection.

                                                                                                                    1. 2

                                                                                                                      What if you already checked it once? Why should you be forced to check each time you access the contained value.

                                                                                                                      Then extract the value the first time? Why would you design an entire API around “people might unnecessarily check for whether some value is present multiple times”? Leave Nones at the edges of your program. Once you’ve determined that there is Some(x), everything else should be dealing with x, not the Option it started in.

                                                                                                                      The question, if anything, illustrates exactly why these are bad operators to have in the API.

                                                                                                                      Without them, Options are cumbersome enough that you very quickly realize slinging them around in the core of your program is silly; you extract the present value, substitute a default, or deal with an unsubstitutable None at the edge, and design everything else in terms of the value.

                                                                                                                      With these operators apparently a nonzero subset of people think “I should just leave the value in the Option and touch its raw memory because “I know” I checked for presence once upstack”. Which works until someone refactors something and suddenly there’s a codepath that pulls the trigger on your footgun.

                                                                                                                      Good API design guides the user towards good usage. This clearly doesn’t.

                                                                                                                      It’s not a raw pointer, it’s a boolean paired with a value. There’s no necessary indirection.

                                                                                                                      It’s a raw pointer (T*) you’re getting back in the case of operator* and operator->, the operators this entire thread of posts is discussing. If you’re going to blindly call those you could equivalently have just slung around raw pointers wherever you acquired the option to begin with.

                                                                                                                      1. -1

                                                                                                                        What if you already checked it once? Why should you be forced to check each time you access the contained value.

                                                                                                                        Then extract the value the first time?

                                                                                                                        Hmm I’m starting to doubt that you fully understand how optional works. A value is stored in the optional, not a pointer. Extracting the value results in copying a potentially large block of memory. There are many legitimate instances where it makes more sense to access the value from the optional without copying it.

                                                                                                                        It’s not a raw pointer, it’s a boolean paired with a value. There’s no necessary indirection.

                                                                                                                        It’s a raw pointer (T*) you’re getting back in the case of operator* and operator[], the operators this entire thread of posts is discussing. If you’re going to blindly call those you could equivalently have just slung around raw pointers wherever you used the option to begin with.

                                                                                                                        operator*() returns a reference type. In any case, what is returned is irrelevant because an optional is a boolean paired with a value. It’s for situations when you need to pass around optional data in value form and can’t use pointers. Traditionally, optional return values were limited to pointer indirection (i.e. using NULL) but optional allows you to do the same without indirection.

                                                                                                                        1. 1

                                                                                                                          Hmm I’m starting to doubt that you fully understand how optional works.

                                                                                                                          Spare me your condescension, oh self-appointed C++ guru. I’ve gotten paid more than enough for it over the years.

                                                                                                                          I said “extract the value” as in, store a ref or move it or if it’s cheap copy it, but don’t just keep passing the optional around calling value or (god help you “because you cross your heart checked it upstack”) operator->. Luckily value safely gives you the same T& that operator* unsafely gives you. You can even move it safely, avoiding the need for expensive copies, if necessary.

                                                                                                                          A value is stored in the optional, not a pointer. Extracting the value results in copying a potentially large block of memory. There are many legitimate instances where it makes more sense to access the value from the optional without copying it.

                                                                                                                          Again, if you’re in a situation where: you have an optional value AND it’s too expensive to copy AND you want a reference to the memory in the optional AND you don’t want to pay for the single solitary check one time check when you call value to extract the value THEN either you’re terminally confused because you somehow think:

                                                                                                                          if(opt.has_value) { 
                                                                                                                             SomeVal& v = *opt
                                                                                                                             // do something with v 
                                                                                                                          } else { some other logic }
                                                                                                                          

                                                                                                                          is any amount of checks or derefs or copies faster than the safe alternative of

                                                                                                                          SomeVal& v = opt.value_or(/* some other val */)
                                                                                                                          // do things with v
                                                                                                                          

                                                                                                                          OR you absolutely can’t even deal with the overhead of the single check you do have so you’re intending to blindly do

                                                                                                                          opt->foo
                                                                                                                          opt->bar
                                                                                                                          

                                                                                                                          in which case, again just start with a pointer to the possibly undefined memory in opt and save the middleman.

                                                                                                                          1. 0

                                                                                                                            Hmm I’m starting to doubt that you fully understand how optional works

                                                                                                                            Spare me your condescension, oh self-appointed C++ guru. I’ve gotten paid more than enough for it over the years.

                                                                                                                            It’s not condescension, you were arguing along the assumption that optional as it currently exists has no benefit over using a raw pointer, completely ignoring the fact that it gives optional semantics to value types. That isn’t a dig at your general C++ knowledge, just your understanding of optional.

                                                                                                                            You can even move it safely, avoiding the need for expensive copies, if necessary.

                                                                                                                            std::move() is intended to speed up value types that contain pointers like std::vector. It doesn’t make moving a large struct fast.

                                                                                                                            A value is stored in the optional, not a pointer. Extracting the value results in copying a potentially large block of memory. There are many legitimate instances where it makes more sense to access the value from the optional without copying it.

                                                                                                                            Again, if you’re in a situation where: you have an optional value AND it’s too expensive to copy AND you want a reference to the memory in the optional AND you don’t want to pay for the single solitary check one time check when you call value to extract the value THEN either you’re terminally confused because you somehow think:

                                                                                                                            There are serious caveats in defining value_or() in this construction: “SomeVal& v = opt.value_or(/* some other val */)”. How would you define it?

                                                                                                                            in which case, again just start with a pointer to the possibly undefined memory in opt and save the middleman.

                                                                                                                            I’ve already said that the point of optional is to give optional semantics for those cases when you must use a value type and not a pointer. Do you understand what that means? What if you can’t use the heap/a pointer?

                                                                                                                3. -1

                                                                                                                  That’s not idiomatic C++.

                                                                                                                  1. 7

                                                                                                                    Could you clarify what you mean by idiomatic? What design would be both idiomatic but not permit dereferencing a null/uninitialized pointer?

                                                                                                                    1. 23

                                                                                                                      dereferencing a null/uninitialized pointer is idiomatic C++

                                                                                                                      1. 3

                                                                                                                        and segfaults…

                                                                                                                        It’s true that nullpointers may be valid addresses on some architectures but we should always model the semantics correctly in our programs.

                                                                                                                        But what does not make sense is dereferencing an empty optional/smart-ptr. If a null-pointer is a valid and intended value than you should assign it to eg. your optional/smart-ptr.

                                                                                                                4. 9

                                                                                                                  I mean, the semantics of Option in the abstract just aren’t safely compatible with an operation that blindly assumes Some(x) is present and directly grabs x’s memory, so the choices are either:

                                                                                                                  1. use exceptions to avoid unsafe accesses at runtime, which as you say, not every codebase will appreciate

                                                                                                                  Or

                                                                                                                  1. just don’t offer those operators on Option. They fundamentally compromise the entire point of the type. If you want to make blind assumptions about memory being initialized, use a raw pointer; Option should enforce its safety benefits.
                                                                                                                  1. -3

                                                                                                                    That’s not really how C++ works.

                                                                                                                    It offers those operators, you can use them if you want to, and if you don’t want to (which you clearly don’t) then don’t use them.

                                                                                                                    1. 13

                                                                                                                      It offers those operators

                                                                                                                      It offers implementors the choice to implement those operators on their types. There is absolutely no requirement that all types do so, let alone do so in a way that provides unsafe semantics. Their presence on std::optional makes little sense given the semantics of Option as an Abstract Data Type

                                                                                                                      1. 2

                                                                                                                        Feels like Java 8’s Optional type all over again.

                                                                                                                        1. 2

                                                                                                                          As bad as Java 8’s Optional#get is, it at least always aborts control flow via an exception if an empty value is forced unwrapped.

                                                                                                                          From what I can see, the undefined behaviour of C++ theoretically allows an implementation to not abort control flow and perhaps use zero values or something, which makes reasoning about control flow invariants in portable code harder.

                                                                                                                          1. 1

                                                                                                                            Why is Java’s Optional bad? Having used both Java and Rust, the only thing I really miss is pattern matching, which is a separate concern from the type itself.

                                                                                                                            (I already know about the complaint that the optional itself can be null, but… baby steps, I guess.)

                                                                                                                            1. 2

                                                                                                                              “Bad” is honestly too strong a word for it; it got a lot right.

                                                                                                                              That said, it gave a force-inwrapping operation the name get which is both too easy to discover and doesn’t signify the exceptional case well enough. A member of the JDK team admitted it was a mistake which is why it’s being deprecated in a future JDK for a more verbose name.

                                                                                                                              The way it handles nulls is also annoying. Not because optionals can themselves be null like any other object, but because returning null in areas like mapping lambdas implicitly converts into empty optional rather than a present optional of a null. While getting rid of nulls like this seems like a good idea initially, it breaks the monadic laws which breaks composability in places.

                                                                                                                              This article from Atlassian details the problem: https://blog.developer.atlassian.com/optional-broken/

                                                                                                                              1. 1

                                                                                                                                Eh. I think of the Java 8 stuff as occupying an intersection of two similar languages that don’t share all of their idiomatic patterns. I know that winds up resulting in yet more inconsistencies in spots, but I don’t know how much that bites people in practice.

                                                                                                                                Then again, I acknowledge most of the stuff I’ve been working with is greenfield— there’s nothing to accidentally refactor the wrong way.

                                                                                                                      2. 1

                                                                                                                        It offers those operators, you can use them if you want to

                                                                                                                        meaning it offers you the option to introduce bugs to your codebase if you want to?

                                                                                                                    2. 3

                                                                                                                      What should it do? Throwing exceptions would make the API useless to many large consumers.

                                                                                                                      And undefined behavior makes it usable?

                                                                                                              1. 11

                                                                                                                For reference:

                                                                                                                Copy elision [is the only allowed form of optimization (until C++14)][one of the two allowed forms of optimization, alongside allocation elision and extension, (since C++14)] that can change the observable side-effects. Because some compilers do not perform copy elision in every situation where it is allowed (e.g., in debug mode), programs that rely on the side-effects of copy/move constructors and destructors are not portable.

                                                                                                                https://en.cppreference.com/w/cpp/language/copy_elision

                                                                                                                1. 2

                                                                                                                  In other words, constructors and destructors are not necessarily good ideas, and should be utilized sparingly.

                                                                                                                  It’s a shame there are so many footguns in CPP.

                                                                                                                  1. 2

                                                                                                                    Well, they’re OK as long as they’re pure, i.e. don’t have observable effects, which is good practice anyway.