1. 10

    I read this and I don’t see how it is exceptional to “all the others”. All I got was a list of things that need to be improved.

    1. 4

      The benefit is reproducible hermetic builds and never having builds with undeclared dependencies that only work by accident. No big deal for tens of files, but absolutely essential with thousands of files in half a dozen languages. I would have killed for bazel at one point in my life (but not installed a JRE… There are limits).

      As for why the post just highlights problems, it sounds like they’re making a new tool. Need to have problems to solve for that.

      1. 4

        In which case “Recommended Improvements for Bazel” seems like a more apropos title. That’s really my only issue with the piece: the title doesn’t match the content.

    1. 5

      I think the idea here must be to not allow the network traffic of background downloads to take resources away from the foreground use of the PS4

      they’re… on freebsd… they could’ve easily used ipfw queues to give lower priority to store downloads… omg

      1. 4

        IIRC, the downloads are actually handled by the ARM SoC in the southbridge, for downloading while the main AMD SoC is off.

        1. 5

          You’re half right, in that Sony added an ARM coprocessor to the console for that purpose. It’s just that something went wrong, and the downloads have to be handled by the main CPU even in rest mode :) Sony never elaborated on the exact reason. They just implied that the ARM processor wasn’t fast enough, but that sounds like total nonsense. A potato can handle a 100Mbps HTTP download.

          Source: http://www.dualshockers.com/ps4-consumes-70w-while-downloading-in-standby-because-it-uses-the-main-apu-and-not-just-the-arm-chip/

      1. 3

        Given all the aspects in which it would seem to be technically superior, how come OpenBSD isn’t eating away at some of Linux’s market share?

        1. 12

          Network effects. Default choices. AWS launches with linux support, so people pick linux, then they pick docker, etc etc etc.

          Linux does solve a great many problems that OpenBSD does not. (Well, it offers a great many features and buttons and such which look like solutions. :)) I’m not sure that one operating system that solves all problems is the best approach, though. there’s a lot of disagreement about how even a desktop OS should work and how components should fit together.

          1. 5

            I had also heard that performance on newfangled hipster tasks (http service, MP scheduling, databases) was nontrivially worse, so I started trying to run an OpenBSD instance on AWS to run some side by side tests in order to invalidate those rumors. Apparently one guy once heard of a file you can UUCP from sunsite.unc.edu that enables 80-character mode so that you can give it a shot, but that was from a Byte magazine reader article so I cannot vouch for its veracity. I had to give up when my eyelids started twitching uncontrollably.

            The reputation for security doubtless partially stems from the fact that installing the thing requires delivering the One True Ring to a locked basement cabinet guarded by leopards in an abandoned mausoleum underneath a nuclear waste disposal site in Afghanistan. Minimization of attack surface and all that. Somewhere, a Haskell core developer is putting down his glasses and rubbing his eyes in admiration.

          2. 9

            Hardware support is what keeps me from switching.

            1. 7

              And historically, perofrmance. I haven’t seen any good performance comparisons in the last few years (that I remember anyway), but several years ago there were several comparisons done that showed somewhat poor openbsd (and netbsd as I recall) scalability (multi-core performance, etc). Linux has often been one of the top performers in those types of benchmarks.

              Go fast until the wheels fly off?

            2. 5

              The following HN comment from Brendan Gregg might be worth a read. It’s answering in detail basically the same question, just for Illumos rather than OpenBSD: https://news.ycombinator.com/item?id=12837972

              1. 8

                The reply is a touch acerbic, but makes some good points too I think. Like why not just give up when linux is 1000 developers ahead? Because half of them are wasting their time inventing ten different tracing frameworks. Ok, admittedly it would be nice for openbsd to have one perf tool, but I think we’re not so far behind as it may seem.

                Also, for something like inteldrm, there’s a team of devs who keep churning the code, which causes kettenis some struggle when he tries to import it, but on the whole we get most of the benefits of that driver at a considerable manpower discount. The precise economics aren’t quite that simple, but not all of the effort spent on linux is locked in linux either.

                1. 3

                  Sure, I agree on both of those points. Some effort will be wasted to duplication and churn, some will have little effect as it can be easily reused by other projects.

                  But if the larger developer base is at all positive, it will add up over the decades. E.g. 20 years ago the common wisdom would have been that the BSD TCP stack was both battle hardened and state of the art, while the one in Linux was flaky and scaled badly with load. It’s definitely the other way around these days (to a different extent for different BSDs), with a steadily increasing delta. And then repeat the same story over dozens of subsystems.

                  Clearly there are things that more manpower doesn’t help with. Like deleting code to reduce attack surface; I’m pretty sure that the larger the team, the harder that is to achieve :) It’s not that developers of non-Linux operating systems should just give up, or anything like that. But the assertion at the start of this thread on Linux’s technical inferiority doesn’t feel realistic to me.

                  1. 2

                    That interesting comment reminds me a lot of the Worse is Better effect that got UNIX to dominance in the first place. There were better systems including some focused on good docs, reliability, and high security. I write about them a lot. ;) They lost with them being in close to legacy mode or gone depending on the product/project. The Linux situation, esp as Brendan Gregg describes it, looks like a repeat of that with Linux doing it to the other UNIX’s. I already knew this seeing the likes of SGI, IBM, and Oracle get on the Linux bandwagon despite having proprietary UNIX’s to push. In contrast, OpenBSD is trying the UNIX version of The Right Thing. It will fail in broad sense because that always happens.

                    It’s destined to a niche of those that care about The Right Thing. You’ve all done a great job keeping it up despite your filter on the kind of contributors you accept and not giving into market’s crap that much. I saw Theo backtrack a bit with VMM admitting OpenBSD might have to do a little more for marketability. Still, the project will probably need a like-minded sponsor with lots of money and/or talent to break past what we’ve seen so far since The Right Thing is always swimming upstream vs Worse is Better which sometimes creates tsunami’s. Those of us focused on quality might always be also-rans in larger scheme of things. (shrugs)

                2. 5

                  Linux is easier to use.

                  How do you search for a package in OpenBSD? How do you get a description in your search results? You can search package names by downloading the index.txt file from the packages directory on a mirror. But anything more sophisticated you have to use the ports tree and the janky makefile system with non-intuitive syntax and cryptic errors to do any interesting searches.

                  It’s actually possible to download and install a Linux distro that comes with a desktop environment, installs quickly, and works fine right away. When I was first getting into non-Windows operating systems in middle school, I tried out OpenBSD because the whole secure OS thing sounded cool to me. I screwed with it for hours, felt like I was getting nowhere, then gave up and installed Linux Mint.

                  I run OpenBSD now, but I’m also a systems programmer / SRE at a database company, so I’m not exactly an “average user” that represents the trend of the market.

                  1. 6

                    pkg_info -Q is what you want. That said - it looks like it doesn’t respect the -d or -c flags (I have a diff to make it work with it though)

                    1. 3

                      yep. i use pkg_info, i browse the ports mailing list, read the website, and also look at my mirror of choice from time to time.

                      i just use cwm that’s in base. i had previously used gnome and xfce from packages. i reckon my willingness to read documentation mostly gets me to where i need to be… other things figured out by trial and error, mailing list, and searching the internet.

                      1. 2

                        You make it seem like reading the documentation is a bad thing? This should be the preferred approach.

                        On linux the preferred approach is asking half assed questions on stack overflow.

                        1. 1

                          i don’t think it’s a bad thing. i’m implying that everyone saying openbsd is difficult to install or use aren’t reading these things and i don’t really understand why.

                      2. 2

                        Ahhhh. I’m somewhat amazed that I didn’t know that. But I will still call it unintuitive. See, I expected something like pkg_search, or to find something searching for “search” in man pkg_info. Searching “OpenBSD search for package” on Google returns a bunch of ancient results about ports. It IS mentioned in the packages FAQ though, but the preview text on Google mentions ports specifically. So I must have searched for it, seen those results, rolled my eyes, looked in the man pages, seen nothing, and decided to just download the index file.

                        Compare to most Linux package managers, which when run with no arguments tell you the commands to install and search. The word “search” specifically is much more well known than “query” because of Google.

                        So even though I’m a bit lazy and it’s clearly my fault for not knowing this as a sophisticated user, I think helping rather than blaming the user isn’t the right strategy towards getting adoption. Linux is really good about that, OpenBSD not so much.

                        1. 2

                          or to find something searching for “search” in man pkg_info.

                          Query is close. I guess it’s a difference in terminology.

                          Searching “OpenBSD search for package” on Google

                          This seems to be a common trend among Linux users, google is consulted with more authority than local documentation. I personally do the same thing.. when finding issues with linux machines I always go to google first.. the crap documentation (often lacking entirely) has trained me to do so!

                          I think helping rather than blaming the user isn’t the right strategy towards getting adoption. Linux is really good about that, OpenBSD not so much.

                          Where was there lack of help and who is doing blaming?

                          1. 3

                            I realize query and search are close, but search is definitely the layman terminology.

                            Googling isn’t just a Linux strategy, that’s what Windows and Mac users do too. Apple in particular is pretty good about making their support pages show up on Google.

                            I didn’t originally mean people blaming the user directly, but rather a UX that “blames the user” in its design, and for OpenBSD I mostly see this in docs or commands. Not being able to find the -Q flag in the man page by searching (querying?) for search is poor UX. It implicitly becomes the users fault for not reading the whole manual. There are no examples either, where surely a common operation like search would be demonstrated. And OpenBSD commands don’t self document or provide assistance, whereas Linux commands will often list the most common usage if you don’t type a valid command. Using OpenBSD feels a bit RTFM, wheras on Linux stumbling around in an interactive session is much more viable, as most things try and point you in the right direction.

                            This goes both ways, on OpenBSD it’s way more likely that if you can’t figure something out, it actually is documented somewhere. But that documentation could be more accessible and searchable.

                            But also user blaming happens directly on misc@. I am part of no other community that ever makes me think “wow these people are such outrageous stuck up assholes, I’m not even sure I want to be a a part of this anymore.” Mostly OpenBSD people are intelligent, articulate, and kind. For example, I’ve had nothing but good experiences talking with OpenBSD folks on lobsters. But wow some of the stuff on misc makes my blood boil.

                            I’ve considered contributing to OpenBSD docs to make them more accessible, especially FAQs / new user guides, but my experiences on misc have always stopped me. I worry that I’ll be shot down for going against the OpenBSD philosophy, and I won’t even be rightly told why, just be told I’m a moron. It sucks, because I love OpenBSD and I want to share it and make it easier for people to learn about, but I feel discouraged from contributing.

                            1. 1

                              again i feel the direct opposite here.

                              i was able to find everything i needed without asking for help and without googling for the most part.

                              still don’t understand how you missed -Q in the pkg_info man page because it’s right at the top in synopsis.

                              maybe i’m a stuck up asshole too.

                              1. 1

                                maybe i’m a stuck up asshole too.

                                I don’t think so.

                                still don’t understand how you missed -Q in the pkg_info man page because it’s right at the top in synopsis.

                                I wanted to search, I scanned the first line for search, /searched for search, moved on. I’m just too impatient. But a lot of people are too impatient, and worse a lot of people just don’t care enough to persist.

                                1. 1
                      3. 3

                        i’m an average to below average user and i use openbsd as my daily driver.

                        i don’t program and my knowledge of computers is intermediate at best.

                        i also work in a non-tech field.

                        i think it’s the easiest and most straight-forward OS to use.

                        so, obviously i disagree with this.

                        1. 3

                          I wonder what you’re doing on lobste.rs :)

                          1. 2

                            Interesting! So, how do you look for packages you need to install? How’d you get going with a desktop environment?

                        2. 3

                          The file system isn’t great…

                          1. 11

                            Please. Greatest filesystem of the 80s. Best decade, best filesystem.

                            1. 2

                              That was Files-11 on OpenVMS with versioning and stuff. Especially integrated with high-uptime stuff like clustering and distributed lock manager. Or NonStop’s Guardian with its crazy reliability. Or first system (SCOMP) certified to high security that added ring and segmented protection to them with system-wide security policy.

                              I do agree it was one of best decades for applied IT with all kinds of awesome shit happening based on what CompSci had worked on in 1960’s-1970’s with better hardware available. A lot of things started coming together that couldn’t before.

                          2. 1

                            Same reason as many other similar situation throughout the industry. People know something else, a lot is built on other systems, marketing, and once you have a certain amount of people using tech X it’s really hard to use something else.

                            Another side effect that kicks in with Linux Distributions, Operating Systems and Programming languages is that at many places if you introduce a technology that isn’t the currently dominant one you will be personally blamed for every single bug or different obstacle it has. It will often be blamed on it, despite also existing on the dominant OS.

                            Something related is that sadly a lot of software that companies and people end up using at some point is written in unnecessarily non-portable ways. I once worked at a company that used a software running a bash script (stored inside a string in a program) and it had a typical non-portable #!/bin/bash instead of #!/usr/bin/env bash. As usually I’d report that and even create a pull request on a huge (in terms of stars, in the tens of thousands) and very hyped software, expecting it would be accepted. After being baffled that the authors didn’t even know about /usr/bin/env and what it does, a link to Wikipedia didn’t help either the conversation stalled. Since that project was being used by the front end developers I kept patch sets for such things around.

                            And while those tiny things are really not hard in general these tiny things add up. I know the problem even exists if you don’t use Ubuntu, but for example Alpine in the Linux world.

                            Having these kind of portability issues leads to the dominant technology in a field to quite often not be the best. The most famous example is how long Internet Explorer stuck, despite Opera and the Mozilla browser, until Firefox came along.

                            When there is a lot of users and developers even pretty bad technology can stick, because there will be widely used workarounds for problems, there will be progress on fronts, even when the architecture is flawed, etc. Change is rarely that quick, especially when many parties are involved and put their money and effectively lives on it.

                            Of course that doesn’t mean that Linux is bad and OpenBSD is great, just that technical superiority or being a bit better than others usually is far from a good indicator for dominance in a field. Especially if marketing and politics have a big presence, but even without. People simply need to have heard of it and a good reason to switch and get into something new to them and settle there. When there is no direct financial effect, that is significant enough people tend to not just switch utilities from one moment to the next. And even when you know OpenBSD really well and are convinced its better introducing that in an existing company might not be an option. Of course you can always switch, but why give up a safe, well-paying job with great coworkers for an operating system?

                            1. 1

                              Docker / namespaces

                            1. 8

                              Honestly, I found this utterly fascinating for some strange (fetishistic?) reason.

                              When I get home, if I remember, I would like to grep through four additional large and source-available and influential older code bases that seemed to have escaped mention: IBM MVS 3.8, Xerox UTS/CP-V, MIT/GE/Bell/Honeywell/Bull/etc Multics, and UM MTS (Michigan Terminal System).

                              These all should predate BSD in many cases by decades and might well have earlier references.

                              Maybe someone will beat me to it.

                              1. 5

                                I had a look through MULTICS, didn’t find anything plausible. The other three sound like they’re worth a shot :)

                                (MULTICS was a bit of a pain. Also AFAIK the only public source for it is a late-stage source dump with no versioning history; so it’s hard to say with any degree of confidence when any specific part was written).

                                1. 3

                                  There are actually two releases, if I recall correctly - an unofficial “leaked” one that originated from MIT, and the officially blessed one from Bull.

                                  While neither release contained modern revision control, major edits and changes were always described in the comments at the beginning of the file, so you are right, it is of little help here; unless you could discover an “XXX” comment referenced in a source file with a pre-81 header — but even then it would likely be impossible to prove the edit date definitively.

                                  I think the sources for the other older systems I mentioned resides on SHARE and Bitsavers.

                                2. 1

                                  For posterity, there wasn’t anything of great interest in these codebases.

                                1. 1

                                  “The companies that designed these protocols happen to control the servers and the client application program, but not really the client OS”

                                  This isn’t entirely the case for QUIC, since Google does develop Android, which is easily the most popular OS in the world.

                                  1. 4

                                    Google develop it, but that’s where it ends. They have basically no control over about getting any kernel level changes installed on actual running devices. 75% of the active Android Install base is on versions over two years old. And the much slower release cadence makes things even worse.

                                    If they add a new TCP feature today to Android, in a year it’ll probably be running on 0.5% of devices. If they add a feature to QUIC on mobile Chrome, it’ll be on 80% of the devices in two months.

                                  1. 5

                                    This article misses a few things:

                                    • You don’t need N to be a power of 2 for the overflow to work correctly, you need UINT32_MAX % N == 0
                                    • …but if N isn’t a power of 2 you need to do % N instead of & (N - 1)
                                    • If you implement this without assert/static_asserting that the above is true then you’re just asking for trouble

                                    This trick can be a nice win when writing lock free queues, but for normal queues it’s so marginal it’s probably not worth it.

                                    1. 7

                                      An actual modulo is just too expensive (unless N is a constant, and the compiler has an optimization for that). For the other two designs you don’t need to use mod to support arbitrary values of N, a single conditional will suffice to wrap the values. That won’t work when the indices have an unlimited range.

                                      (Also, surely only powers of two are going to be factors of another power of two?)

                                      1. 3

                                        (Also, surely only powers of two are going to be factors of another power of two?)

                                        Yeah you’re right, it’s funny I never noticed that.

                                      2. 2

                                        Turns out my entire post is wrong. In lock free code, you actually want the opposite of this trick - keeping reader/writer pos separate. If you have head + length you need to update both variables (e.g. enqueue is rougly head++ length–) which requires a CAS loop that’s twice as wide.

                                        If you’re doing an SPSC ring buffer you can keep the reader/writer pos as non-atomic ints in separate cache lines, and add an atomic flag to each node saying whether it was last read or written. You don’t waste any slots and you only get contention when the queue is nearly empty/full.