1. 46
  1.  

  2. 8

    A better description of what looks like link time reordering of objects - http://marc.info/?l=openbsd-tech&m=149887978201230 :

    A unique kernel is linked such that the startup assembly code is kept in the same place, followed by randomly-sized gapping, followed by all the other .o files randomly re-organized. As a result the distances between functions and variables are entirely new.

    1. 19

      re OpenBSD’s development

      I’m sure it took hard work. Might stop some attacks. Good work getting it done. :)

      re: the title and the OS comparisons rather than OpenBSD team’s good, technical work

      I aways love the comparisons to Windows. The Windows OS was designed to create a monopoly sucking billions of dollars out of people and big companies. It succeeds in that every day with the company founders being billionaires. There’s a bunch of other groups that focused on creating profitable companies that straight-up hire developers to work on FOSS using the piles of money they’re receiving. FOSS-loving people doing that could potentially do more good for FOSS by having a pile of money to use to support it with paid programmers. Most of Linux’s development comes from proprietary companies for instance. So, Microsoft did it better than all in terms of both making bank and being able to fund whatever they care about.

      Now, on the technology. That’s MS Research mostly with some in operations (eg Steve Lipner). So, Steve Lipner came in from DEC watering down Orange Book for Microsoft into SDL. They have fewer 0-days in kernel these days than most FOSS software despite lots of rewriting. MS Research being real gem where they have tools preventing drivers from crashing kernels, verifying pointer safety in C code w/out checks, a language competing with SPARK for provably-safe code that’s low-level, design-by-contract for high-level code, an OS verified to the assembly for safety, protocol verification with Lambert, the Haskell stuff with SPJ, something for asynchronous recently, web services I think, ASM’s for modeling, and so on. The list goes on. In every aspect of development, they have tools that can achieve near perfection of the deliverable with useful prototypes or real-world examples of doing that. Most FOSS, including OpenBSD, have shown no interest in doing something similar despite having the free, labor advantage. Barely a comparison or at least not one that makes MS Research look stupid or incompetent.

      So, the author is hilarious talking trash about Microsoft like they don’t know or contribute to security because a product designed for surveillance and monopoly profits lacks (feature or assurance here). Nah, their organization has the talent and does better than almost all of them I know outside CompSci and a few companies. They use the massive profits from that shit product to fund it, too. Maybe better to learn from them imitating their best qualities (dropping the rest) instead of knocking the “drawbacks” of products that profit massively off those “drawbacks.” It’s not a failure if it was the intended goal.

      Note: Similar statements could be made for mainstream, Linux distros that barely care about security vs usability and adoption. Of course they’re behind on security but ahead on their goals. Whereas, groups that cared put Linux on separation kernels or did compiler-assisted transforms that immunized against common forms of attack. FreeBSD got capability security down to hardware level by CHERI team. Almost nobody cared to adopt or invest in them much like OpenBSD and other security-focused projects. It’s the market’s preference whether proprietary or FOSS.

      1. 6

        Most FOSS, including OpenBSD, have shown no interest in doing something similar despite having the free, labor advantage.

        Probably the biggest advantage they have is that some project can be carried to completion slowly over years without risk of running out of money and getting cancelled. Unfortunately the risk of running out of labor is real. It is cheap, but not plentiful.

        Projects that are primarily driven by volunteers working for free on their limited free time tend to have a labor disadvantage, especially for any subproject that would require cooperation and long term sustained, focused effort from multiple individuals. It is difficult to organize, people come and go as their pool of energy and time fluctuates, work gets spread out over longer time and context switching reduces efficiency further. By contrast, even a small team working full time for a few weeks can have a massive advantage on such a project.

        And this is why, without belittling the work the OpenBSD developers have done, I think it is best suited for low hanging fruit. Little things that can be spearheaded by one person, possibly with some help from a handful of others. For example, let’s randomize xyzzy and see what breaks. Randomizing xyzzy isn’t a ton of work. It’s a fairly isolated change, and doesn’t require you to rewrite or redesign any of the existing software; bugfixes and relatively minor changes suffice.

        Finding all the things that break and fixing them can take a lot of time, but they can all be handled as individual incidents in isolation, and then you move on. There may be tough corner cases that require people to get together and reconsider the impact of the change & possible solutions, but most of the work can be done in small steps. Steps that can be easily spread over time and people, without much organization. Bugs get fixed, and the result is slow evolution towards more secure software. Furthermore, it doesn’t burden other developers’ backs in the future: if anything, bugs will become easier to spot and fix as the system grows less tolerant of them. That sort of thing is manageable with patchy effort from people working remotely across the globe.

        Introducing the straight jacket of verification and provably safe code (but who proves that the proof proves the right thing?) is much more like revolution. We all know you just don’t slap C with it, fix a few dozen bugs, and be done with it. No, it’s a ton of work, it may require substantial rewrites, it requires someone to write the contracts. It changes the way things work in the future so it’s not a little isolated thing someone can do alone. No, if you want these guarantees for more than just one snapshot of the code, you need to get everyone on board and keep doing the work in the future. That is so much more work, and much harder to manage without a tight knit team working together full time. Doing the work and getting everyone on board can be as realistic as “rewrite it in rust.”

        I guess if the same limited labor pool were expended into an endeavor like that, the marginal project would be marginalized even further to something on par with an academic exercise. One that you can then point at as an example of something good that no real world security focused project is doing. I guess this underlines the market preference you mentioned, although I associate “market” too strongly with commercial products to use the word in relation with a project of volunteers scrathing their own itches.

        And then, in that alternative reality, I could be nickpsecurity^(-1) and provide a list of all the low hanging fruit – that could’ve easily improved the security of real world software – that OpenBSD weren’t interested in doing because they were all too busy doing something else…

        1. 4

          I don’t believe in “labor pool” as an explanation of the difference. FOSS communities do not have a formal methods culture, at all. When academia shows up with ideas of tools to adopt to improve software production, they are met with a lot of implicit resistance and “I won’t do anything until the tool is super easy and convenient to use”.

          In contrast, Microsoft has managed to introduce formal-methods tools in its development process; I suspect by a lot of top-down authority moves “here is how we’re going to do things now”, but I don’t actually know how the changes happened internally. I’m sure it had costs, because the tools were not necessarily able to scale to the project’s scale on day one, and there certainly was a lot more necessary work poured into making them easier to use after they started getting adoption internally.

          Personally I think that the lack of interest for formal methods and, in general, most sorts of interaction with academia that do not fit the existing “just send us patches in the way we are used to” workflow (those do happen, for example the systems research community contributes to Linux, and the compiler people contribute to GCC and LLVM), is one of the main issues with open-source communities today.

          The Linux kernel, for example has a lot of companies funding its development today, it’s not a volunteer-on-their-spare-time project, yet almost no one is willing to get serious about interacting with academia (somes exceptions I know of are Coccinelle, and the recent work on formalizing Linux memory models and in particular the RCU mechanism, which happened in great part thanks to the motivation of Paul McKenney).

          It is not a “open-source vs. industry” split, because a large part of industry is just as bad at interacting with academia. Actually Microsoft is in a rather unique position there, given the long-term investment they made in Microsoft Research, so it’s understandable that they would be better at else than pretty much everyone else. (IBM still does have some good research groups, and I’m sure Intel has interesting stuff in hardware-land and physics but not that much in software as far as I know.)

          1. 2

            I don’t believe in “labor pool” as an explanation of the difference. FOSS communities do not have a formal methods culture, at all. When academia shows up with ideas of tools to adopt to improve software production, they are met with a lot of implicit resistance and “I won’t do anything until the tool is super easy and convenient to use”.

            Perhaps one can view the lack of a labor pool as a manifestation of the “culture problem.” Actually putting results from academia into use in the real world takes work and effort, and there needs to be someone to do it – labor. Now if you find a dedicated individual with the drive, maybe that work gets done. Alternatively, someone could do targeted funding to get that work done.

            Now as you point out, funding alone doesn’t magically make people work on the things nickp mentions, unless that funding goes toward that specific goal. Mr. Torvalds may be funded to do basically whatever he wants with his kernel (scratching his itches), and there may be other RH, Intel, etc. employees paid to scratch the itches of their respective company. They still do not get to tell each other to undertake some particular project. Just because they’re paid doesn’t mean they automatically start to work on formal verification of the code.

            I still think funding is a game changer. Coccinelle got funding. Would it be interesting enough for you to mention it if it hadn’t received that funding? Maybe, maybe not. OpenBSD got features because they were funded. Not enough volunteer labour to make it happen.

            Of course project goals (and top-down authority if it takes that) would play a role too. But you just don’t get to tell volunteers exactly what to do.. well, one can try:

            “5 weeks ago at d2k17 I started work on randomized kernels. I’ve been having conversations with other developers for nearly 5 years on the topic… but never got off to a good start, probably because I was trying to pawn the work off on others.” https://marc.info/?l=openbsd-tech&m=149887978201230&w=2

            1. 1

              “they are met with a lot of implicit resistance and “I won’t do anything until the tool is super easy and convenient to use”.”

              Interestingly, they still get that reaction even when they make stuff that’s fairly easy to use or works on existing code. Especially for Linux and FreeBSD, academia does all kinds of push-button tech related to reliability, security, configuration, parallelism, etc. Very little of it gets accepted or built on. We can easily see what benefits utilization would’ve brought just by looking at the benefits the work brought in their papers. For example, although Linux coders don’t do much static analysis or automated generation of tests, the academics that use or build such tools deploying them on Linux or FOSS apps almost always find problems. The projects probably accept the bug reports or fixes but won’t adopt the tools that produced them to begin with.

              There’s been a few that made it. The overall trend is negative toward CompSci’s contributions, though.

              “given the long-term investment they made”

              Another angle to look at with that is CompSci people taking initiative emulates that for FOSS projects like Linux. They’re getting the brains and labor free to them without that initiative. Some of the same stuff is being created, too. They still don’t want to invest in using most of it. One specific example I liked was how Microsoft eliminated tons of crashes or blue screens with their driver, verification tool. An academic team made one for Linux drivers, too, as shown below. I’m not sure if this one is available for download but I doubt it would get used, copied, or extended by FOSS folks even if they had the talent.

              https://pdfs.semanticscholar.org/41ab/8a4be4a0ac015a282d70099147063d2fa72c.pdf

              1. 3

                I see what you mean, but I find your vision to be a bit too unbalanced. In my experience with academic software production, it’s exceedingly rare that a solution can be used as-is by people other than the authors at the scale of a realistic software project. Even more so for program analysis / verification tool, who are by nature incomplete and that a majority of academics develop in an exploratory fashion (trying experiments to see what sticks) rather than with the mindset of creating a robust long-term tool.

                In any case I think it’s not great to point the finger at one group and say: “it the fault of those people”. Even when it might be true (and I don’t think it is the case here), this is not an efficient approach to get people to change their habits! I prefer to discuss the problem in term of the need for cooperation, with efforts coming from both sides (there are also academics that don’t engage with FOSS communities enough, for example). But this requires both sides to be interested in the interaction, and this is matter of community culture; I agree that, on most FOSS communities’ side, there is unfortunately a lack of verification culture, and in general a lack of interest for interaction with academia on software development. (Whereas in my experience, academia tends to value “impact” into FOSS communities as part of its culture.)

                1. 1

                  Oh I’m not trying to imply they’re always very usable. Just that the ones that are pretty usable go unused normally. You’re also right about the verification tools. I really cherry-pick the ones I submit here. Most of the work isn’t as practical. I also agree cooperation is better than finger-pointing to increase odds of acceptance. The solution to FOSS side might be just better marketing of the total benefits (includes maintainability & less debugging) instead of just “it might find or stop bugs.” Better interfaces on the tools, too, with them covering more corner cases. Better packaging & auto-config as well. At least some are on Github now.

            2. 2

              “Projects that are primarily driven by volunteers working for free on their limited free time tend to have a labor disadvantage, especially for any subproject that would require cooperation and long term sustained, focused effort from multiple individuals.”

              Ok. I misspoke in prior post. Let me modify it to say they have potentially a labor advantage where the contributors are free and not forced to use whatever tools companies demand. You’re right in that the actual affect is usually either no contribution or low amount that’s hit and miss. The big ones are outliers.

              “Finding all the things that break and fixing them can take a lot of time, but they can all be handled as individual incidents in isolation, and then you move on.”

              This is an interesting perspective. It could be a psychological motivator for some of these activities. I know two others that we must remember, though. One is backward compatibility with UNIX model. That architecture dictates quite a bit in terms of what they can do with what results. The other is they prefer mitigations that have barely any observable slowdown that also work with little effort by users. The usability aspect is great default but the other thing means they’ll use risky mitigations or obfuscations. Those addressing the root causes almost always have a significant slow-down if still using C language. So, UNIX model, on by default, little to no performance hit, and maybe like you said easy to work on a little at a time.

              Your point would probably apply to other projects even more since the OpenBSD team is unusually committed to doing the hard work.

              “Introducing the straight jacket of verification and provably safe code (but who proves that the proof proves the right thing?) is much more like revolution. We all know you just don’t slap C with it, fix a few dozen bugs, and be done with it.”

              You actually can if aiming at medium assurance. As I told one of them, the least-painful option would be to create a safe variant like Cyclone that integrates well with C in terms of calling it or compiling to it.

              https://en.wikipedia.org/wiki/Cyclone_programming_language

              New code can be written in Cyclone or similar language. Old code can be gradually rewritten over time or they can do a big rewrite. The language differences amount to annotations or differences in structuring where you usually don’t have to change the actual algorithms. The port wouldn’t be easy since they never are. It would be way easier than a non-C language and easier than trying to make everyone experts at never introducing vulnerabilities in C code. Integration with tools such as Frama-C, KLEE, affine types, or assertion-based test generation would just make the quality go through the roof if supplementing their code review.

              That’s all before we even consider things like formal verification done by hand or really changing the structure of the OS. The safe language option is quite doable. Especially seeing what Redox OS did in Rust and Muen kernel + IRONSIDES DNS did with SPARK Ada. Those projects would be harder given the borrow checker or annotations than C developers using a safer C or something w/ lightweight, verification tools. What stops them is personal preference of the team far as I can tell. They just like using C, want a UNIX, prefer to do the code review, and so on.

              “who proves that the proof proves the right thing?”

              That’s barely relevant even though I see why you’d ask. Your TCB always includes your requirements, design, source, tooling, underlying platform, and so on. All you can do is reduce it and make it easier to verify. Decades of evidence shows formal specifications in particular find problems in many parts of the lifecycle, esp ambiguities or interface errors. Formal proofs can find nearly impossible-to-spot bugs but seem to be best for tiny, critical stuff that’s similar to prior, verified work so you know it’s possible to do in the first place.

              Far as how deep trust goes, a lot of this work is done in Coq or HOL specifications. HOL itself has been verified at a logic level with another person making a low-TCB implementation, HOL/Light, being a few hundred lines of code IIRC. Foundational, proof-carrying code and other methods essentially use a prover to generate all the transformations from high-level specs or source down to assembly producing proof terms in as untrusted way as possible. The trusted, lightweight checker such as HOL/Light then confirms each action was logically valid in the rules of the logic and in terms of the formal specification. You essentially trust a pile of unambiguous specifications which are easier to machine-check plus a checker that’s easy to check. Examples from a leading researcher:

              https://www.cl.cam.ac.uk/~mom22/stacks.html

              So, there that is for your curiosity. They’ve taken it much farther than I ever expected reading about the old days of Gypsy Verification Environment, VDM, and so on. With work like CertiKOS, the field has come a long way. :) That kind of thing takes a lot of PhD’s. So, I don’t ask FOSS projects to do that much. Most I’d push is Cleanroom w/ safe languages or Altran/Praxis’ Correct by Construction. The latter combines good design, formal specs, code what one can in SPARK or something similar, thorough review of specs/code, and plenty of testing. Their defect rate is usually around 0.04-0.05 per thousand lines of code with some provably immune to common vulnerabilities. It seems practical given they charge a 50% premium on top of going rate for software projects by experienced professionals. As you said, though, raising the bar on quality using such specialist techniques will reduce contributions a lot. It can probably be done in a quality-focused project such as OpenBSD, though, since they already raised the bar quite high.

              1. 2

                Let me modify it to say they have potentially a labor advantage where the contributors are free and not forced to use whatever tools companies demand.

                That is a very good point.

                You actually can if aiming at medium assurance. As I told one of them, the least-painful option would be to create a safe variant like Cyclone that integrates well with C in terms of calling it or compiling to it.

                “Least-painful” might actually be a fair choice of words in that it admits the work is far from easy. When you say just create a safe variant like Cyclone, you’re perhaps a little too optimistic and assuming that it’s not a hairy problem to solve. Arguably the biggest safety improving feature of cyclone is fat & bounded pointers. Neither are compatible with C, so the project quickly becomes one of rewriting all the code, or perhaps just writing a whole new newc-libc to coexist with old libc. And then same for every other library the applications depend on. Unfortunately, “legacy” C code does not benefit from this. Notnull pointers could be compatible with C, but you still can’t eliminate null checks from functions accepting such pointers if you want to let legacy C call these functions and still have safety. So there’s not much point, and frankly null pointers aren’t such a big safety concern in application code anyway…

                It gets hairier still when you consider allocators or close-to-the metal code where limitations like the inability to convert integers to pointers just won’t fly. Okay, so perhaps we can drop that one feature too because it doesn’t provide meaningful assurance anyway – applications that need to treat pointers as integers are vanishingly rare and ought to get caught in review. Code that needs to do it still needs to do it.

                Then you consider regions. Again not useful feature if you’re interfacing with any legacy C, as it won’t understand these. Then again, current C compilers actually do a fair job of tracking the easy cases of dangling pointers to automatic storage. Complicated cases are still there of course. But these are a rare problem to begin with, and mitigations on the stack (canaries mainly) do a fair job of making such things hard to exploit. And that works for all C software.

                Heap regions are the more interesting feature. Until you realize you can’t free these heap pointers and cyclone “deals” with the problem by using a conservative garbage collector. Nope, nope, that is a non-starter. So now our developer needs to reinvent rust in order to create this little safe variant of C? Or maybe we should forget that feature too?

                There’s a bucketload of nontrivial design issues before you even get around to the implementation. And then there’s the implementation. It doesn’t look like OpenBSD has too many compiler hackers to begin with, and just having an up to date C compiler that supports the relevant architectures has been a depressingly uphill battle. There’s dead ends like pcc and tendra. Old versions of gcc that were kept around for their old-arch support. Or license. Now there’s clang, which went straight into “gigantic, nasty, and unavoidable.” If this is how life goes with C compilers, I don’t see them designing, implementing, and maintaining a dialect of C (including a copy of libc) to go along with it. Not anytime soon.

                I’d argue that a more likely route is, once again, evolution in small steps. That means, on one hand, safer apis that make things like NUL-termination of C strings (which cyclone seems to be obsessed with) less and less of an issue. On the other hand, attributes like bounded and nonnull along with improved static analysis. I’m somewhat optimistic about C being slowly evolved into something that can have some additional guarantees it does not have now, and something that rejects bad or likely-bad code constructs. I wouldn’t even rule out the possibility of something like fat pointers coming to live alongside normal pointers. But slapping on a bunch of restrictions is a huge deal. I’m not optimistic about revolution.

                1. 2

                  Interesting comments. I know too little about C programming right now. I’ll have to think on some of this over time. One I can address immediately, though:

                  “Until you realize you can’t free these heap pointers and cyclone “deals” with the problem by using a conservative garbage collector. Nope, nope, that is a non-starter.”

                  There’s been plenty of GC-using operating systems. There’s usually some unsafe stuff at the bottom. Past that, they work fine with the Oberon-based systems such as A2 Bluebottle being highly responsive. More than the C-code competition I use. There are also low-latency and real-time GC’s. Go makes a lot of use out of one in performance-sensitive code. Then some OS designs such as JX allow one to swap out different collectors for different parts of the system based on their needs. So, technologically, it’s not a non-starter so much as a proven solution that worked even with the crappy GC’s some of the older OS’s used. A mix of non-real-time GC’s, real-time GC’s, ref counting, and unsafe is a valid strategy that stops most problems by default. I’m also for adding the option of Rust-style borrow checking which was inspired by Cyclone by the way.

          2. [Comment removed by author]

            1. 18

              Only a systemd unit file away.

            2. 2

              Which vulnerabilities would this have thwarted/made more difficult?

              1. 1

                Someone could compile their own kernel to turn off security features and allow exploits. Then it’s just a matter of copying the kernel over when you are able and SSHing in later. It’s even possible to package this kernel and market it so other people are encouraged to install it. Then you can just keep track of the IPs that download it and try your exploit to see if you get access.

                https://github.com/lucyoa/kernel-exploits

                http://www.tldp.org/HOWTO/Security-HOWTO/kernel-security.html

                This is obviously not something that would be that easy, but if you could get a copy of your kernel onto a VPS host which is used in server creation, you could very quickly gain access to lots and lots of servers.

                tldr: join the tin hat society because everyone is out to get you :P

              2. 2

                Are there classes of exploit that this is better at defeating than KASLR? It’s definitely neat but I don’t understand the attack vectors well enough to understand why this is useful.

                Does OpenBSD to Kernel ASLR too or is this an alternative to implementing it?

                1. 3

                  (Speaking in general, not directly related to OpenBSD). KASLR doesn’t really defeat exploitation of the kernel due to kernels having info leaks as features and the extremely limited amount of kernel virtual address space. Take a look at this article for a deeper look.

                  1. 1

                    are forum posts really articles now?

                    1. 4

                      I posted most of my designs and essays on Schneier’s blog since there used to be highly-skilled, security engineers with diverse backgrounds. Got great peer review with lots of designs. One time, a reader mentioned one on Qubes mailing list with Joanna dismissing it like you did on same grounds. I slammed her for that pointing out what matters is the actual content plus source. Not where it’s posted. On top of calling out related work that was better designed that she should’ve imitated. Although she dismissed my proposals, I noticed she ended up adding a trusted path and blasting Xen folks about security just like I said she would.

                      All starting in design discussions of microkernel-based systems in the comments of a blog. Yeah, good content can show up anywhere. One day we might even see a PhD-grade post on 4chan. I’m not holding my breath on that one but it’s possible. Judge the content not the location unless you have to blacklist a location out of necessity. 4chan comes to mind again. ;)

                      1. 3

                        yeah i reckon you’re both right and i’m just being an old man

                        1. 3

                          I chalked it up to differing opinions and perspectives. :)

                      2. 3

                        Call it what you will. The post is in the section of the forum called “Blog.” Blog entries can be called articles.

                        Whatever floats your goat.

                      1. 3

                        A lot of KVA info leaks leak a single pointer. From that you can calculate the kernel base address and everything else is relative. The order never changes.

                        There are of course bugs which allow arbitrary reads of kernel memory, but they’re usually rarer. The common case is some data structure is copied out, but a sensitive field isn’t zeroed. Attacker doesn’t have control over what value they read. You get one function pointer address and that’s it.

                      2. 1

                        This seems like a super elaborate but largely unnoticeable prank. “Hey, Jimmy, notice anything different about your cat?” “No.” “Well, you’re in for a surprise! I created a device that creates a portal to another dimension, and then I took your cat and replaced every atom in her body with a different atom that I stole from that dimension.” “Oh… okay?”

                        What does this actually do? I’m assuming perhaps it thwarts hackers from gaining access to the kernel by exploiting vulnerabilities that expose memory locations (such as Heartbleed), similar to how GameBoys and other game systems are hacked through memory for emulation purposes. Is this accurate?

                        1. 1

                          Yeah, I feel the same way. It’s a neat trick, I assume it helps security in some way, but I have no idea how or why I should care.

                          The only thing I can think of is that it partially defeats vulnerabilities that depend on overwriting kernel code in specific locations, but it seems like it’s solving a symptom, and not the real problem.

                          1. [Comment removed by author]

                            1. 1

                              Full disclosure: I’m going to be annoying and nitpick one word: impossible. ;)

                              Nothing is impossible when it comes to manipulating weird machines.

                              The purpose of exploit mitigations is to drive up the economic cost of the attacker. The more costly it is, the fewer potential attackers there will be.

                        2. 1

                          Will this break reproducible builds so you won’t be able to check if the binaries have been modified from the source?

                          1. 3

                            I’m not sure OpenBSD builds are reproducible to begin with. But if they were, you could checksum the object files to verify that they match the source. Then assuming your linking process hasn’t been tampered with, the kernel will be legit if the objects are legit.

                            1. 3

                              you can checksum the linker, too, if you trust the source from which you got it. But, of course, it could have a Thompson style recognizer that subtly inserts malicious code that no one knows about… :)