1. 6

  2. 9

    Any time this is brought up, I feel obligated to post a key issue: you need verifiable systems with verifiable builds, not unverified systems with reproducible builds. Mainstream security often solves the wrong problem with wrong method. Interestingly, the people who invented the right methods, Schell and Karger, also invented the “Thompson attack” and later high-assurance security to combat it all. Wheeler added to high-assurance field with a FLOSS page on it and (relevant here) a page on SCM security w/ links to everything from FLOSS to high-assurance considerations. Others in CompSci have gone further with “certified compilation” (CompCert), “proof-carrying code” (FLINT), and “simple all the way down” (eg Edison, Oberon) systems amenable to verification for correctness or trustworthiness. All that ignored but at least they keep [incorrectly] citing the compiler-compiler attack and solution least likely to matter to one’s actual security & pouring work into a tiny aspect of trusted distribution.

    So, here’s what you do if you want to go all in. My method. Ignore reproducible builds since that’s some crap that requires trusting whole systems and compilers likely to have 0-days or optimize away security checks but too complex to eliminate such risks or even spot all their uses. Ridiculous. Instead, start with a macro assembler or interpreter for idealized assembly that even amateurs can tell is correctly parsed, translated, and turned into binaries. Examples include simple LISP’s, P-code, Hyde’s HLA, and so on. Let’s call it 2GL. Then, implement a LISP, ML, Oberon, whatever with the 3GL statements in macro or template form that basically directly convert the 3GL language into the 2GL equivalents so 2GL’s compiler/assembler can handle it. Code initial, non-optimizing compiler in 3GL, test using arbitrary hosts in whatever language/models, convert final code in 3GL to 2GL form for side-by-side viewing during human verification, and run that through compiler/assembler that handles 2GL form. This is first compiler. It’s multi-stage so optimizations can be added in as intermediate steps in source in 3GL language. With first compiler done, now just add (or reject) those optimizations or security-relevant extras in source form then run through first compiler. Latest compiler to be used for most speed or security with first used for most trust.

    Now, most people say this just shifts the problem to a different source & optional, binary verification. It’s definitely different: verifying p-code, an Oberon subset, simple LISP, or ML subset instead of GCC + target program. A world of difference whether you’re talking formal verification a la CompCert or just spotting weird stuff in the assembly. One set of source files & a binary that many have looked at which is simple enough to be implemented by one person with about any tool on any platform. Precedent is P-code alone got Pascal/P on 70+ architectures in 2 years by non-compiler experts. Then, trusted signature and distribution of much simpler source files. The binaries get the benefits of site-specific optimizations, security enhancements, obfuscations, whatever since you trust and/or certify the compilation, not the binary. Which you should. If using a safe 3GL like Oberon or ML, your compiler is also more likely to be correct in general plus immune to certain classes of attacks. Kind of relevant given the threat profile.

    So, verifiable builds with safe, simple languages + optional microkernels hardware a la 1960’s-1980’s high-assurance vs reproducible builds with unsafe language, huge compilers, huge OS’s, and who knows what actual security. Even medium assurance done this way with functional languages, interface checks, QuickCheck, static analysis, and so on would probably be way better than status quo.

    Relevant links

    High assurance software design - nice intro http://web.cecs.pdx.edu/~hook/cs491sp08/AssuranceSp08.ppt

    Landwehr’s 1983 summary of high-assurance security: http://www.landwehr.org/1983-bats-ieee-computer-as.pdf

    Note: Advice for developer section with recommendations that ended up in Orange Book and Common Criteria are pretty accurate today. Keep getting rediscovered in various, alternative forms. Shows value of evidence-oriented approach high-assurance takes even if you only use a subset of its techniques.

    Wheeler on FOSS tools for high assurance http://www.dwheeler.com/essays/high-assurance-floss.html

    Certified compilation (HW w/ need certified “synthesis”) http://compcert.inria.fr/

    Original work on subversion http://csrc.nist.gov/publications/history/myer80.pdf

    Example of it in action http://www.cisr.us/downloads/theses/02thesis_anderson.pdf

    Wheeler on SCM security http://www.dwheeler.com/essays/scm-security.html

    Example of ASM to intermediates to full language project https://speakerdeck.com/nineties/creating-a-language-using-only-assembly-language

    Edison System on PDP-11 + other good, simplifying work here: http://brinch-hansen.net/papers/

    1. [Comment removed by author]

      1. 3

        “what do you need it for? perhaps you desire something specific that reproducible builds is not exactly aiming for.”

        Aside from debugging stuff, I thought the purpose was to increase assurance that one was using non-subverted software by ensuring everyone had same binaries from same source. Partly inspired by issues like Truecrypt source to binary mismatch that worried people. Essentially a security measure against subversion at either binary production or distribution phases. If that’s true, then my points stand.

        “If you don’t trust your system and compiler then a reproducible build at least lets you know that your compiler hasn’t targeted you specifically - you have the same build as everyone else.”

        Let me show you what reproducible builds do in practice. Using all risks in example, the reproducible build ensures the backdoored software containing extra 0-days at source level, leaks due to covert channels, and checks removed by optimizations is what everyone is getting in binary form. So, you at least have proof they’re all vulnerable to what was being ignored. I mean, if all that risk is accepted & constitutes 99% of compromises, then why put so much effort into the last phases to allegedly increase trustworthiness of the system? That’s what’s ridiculous about what effort is being performed and what’s consistently ignored for long time.

        You noted effort put into reproducible builds updating 23,269 packages. Whereas, Niklaus Wirth’s solution involved writing just two: a P-code interpreter and a back-end for P-code in compiler + stdlib (or equivalent). He compiled his compiler and everything else to P-code for distribution. The result was that amateurs in spare time, who just had to implement P-code + OS calls, ported it to 70+ architectures in two years. All those systems' users had the same software. I can’t overstate the combination of high impact & low labor. Likewise, all kinds of undergrads successfully implement Scheme interpreters or simple, imperative compilers as part of routine coursework. Academics like Wirth and Hansen built whole OS’s and stuff whose docs + source code in safer languages are available. Unlikely to be subverted, simpler than most UNIX/C stuff, language-level safety featuers, and easier to compile. Took 2 years with 1-2 people on average for such systems with more continuously appearing (eg JX OS). Bonus: A2 Bluebottle also runs lightening-fast in VirtualBox on a Core Duo 2 that was mid-range 8 years ago.

        Now, let’s say we’re adamant about doing it in C & for our C compilers. Well, it will take a while to show those C compilers function correctly & with no vulnerabilities as Csmith testing showed. Haha. Still, say you intend to fix all that incrementally with a C starting point. Then, my method dictates you really just need two projects: a simple interpreter/compiler (lang A); a simple, non-optimizing, C compiler (lang C) whose source was translated to that; optionally a third creating a standard C equivalent of your optimizing compiler if it uses non-standard C. User can build own lang A compiler as it’s close to P-code, LISP eval w/ macros, or something similarly tiny enough for one amateur. Then, they just run source of lang C compiler, ported to lang A form, through the lang A compiler/interpreter. Done. The resulting C compiler now compiles anything else including optimizing compilers like Clang or GCC. Anyone wanting same thing uses same compiler flags at each step in a chain. Those can be hashed & signed with likely standard configuration per release. You don’t care if they tampered with the binary because you generate it with trusted tool from trusted source via SCM security. You also get option of local optimizations for performance and/or security using source. Far as bootstrap tool, the ideal CompCert went commercial via AbsInt but QBE and TCC aim for simplicity. Might be usable for this.

        “ It doesn’t stop you using safe, simple languages or doing your own verifiable build stack. It only makes such a project more trustworthy.”

        Nah, your tooling from source to compilation to distribution already has to be trustworthy before a reprodicible, build process outputs something trustworthy. You certainly can use unsafe, incomprehinsible, buggy tooling supplied with reproducible binaries to build a subversion-free, correct system. Just adds a whole mess to the Trusted Computing Base (TCB) that your assurance argument & resulting security depend on. Given all the compiler bugs & complexity of GCC, I’d say you don’t have a believable argument at that point. You need something verifiable & easy to get correct first, then a safe way of building on it, and then trusted tooling/distribution built on that. That’s kernels, compilers, type systems, proof assistants, crypto… you name it all the same composition & verification requirement for trustworthiness. Empirical results from academic studies and field use of various software strongly supports that.

        “it only makes things like randomized testing more meaningful since we are all testing the same binary.”

        Those I mentioned all work on source, not binary, code. You can and should still test binaries, esp source-to-binary equivalence. Yet, the above scheme assures they’ll either be equal if relying on GCC’s or probably correct if relying on CompCert’s or FLINT’s. Plus more easily verified from the bottom up by users that aren’t domain experts if most optimizations are avoided in first compiler. Much stronger claim than you get right now.

        Note: The Redox team has a bootable, semi-usable OS with safe language, microkernel, and user-mode drivers. With similar effort in Oberon, they’d have well exceeded what’s in the requirements I stated given it’s a tiny fraction of their effort. But UNIX and C lovers hold tight to the root causes of their problems to point that they’ll put in endless work on all the symptoms instead of tiniest work on root causes. It’s psychological or social, not technical. ;)

        1. [Comment removed by author]

          1. 3

            “It really should be a deterministic process. We expect source => binary, not source + date + locale + … => binary. ”

            That much is true.

            “If I build the same program as someone else and there is a difference tooling can spot that and trigger an alert - something fishy going on.”

            I get the point of it. Mine is you still can’t trust the resulting binary. So, the focus should be on eliminating the risks in that process that get you more often. As a side effect, you don’t need reproducible builds in the process given you can trust a robust, verified compiler to produce a trustworthy binary. You can add reproducible builds for determinism or debugging, though. An example outside high-assurance, work like FLINT achieved close to what I described by leveraging safe language easy to use for compilers, type system, and multiple passes. Others went for simple, safe imperative languages. Morrisett’s Cyclone and others like it (eg Verisoft’s C0, Vault) modified C for safety for easier porting. The UNIX & C crowds aren’t doing much about majority of risk in their tooling but put high effort into projects like this with tiny, security gains. It’s a weird trade-off worth pointing out, esp if alternatives solve even more problems.

            “If somebody pulls a Strawhorse”

            If they can modify arbitrary apps on your system, then you’re already owned from a technical perspective. They can exfiltrate however they choose using overt or covert channels. I used to do it in TCP/IP headers & timing. Defense against app or system compromise requires host-level INFOSEC methods. Has nothing to do with reproducible builds. Also, avoiding such an attack requires security and/or containment of any applications processing outside data. That cries for a memory-safe, interface-safe language whose tooling is robust and handled in secure SCM system. C language & compilers haven’t made that cut since empirical studies from 90’s showed use of it doubled defect rate over Fotran, C++, or Ada. Esp severe ones.

            “ the whole ecosystem is built out of configure scripts and gcc”

            A lot of it is. It will be work to fix and should be fixed. If they stick with all that, then I’ll have to do a lot more work than reproducible builds to trust the resulting system. I still assume such boxes will be compromised & work around that.

            “Fixing up stuff like linux, openssl, openssh, nginx, apache as well as the toolchain itself gives real benefits today”

            That’s what I’m arguing for them to do instead of side projects like reproducible builds. You seem to want an approach that leaves in all the C, GCC optimization, etc risk in the equation for moderate to high attackers to hit but solves one, lesser risk. Let’s say I accept that. The Wheeler link showed SCM security requirements with Aegis as an example of key features & Shapiro’s OpenCM much better architecturally. These protect the build system in integrity & access control with Shapiro’s wisely using append-only storage. Other projects have distributed verification or made package managers easy to use for source-form apps. Improving and integrating such things then putting the above apps/kernels in it will ensure you have right source to build with plus make it easier to do that. What’s largely a SCM problem needs a secure, SCM solution. Otherwise, you got potentially insecure SCM and they still have to trust you did binary right without further verification.

            Verifiable builds + secure, usable SCM w/ distributed checking are the right answer here. Reproducible builds should be a low-priority addition to that considered after former solution is done and in use for critical projects. And only if it’s more security ROI than other possibilities like further code auditing, safe rewriting, and so on of projects you listed. I think it still wouldn’t be that important relative to other stuff on backlog as so much low-hanging fruit exists and is ignored in that ecosystem.

      2. 2

        I’m enthusiastic about this approach, personally, but a lot of people I respect are skeptical that it’s possible in a practical sense. Still, I’m glad to see someone else mention it, especially with so many citations.

        1. 4

          Far as practical, here’s you one more:


          That’s Bell of Bell-LaPadula describing Walker’s Computer Security Initiative from start to sad finish. The idea was computer security could become an engineered, commercial, & regulated good by the right combo of standards, evaluation, and incentives. So, they created those with incentive being DOD would only buy what was certified with higher security getting more money [in theory]. As Bell notes, it worked with both commercial developments happening and academic ones supporting it. Systems built or leveraging high-security tech included terminals, guards (firewalls' predecessors), databases, Internet stuff, email, messaging, IOMMUS’s (in 80’s), and so on. NSA intervention & DOD acquisition policies then killed the market for high-assurance security. Recently, we’ve seen the same model happen for safety via DO-178B with many high-quality offerings by commercial sector. Some proprietary companies still push it with niche products/services. So, it’s a proven model. Interestingly, the only one that didn’t deliver at all in high-assurance was FOSS despite the advantages of free labor & no certification cost. A few, high-security works were open-sourced after the fact or along the way by teams developing them but overall development model was same as proprietary with them usually academically funded by NSF, DARPA, etc. Like academics in early days of high-assurance.

          So, it can happen again. Trick is they have to sell a result, not security itself, for not much more than market price of normal stuff. A good example is with LOCK platform where high-security requirements only added around 35% percent to cost of development. Altran/Praxis Correct-by-Construction is another good one with 50% premium for nearly, defect-free software. So, one example would be applying this to a database system or key app that enterprises would buy for large $$$ but funneling a good chunk of money into assurance activities. Start with modular, medium-assurance app in safer language with checks on and extra hardware supplied to negate performance effect. Establishes baseline of expectations. Each year of sales supplies money to redo key parts of TCB like kernel, networking, filesystem, crypto, compiler, or whatever in high-assurance way with independent review. These might even be put into a non-profit to pool in resources from other companies that can benefit from them. This model, called “incremental assurance” in the 90’s, can combat both the problems of long-term software funding and the fact that most don’t care about security so much as paying for something that works.

          I’m not saying it’s easy, it will be majority, or anything like that. It’s just practical in the sense that it’s achievable within the framework of how businesses usually operate.

      3. 2

        CONFIG_PAX_LATENT_ENTROPY (Generate some entropy during boot and runtime)

        When this option is enabled, the generated binary code will contain some random bits generated by GCC at build time, as entropy. Enabling this option will lead to irreproducible builds. So the option should be DISABLED now.

        Beyond the reason given, the help text for this option explains that the entropy gathered in this way is not cryptographically secure, which seems like a much better reason to keep it disabled.

        That said, kudos to initiatives that promote use of grsecurity/pax.