1. 8

    SSL libraries aren’t bananas, though, and I think this is where the argument falls down. For instance, SSL libraries don’t have immune systems – if a vulnerability exists in one instance, it exists in all of them.

    Yes, having fewer implementations means that vulnerabilities affect more users – but, it also means that the vulnerabilities are easier to fix, and more effort is concentrated in the implementations that exist.

    To put it another way: if we’re not (collectively) capable of writing one correct implementation, why would it be easier to write three, or five, or ten, correct implementations? The problem isn’t diversity, it’s the fact that we don’t know how to write a correct SSL implementation.

    (Moreover: I would speculate that we don’t yet have the tools to do so – formally verified systems seem too cumbersome to use, C is too unsafe, and Haskell doesn’t seem to allow control of resources sufficient to prevent information leaks).

    EDIT: added the word “yet” to the previous sentence: “we don’t” -> “we don’t yet

    1. 5

      formally verified systems seem too cumbersome to use

      This was posted on Lobste.rs about a month ago, and seems more relevant now:


      1. 4

        Not all the problems are technical. The way in which the organization behind the project is run probably matters more than any given piece of code, and that is closer to being ‘like bananas’ (i.e. we get benefits from diversity). I’ve a blogpost detailing some thoughts about that here: http://rz.github.io/articles/2014/apr/security-orgs.html

        formally verified systems seem too cumbersome

        fwiw, it is not impossible: see https://github.com/vincenthz/hs-tls

        1. 3

          Is hs-tls actually formally verified, though? It looks like it’s just written in Haskell, which is a big bonus on ‘free theorems’, but falls short of formal verification. Moreover, I’d be afraid of information leaks via timing, etc., as noted.

        2. 2

          SSL libraries don’t have immune systems

          Use of valgrind and automatic methods for catching use after free are somewhat like immune systems. Sophisticated type systems, design by contract, or comprehensive unit tests are somewhat like immune systems.

          To put it another way: if we’re not (collectively) capable of writing one correct implementation, why would it be easier to write three, or five, or ten, correct implementations?

          That’s not the point. The point is that the various systems aren’t all vulnerable to the same single bug all at the same time, or at least to minimize the chances of that happening.

          1. 2

            As a user of an SSL implementation, though, it doesn’t really matter to me whether I’m vulnerable to some particular bug – it matters whether there is a vulnerability.

            Having everyone be vulnerable to different bugs instead of the same bug isn’t really much of a benefit – all the users are still vulnerable, just in different ways. It’s only a benefit if there are fewer bugs, and I don’t think that more implementations in total reduce the number of bugs in each implementation.

          2. 1

            Some formal verification can be annoying but I definitely think we should still try at least some. Someone here in Boulder has recently started a very exciting Idris library for this:


            1. 1

              Yes, having fewer implementations means that vulnerabilities affect more users – but, it also means that the vulnerabilities are easier to fix

              Why is this the case?

            1. 6

              In my experience people tend to be very polarized on the topic of testing. There are die-hard TDD fans and people who think of TDD as a waste of time. Ditto for integration testing and other kinds of testing. However, it seems to be relatively uncontroversial that a mature-enough or large-enough project needs testing.

              The article does a good job of highlighting one of the problems with testing in django that is probably more general: django makes it easy to be unclear about the kind of test one is writing and provides tools that encourage writing integration / end-to-end tests that input http requests and assert on responses. Writing good integration tests is much more difficult than writing good unit tests, though, and so these tests often end up being brittle and burdensome without much gain.

              Prompted by http://blog.testdouble.com/posts/2014-01-25-the-failures-of-intro-to-tdd.html and associated discussion I stumbled upon https://www.destroyallsoftware.com/. Just about every screencast is worth watching a couple of times. The screencasts really made it click for me in terms of how TDD is supposed to be used, highlighted the utility of different types of testing, and gave me a way to think about the tradeoffs of having/not-having different types of tests and such.

              1. 1

                This looks really cool. I’ve wished for something like this to exist for python for a long, long time. At the same time, though, a lot of architectural benefits come from elevating front-end code from being an afterthought of the backend application to being a first-class citizen, an application on its own. In a nutshell, separating the two and having them talk over an API allows the developers to code using the strengths of the front and backend environments instead of making concessions to find the common ground.

                This approach, provided it matures enough, will probably lend itself very nicely to rapid-prototyping so long the migration from the prototype to the more mature code can be painless and gradual (e.g. the way you can slowly strip out django component by component until you are no longer using django).

                1. 2

                  Sendgrid is, no doubt, at fault here. However, isn’t the real security hole the wide-spread practice of sending password reset links over email? Pragmatism and practicalities aside, a determined attacker could just snoop emails without gaining control of the send grid account. Or am I missing something…?

                  1. 3

                    How else would you reset a password when you can’t validate the user?

                    1. 3

                      There are several services for which I would love the option to disable password resets. I promise to never, ever, ever lose my password and in return you promise to never allow some bumpkin to reset it.

                      1. 2

                        well, the increasingly used alternative is to use a phone in some way (and that’s what happened here, although it saved the day via two factor auth rather than as part of the password change).

                        google keeps pestering me to confirm my cell phone. do they use it to confirm password changes? i guess i should stop ignoring them.

                        do third party services exist that make incorporating cell phones into password resets practical for smaller sites?

                        1. 3

                          Yes, Google will send you an SMS to confirm password changes with a multi-digit code. (In other words, multi-factor auth).

                          And yes, there are services out there doing just that. I don’t recall any names, and I’m not going to shill for them. However I will say it’s pretty easy to write your own with a platform such as Twilio.

                          1. 1

                            thanks; i just enabled two factor auth with google. it works well and seems to be nicely thought-out (including emergency backup codes and separate per-application passwords for places where sms won’t work).

                            [actually, what i was asking, although i wasn’t clear, is whether they use the phone when you change passwords, even if you’ve not enabled it otherwise.]

                            1. 2

                              It seems well thought-out, until you realize that there’s nothing application-specific about “application passwords” and that if one of those is compromised, an attacker avoid all the security provided by 2fa.

                    1. 3

                      I’ve always thought that the reason that you just don’t do search on your database is that the performance hit is terrible. Not only because of the search being slow, but because it impacts the rest of the system. In that context, platitudes like “The full-text search feature included in Posgres is awesome and quite fast (enough).” are not very useful. It seems no surprise to me that postgres has some sweet search features (and probably has had them for a while), but the performance thing is generally a show-stopper for anything in production.

                      Does anyone know of some benchmarking or quantitative performance data with a real production system using postgres for search?

                      1. 7

                        No intentional bench marks - just personal experience.. Using tsvector and trigrams I was able to reduce query times from ~18 seconds to ~ .5 seconds for fairly complex queries of multi-million row databases.

                        Another nifty feature is the highlighting / headline stuff!

                        1. 2

                          Thanks! This is a useful data point. I think I’ll try to move all my search needs into postgres next time i’m working on that.

                        2. 1

                          My understanding is that the indexing postgres does for FTS is relatively quick and not a major perf hit in production. The problem I’ve had with using it over, eg. ES is more to do with features – in particular it has to do with things like Faceting, custom ranking constructions, etc.

                          It may be also that – since FTS does get used more often than other components, it could be detrimental to have it running on the same hardware as the rest of your system, though that strikes me as solvable-by-wallet rather than necessarily solvable-by-developer. That distinction might be appealing to some folks who don’t like having a preponderance of tools. Personally, I don’t mind it so much.

                        1. 5

                          Invariably in a post like this there are at least a few lessons. To my mind, the most interesting one is how by working on X (coffee ordering system) you can get to product Y (group small-debt tracking), where Y is very different from X, but something people want ™. As is well known in startupland, this happens with products quite a bit. But I’ve also seen it happen with features. A developer’s mind is trained to have a bad reaction to feature creep, but when you don’t know what the feature or product should be feature creep is almost desirable – from a product perspective, not a software one, as the software will probably be worse with the feature creep.

                          1. 3

                            The article is spot on. What seems to have happened with text editors in the past few years is that we’ve traded off a lot of power in favor of easier learning curves and UI. The challenge for the next big deal text editor is to achieve both of those two things without trading off too much power.

                            1. 4

                              Maybe there’s no magic, but time is finite.

                              In the context of programming, things get called ‘magic’ when the amount of time it takes to understand what they do and how is large because of how complex (or worse, complicated) they are.

                              1. 1

                                sometimes “magic” is just a convenience wrapper. Hence: virtualenvwrapper!

                                1. 1

                                  Exactly. If you offer an abstraction there are two choices of investment a user may make: learning enough to use it to get their job done and learning enough to understand how it was built. Magic occurs when the first investment is significantly lower than the second.

                                  In a world of leaky abstractions, it’s often useful to spend enough to get to the second level. If a job changes significantly enough that the leaky abstraction no longer well-solves it then knowing how it was originally built will provide relevant information for the abstraction user to either extend to rebuild the relevant structure to solve the new problem.

                                  In this way, “magic” programming—again, programming where the second investment barrier is much higher than the first—can produce a large group of programmers who are effective in some parts of a problem space but very ineffective outside of that subspace.

                                1. 4

                                  Sorry, no. I don’t do 24-hour homework assignments on the chance that I might get an interview.

                                  I used to send companies like the above a quote for my services when asked to perform such things (much to their chagrine), but sadly I have more important things to do these days.

                                  1. 2

                                    The fallacy here is thinking that all jobs are the same. The reason you want to devote a fair bit of time (e.g. 3 days) to figuring out whether you should work for a given company is that if you aren’t working the best job possible (which very well may be your own thing) you are mis-investing your time. So, while you should only do something like a 24-hour interview with a company that you’re already excited about, to veto such a processes a-priori is only to deny yourself information that is very relevant to the decision. That you happen to give them information about you during that process is incidental (from your perspective). Furthermore, should you be working for a company that does not put adequate effort into deciding who they hire?