1. 8

    This is another fantastic article about distributed systems testing by @aphyr. I have been really glad to see that he’s been breaking down how to actually use Jepsen–perhaps we’ll start seeing more and more people applying Jepsen to their projects.

    With respect to etcd in particular, this has finally pushed me over the edge to the point where I’d consider building a system with etcd, rather than only considering ZK.

    1. 7

      I would actually hold off if safety is critical. Given how ZK, Doozer, Chubby, etc went, it’ll be another five years or so before they iron out all the kinks, haha.

      1. 1

        Yeah I was gonna say, this seemed to me to argue that sticking with Zookeeper for now is probably the right choice.

        One relevant question for @aphyr: IIRC your original ZK article was with an old version of Jepsen without the linearizability checker, have you tested ZK with knossos at all?

        1. 16

          Not yet, no. Each post is between 50-100 hours of work, and this is all nights+weekends, so it takes a while.

          1. 4

            Yeah I understand, just curious. Thank you for doing these, we all appreciate it :)

            1. 1

              Is there any way I can tip you? A paypal account maybe? I have learnt a lot about distributed systems from your blog. I want to show my appreciation by sending some funds.

        1. 1

          I would go with “OpenBSD will help leverage the synergies.”

          On a more serious note.. You could show them cases where OpenBSD was OOTB more secure than $linuxdistro. I can’t remember specifics ( read: my first google search didn’t find it ), but IIRC there was an example of this with Apache ( maybe 1.3, feels like it was in base ) a while back.

          1. 1


            This seems to be a good comparison.

          1. 5

            Disclaimer: I run an OpenBSD consulting firm

            One problem many companies have with open source is that there’s no “downhill” option. If something breaks, there’s no one they can call for support and pass the buck to and tell everyone that the vendor is working on it. With Cisco and Micros~1 you can get (expensive) support contracts and always have someone available to answer questions (even if that answer is often “it’s a bug, we’ll fix it in 9 months”).

            Even if you’re personally capable of supporting these OpenBSD servers, it might help you make your case if you can get one of the OpenBSD consulting companies (not mine) to prepare a quote for supporting your servers. Even if you never have to use them, management might be more receptive to the idea if they know they can call someone in your absence or think that you have someone to call.

            1. 1

              I’ve already included that in the proposal. I am leaning towards M:Tier for support and I want the network architecture to be reviewed by Henning.

              I’m sure I’m not competent enough to troubleshoot serious issues – I may be able to do day to day monitoring and restart processes when they crash, or do some minor maintenance like rotate logs but I’ll have to delegate serious issues to someone else – someone who knows the system well.

            1. 2

              So, what’s the way out of this situation we find ourselves in? Is there really no way we can have any privacy anymore?

              Its incredibly sad to see services we love – lavabit, groklaw – shut down this way.

              1. 5

                “If you choose to live in fear, you’re letting the terrorists win” or something like that?

                What do you get when the headline of a newspaper is about a suicide? More suicides. What do you get when the front page of every tech news site is dominated by stories about sites shutting down? Chilling effects, indeed.

                If hysterical overreaction isn’t the appropriate response to terrorism, it doesn’t seem the appropriate response to government abuse either.

                1. 1

                  We need to change the political system, however, I’m not sure the best course of action or what should be changed.

                  What we all can do, as engineers, is devise a better protocol than XMPP / SMTP that is encrypted and secure by default.

                1. 1

                  Hello! Thanks to @conroy for the invite over at HN!