fixed link

The non-syncing of spec code with implementation code really feels like the big barrier to making this usable in general.

One idea I had to tackle this issue in a language like Python would be to allow for executable doc-strings within the code that could let you write specs inline, and have those be parsed out (but by default it would use the actual in-code implementation)

That way you could write simplifying specs for certain parts of the code (say, the result of input will be any string instead of waiting on stdin when checking), while still avoiding duplication because most code is straightforward

Though to be honest this might be very hard to get right. I feel like it’s a bit like the ORM/Type System issue, where type systems are usually rigid and don’t give much “type-check-time” flexibility, but ORMs are usually defined dynamically (relative to the type system)

Human brains are really bad at reasoning about the correctness of concurrent systems. Even when we use files in simple ways we are usually skirting several race conditions that just happen not to pop up while running the code on a single laptop for a few days. After starting to write my networked code in ways that let me do randomized partition testing in-process, I’ve been so humbled by the dramatic number of bugs that pop out when you just start to impose possible-yet-unanticipated orders of execution on concurrent systems.

I’ve started to expand the approach to file interfaces that record writes since the last fsync, and can be “crashed” during tests at different points. Concurrent algorithms that can be deterministically replayed in any interleaving of cross-thread communication points. This has been fairly straightforward on systems that run on top of libpthread, but I’ve been struggling to apply these techniques to existing go codebases, where concurrency is not interactive from go programs themselves. External instrumentation of the process at runtime from ptrace gets gnarly really quickly.

I wish that as a language that encourages the usage of concurrency more than most others, go also embraced understanding its concurrent behavior more. Does anyone know of better options than using rr’s chaos mode on a go system after forcing all goroutines to LockOSThread?

This is pretty far off-topic, and most likely to result in a bunch of yelling back and forth between True Believers.

Flagged.

EDIT:

OP didn’t even bother to link to the claimed “increasing evidence”. This is a bait thread. Please don’t.

People don’t go nearly far enough with generative testing IMO. I have really great success while using it to test distributed systems by using a generator to seed clusters with client requests at specific time steps, and I also generate a schedule of network weather. For the client requests that receive responses, it’s often useful when testing consensus algorithms etc… to check that they linearize.

For concurrent systems, I generate small sets (2-4) operations against the full concurrent API and run them on multiple threads and record their return values. If it’s impossible to then find a single sequential schedule that has the same return values for the same requests by running different permutations of the concurrent ops on a single thread, then I just found a violation of atomicity.

For systems that write to disk, I have a file interface that when run in testing mode records a log of writes and fsync calls. I’ll generate operations against the API that persists data, and a crash time when writes are either partially applied or completely deleted after the last fsync call. Upon restart if the system is in an inconsistent state, then I just found incorrect usage of the filesystem (so many databases fail to do this correctly).

People tend to just test things that adhere to a function signature, and you get great results for very little effort by doing this, but there are mountains of gold if you get more creative with bigger and messier systems.

Have you looked at secure scuttlebutt or dat? They are currently being used successfully for helping periodically-connected communities exchange information asynchronously, like remote villages and sailors etc…

Hey icefall, one thing that would complement this presentation is a page listing every paper and tool in it with links to them.

It depends what you mean by resiliency. I tend to work on things with strong consistency requirements, and to be honest I think the way most people build and talk about distributed systems engineering is pretty gross and unprincipled.

Why is Jepsen so successful against most systems, despite it being so incredibly slow to actually exercise communication interleaving patterns? People are building systems from a fundamentally broken perspective where they are not actually considering the realistic conditions that their systems will be running in.

In my opinion, the proper response to this should be to ask how we can simulate realistic networks (and filesystems for that matter) on our laptops as quickly as possible, without requiring engineers to work with new tools.

My approach is to use quickcheck to generate partitions + client requests and implement participants as things that implement an interface that usually looks like:

• receive(at, from, msg) -> [(to, msg)]
• tick(at) -> [(to, msg)]

And this way a distributed algorithm can be single-stepped in accelerated time, and for each outbound message we use the current state of network weather to assign an arrival time / drop it. Stick it in a priority queue and iterate over this until no messages are in flight.

Then as the “property” in property testing, ensure that linearizability holds for all client requests.

With something like this, every engineer can get a few thousand jepsen-like runs in a couple seconds before even opening a pull request. They don’t have to use any tools other than their language’s standard test support. You can write the simulator once and it has very high reuse value, since everything just implements the interface you chose. Way higher bug:cpu cycle ratio than jepsen.

This does not replace jepsen, as jepsen is still important for catching end-to-end issues in an “as-deployed” configuration. But I really do think jepsen is being totally misused.

We should build things in a way that allows us to quickly, cheaply measure whether they will be successful in the actual environments that we expect them to perform in.

Maximize introspectability. Everything is broken to some extent, so be sympathetic to future selves that have to debug it in production while failing spectacularly and causing everyone to freak out.

One kind of concurrency that few seem to consider until it’s time to upgrade: multiple versions of your code may be live in some situations. Did you build your system in a way that ensures this is safe? Did you build your system in a way that allows you to un-deploy a new version if it fails unexpectedly? Or did you build in points of no return?

One reason why I don’t do distributed systems fault injection consulting anymore is because of the egos of people who can’t accept their babies have problems. That got tiring really quickly. The #1 most important thing to building a reliable system is being humble. Everything we do is broken. That’s OK. So many engineers who learn how to open sockets begin to think of themselves as infallible rockstars. It’s really hard to build systems that work with these people.

At first I was rolling my eyes but then realized I was being an elitist. Infrastructure must be sympathetic to its users. Part of that IS forward-compatibility, but security issues like this trump those concerns IMO.

This reminds me a bit of a technique Joe Armstrong mentioned, maybe in an in-person conversation at a conference or maybe more publically, I forget. He said that he will throw away code that takes him more than one day to write, and either start over the next day or do something more important. His justification was that if it takes longer, the approach is probably shit. At the time, I thought it was a pretty extreme approach, and just made a mental note of it and moved on.

After spending the last year+ making a reasonably well tested database, I’ve caught myself having taken on this approach myself, arriving at it after an endless series of devastatingly complicated bugs that have deeply humbled me. It was quite a surprise when I realized it was exactly what Joe had mentioned, that I didn’t really believe was an effective strategy at the time.

Why do this? For me, it keeps the complexity manageable. If I can’t keep the whole thing in my head when I’m creating it in one shot, there’s very little chance I’ll be able to debug an issue in it in under one or two days. I will create far more bugs when I can’t wield the model of it easily in my mind.

How long do you want to spend debugging an issue in a system? Combine this approach with bwk’s “debugging is twice as hard as writing a program in the first place” rule of thumb and throw the whole thing away when you get to 1/2 of your desired debugging budget!

Transactions for sled! I’ve been going through the literature and arriving at a concurrency control scheme loosely based on cicada but a little less complexity around timestamp synchronization, but giving it a clear path toward implementing less centralized timestamp allocation in the future if 100+ core machines become targets. It has been super interesting as a distributed systems person working on a main-memory database to see techniques from the distributed world being applied recently to reduce coordination costs of transaction techniques. I’ve been holding off on releasing sled as alpha because of a few known rare dataloss bugs, but I think it might be better to just roll forward and be clear that it’s got bugs still, which will be the case whether I know about them or not, because it’s still so young.

I’m working on two projects in my spare time right now:

-going through Type-Driven Development With Idris, to learn Idris for the dependent-types goodness

-trying to teach myself the nuts and bolts of type theory by implementing Hindley-Milner type checking in the toy programming language interpreter I’ve been working on for a while. I’ve found a few resources about exactly how to go about doing this (most notably, this Haskell conference talk and this section from Stephen Diehl’s incomplete tutorial on implementing a Haskell-like functional programming language. I’ve actually had a bit of trouble translating the code from those resources, which is in Haskell and assumes a particular design for the AST data structures, to my own interpreter, which is in Rust and has a different AST design. If anyone is knowledgeable about actually implementing Hindley-Milner in a real programming language, I’d love to get some advice.

I foresake C as someone who works with it all the time. The author of this post makes the point that it’s worth knowing. I think that’s totally true if you interact with low-level systems.

I definitely don’t buy the point about distributed systems usually requiring C because of performance reasons. As a distributed systems engineer, most of my peers work in memory safe languages, with a few things along data paths being in C, and every once in a while people may peer into the kernel to reason about an issue in the networking stack, but I’d imagine that most people who bill themselves as distributed systems engineers today are terrible at C and it probably doesn’t hurt them very much.

When I foresake C I don’t advocate for its ignorance. I advocate for learning all you can about memory corruption, and being honest as a biased human who is building things for other biased humans and with other biased humans. C is a great language to use for learning about bugs and exploitation techniques. There is too much macho bullshit from prominent C engineers, and it catches on with incompetent engineers who make the world a more dangerous place.

Many timeseries projects today seem to be borrowing techniques from Gorilla, particularly around compression, so it’s pretty relevant.

Thank you for the wonderful comments last week.

I wrote an Earley parser. And a Pratt parser. The Pratt parser is what I’ve been looking for all this time: a modular recursive descent parser. What it lacks in formalism it makes up with in brevity and power-to-weight.

Now, I need to choose a host language. I’d like to pick Rust, but I’m not sure it has a ready-made GC solution right now, and I don’t want to go down that rabbit hole. That leaves C++, JVM, or OTP. Any thoughts?

Rushing to get my lock-free rust bw-tree-backed embedded database to an alpha state before FOSDEM next weekend, where I hope to encourage a few people to give it a shot for low-impact workloads. Thinking about ways of applying real-time scheduling to threads in a concurrent property testing library I’m writing to tease out bugs in the bw tree to get similar results to PULSE when used with quviq’s quickcheck for erlang. I will name my first child “Determinism” after the past few months of intense debugging…

The recruiters who perform the first layer of filtering will usually have a set of buzzwords or technologies that they have been told are associated with candidates worth talking to. This might include things like prometheus, kubernetes, golang, aws, puppet, etc… If you are applying for a specific job, try to figure out what stack they use, and familiarize yourself with it by working through a tutorial, so that you can mention at least a basic level of familiarity with tech that the recruiters will be often filtering for. To the good companies that want SRE’s who are curious enough to dig into unfamiliar territory, being open about your previous unfamiliarity and willingness to dive in anyway can be a strong positive signal. But recruiters don’t always share the same cultural values around this as managers or future teammates, so use your discretion about how open you are about this with the recruiter. Some teams really value curiosity, but the recruiters often don’t get that message, and will hear that you’ve only done a quick tutorial on something and will mash the “reject” button in their candidate pipeline management interface.

When I hire for SRE teams, I care about one thing above all else: ability to drill into unfamiliar systems that are behaving strangely. General curiosity and demonstrations of learning new things are possible indicators of this mindset to me. I actually usually prefer to hire people who are more curious than people who are less curious and more experienced because the curious people will be getting more out of the job on a personal level, and I love teaching. A lot of teams don’t prioritize teaching though, due to mismanagement, incompetence, laziness, or fear-driven mindsets that are afraid a newcomer will outperform them. Avoid these places like the plague. They are the norm, unfortunately, but they will not help you grow at a rapid pace, and if you can afford to keep playing the interview lottery, you should really hold off until you get a place that gives you a strong signal about mentorship.

I test for drilling proficiency by asking questions about tools that are common, and can be assumed to have some basic familiarity around, like linux or the web, and when we get to a part that they are unfamiliar with, I let them know that they can use me as Google to ask questions about how they can drill into specific behaviors. I ask about how things can fail. What are different ways a file can fail to be created, etc… (they can still use me as Google to look into the specific implementation details of things, or we can strace stuff together, etc…). Basically, I try to show them windows they can use to peer into things I’m asking about the system if they are not already familiar, and then I try to get a sense of how creatively they can use these new tools I’m giving them. That’s how the job will look, so that’s how I try to make the interview.

Most people suck at interviewing. They will ask you about minutae about tools that you may not be experienced with yet. It’s important to keep a calm head, let them know about where your current level of experience is with these specific technologies, and then explain to them how, in the real world, you would dive into the problem to understand and solve it. You exist to solve problems, deflect bad interview trivia questions with confidence in your ability to solve problems by drilling into them. If the team sucks at interviewing, that’s on them, and the team is less likely to be made up of experienced people who are also good to work with. People skills are the most important engineering skills.

If you want a laundry list of tech to get familiar with for modern SRE-like jobs:

• linux fundamentals, what’s an inode, what’s a process, what’s a socket
• AWS / GCE
• kubernetes
• prometheus

Books:

• http://www.brendangregg.com/sysperfbook.html
• https://landing.google.com/sre/book.html

Different teams will prioritize coding skills differently. The most common thing I see recruiters filtering for is making sure you have one scripting and one compiled language under your belt. Golang is a good bet, as it’s taken over the pop infrastructure world.

Have fun :)

Raw sockets are the strcpy of the current age of terribly-written distributed systems. People stand no chance of finding bugs in the systems they create on their own laptops. The fact that slow, incredibly low bug:cpu cycle black-box partition testing is so successful in finding bugs should be a screaming red alarm that we need to write our systems in ways that are more amenable to testing on our laptops in the first place.

That means having a pluggable transport layer that can be used with real sockets in production, a simulator that messes up traffic according to the asynchronous network model, or a hit-tracing fuzzer. If you’re using raw sockets in your core serving logic, your architecture is broken, and you will have far more bugs and higher costs to fix the few bugs you discover.

I recommend looking for a University that does it to learn or work on one of their projects. I suspect it’s very helpful to have experienced people to help you through your first year or two of verifying anything significant.

In any case, here’s a write-up from one of best in field with advice and one book reference. The other books people mention are Certified Programming with Dependent Types by Chlipala and Software Foundations. If picking based on tooling, Coq and HOL (esp Isabelle) are used on best projects in software with ACL2 being used most for hardware.

It also helps to see what lightweight verification is like if you need some motivation, a fallback, or just tell you something aint worth proving. Alloy (see site) or TLA+ (learntla.com) are best for that imho.

Making lots of progress toward stabilizing my rust lock-free bw-tree! Hopefully I’ll have an alpha out soon :) The goal for the next week is to have stable disk utilization while under extreme contention and fragmentation. Now that crash testing and interleaving tests have teased out much of the low-hanging fruit, it will soon be time to turn my attention to dynamic instrumentation for teasing out exotic race conditions :] If anyone is curious about lock-free high performance stateful systems in rust, feel free to ping me, and I’d love to spend some time teaching potential collaborators. There are a ton of really juicy database features that I’d love to delegate to other people who are curious about learning how to build them!