Python [de]serialization IS in C. There are modules for JSON and whatever other common format you want written in C, but the C code still has to create Python objects for Python to use. Pretty much no one uses pure Python JSON libraries, because they’re so ludicrously slow. Parsing any substantial amount of data would be much much much slower than 10ms. I’m not even sure where you’d find one, you’d have to go out of your way to do so, considering even the standard library json module is written in C.

I don’t see JSON mentioned at all in the post as a bottleneck… unless it’s related to the 10ms Cassandra deserialization times…. And, granted, it’s totally possible given the space they are in. They likely get their feed items as JSON. It’s unlikely they actually need to keep it as JSON. Msgpack, or CBOR, or something would (maybe/likely) be faster to deal with than JSON. But, I digress. No idea if they tried.

All of those drivers are already written in C, the Cassandra one is written in Cython as you suggest.

Ah! Ok. I did not know this, thanks for the new context. It seems, however, that these optimizations could be turned off. Not likely in this case, I’m sure.

Like Python objects being… Python objects. Are they supposed to deserialize data into something else in Python? If you can’t deserialize to Python objects then why would you use Python?

Not a fan of your condescending tone. I, honestly, have no idea what’s going on in their application. I have no idea if they are making use of classes, or storing everything in tuples, or lists, or namedtuples, or some dicts, or some other concoction that exists in Python that I don’t know about (Pandas Data Frames?).

The bottleneck in the Cassandra deserialization could be due to the use of the Object Mapper. Did they try without it? Were they invoking some other objects whose init method happened to… I don’t know, accidentally hit disk to load a timezone file that wasn’t always cached?

You’re making a lot of assumptions, and I’m trying not to.

It’s true, they didn’t address those things, I expect because they assumed the reader would already know that serialization formats aren’t handled in pure Python. If you didn’t know that, well now you do.

Never assume the reader is as smart as you.

Let’s discuss more about the ranking / aggregation.

They build a twitter / facebook style feed as a service. The ranking seems to be an ordering thing (not shocking) and the aggregation a grouping thing (as opposed to syndication).

They spent 3 days + 2 weeks to optimize the ranking Python code. That’s not really a lot of time. They even dropped down into the AST module, which means they compiled into Python bytecode. The user can basically create whatever function they want based on a set of predefined primitives. Impressive ideas there, and probably the best you can get given the pure Python circumstances. What if it was written as a C (or Rust) extension? Could they have gotten 20x speed up in a few days? Did they try it? No idea. How long do they cache the ranking bytecode for? What’s the cost of their compilation? Does the Go version just walk a parse tree? Or does it do something even more fancy than the AST -> Bytecode compiler? No idea.

The aggregation related stuff is likely similar–based on some specification that the user provides (a Jinja2 style template), they do some analysis of the template and figure out how to do the aggregation based on the fields. Woah! Also an impressive thing. They even support conditionals. So, they might be using Jinja2 for the actual parsing and rebuilding. From my Python days, I know Jinja2’s parser doesn’t have to be that optimized – you “compile” a template once. The generation part is the part that needs to be fast, and I’m guessing they don’t suffer from slow text generation, but rather the actual filtering of objects, and finding and traversing the things that it all relies on. Is that something that they do every time? Do they cache the result of their compiler (would like to believe so!)? What are the real costs of it all? What differences exist in the Go code?

Don’t have much in the way of ideas here, but presumably the same thing would hold. They could spend some time optimizing just that part by dropping down into C, or Rust. That’s the likely bottleneck, not iterating over a bunch of objects.

So maybe they’ve tried all these things, and it ultimately wasn’t worth the maintenance costs, and the frustration, and things. It doesn’t appear to be the case based on the mention of the AST module that the ranking stuff was ever a C, or Rust thing. It’s likely that they really really tried hard and just kept coming up short. And, that’s fine. It wasn’t my call to make, or my happiness to deal with.

So, yes. Performance matters. And language performance matters… sometimes. And, developer productivity and working as a team together matter, so doing “clever” things is probably not desirable. There’s nothing surprising about their decisions, or reasoning… I’m just infinitely more interested in the stuff they didn’t say. And, I’m really curious about the trend we’re seeing where people suddenly care about efficiency and optimizing resources. The era of “scripting languages” as work horses (that started in the late 90s), is apparently dying, but I don’t know who wrote the blog post “Scripting languages considered harmful (and slow).” (BTW, I welcome this trend, but posit that scripting languages are probably good enough for most things, too)

It’s worth pointing out that “getting rid of the bottleneck” when your entire system is in Python isn’t obviously possible.

Of course! The whole reason this thread got out of hand is because it’s hard to understand exactly how Python can’t work here. We don’t know how much of the workload is I/O bound, and what part is compute. We don’t know what SLAs they target, and how far from them they are. We don’t know if the problems they face are only at peak times, or if this is constant.

I don’t mean to call out this post specifically, but this is exactly the type of post that leads to cargo culting in tech. Often, they are well intentioned posts (like this one), but they lack enough information to take it as anything more than anecdote, but instead some will ultimately treat it as justification to take a completely unrelated workload and say “Go is faster and we need to move to Go,” when Ruby, or PHP, or whatever else would continue serving them just fine and continue to provide them some of the advantages they adopted the language for in the first place.

I think I’d get it wrong too?

I guess my understanding is that a race condition is a general term (something which has two actors, and the sequencing of their operations matters) which includes a data race (the actors both happen to be modifying some data) is a specific instance of a race condition.

Is that about right, or am I missing some nuance in the terms?

This comment made me think; I think another thing that lets people get by is that a lot of practical concurrency is the simpler stuff: a worker pool splitting up a large chunk of work, or a thread-per-request application server where sharing in-memory data isn’t part of the design and most of the interesting sync bits occur off in database code somewhere.

Not that you’re guaranteed to get those right, or that advanced tools can’t help. (Globals in a simple net/http server can be dangerous!) Just, most of us aren’t writing databases or boundary-pushing high-perf apps where you have to be extra clever to minimize locking; that’s why what you might expect to become a complete, constant tire fire often works in practice.