One thing that’s rarely mentioned in this context is capacity planning.
With single-threaded server you have pretty good idea about its runtime behaviour: it scales with increasing traffic until it hits 100% utilisation on a single core. Want to fully utilise 24 core machine? Run 24 instances of it.
With multi-threaded server the capacity planning becomes black magic. 1 instance may be able to utilise all 24 cores. Or it may take advantege of at most 2 cores. Or 4.75 cores. Even worse, the scaling is rarely linear. In short, to understand how the server scales you have to be familiar with the internal working of the server. And of course, any change to the server’s threading model can throw you capacity plan into disarray.
If you wrote your threaded server in the same way as your event based server, you wouldn’t have capacity planning problems. All those scaling issues come from thread synchronization, which you must necessarily eliminate in event-based servers for them to even work.
If you are doing capacity planning you are unlikely to be the same person that wrote the code. Even worse, you are probably planning for multiple different servers, each written by a different developer, having different scaling constraints. Now what?
I think you’ve touched on why the programmer needs to be involved in the whole system: The hardware, the software, the network infrastructure, and even the out-of-hours response process.
Where the programmer is removed from the reality where their software is running on hardware, and that hardware is made out of matter and powered by energy, then you’re going to continue to find efficiency outside the bailiwick of programming.
The most stable systems I’ve worked with were those where the programmers were also the ones responsible for the production uptime (capacity planning, monitoring, on-call) of their systems. This works well for companies running services, but is hard to do in a product company with tons of customers who run the system on their own. In product companies the buck gets passed around a lot more and a dedicated QA team may be necessary to find problems, since engineers will often fail to find problems that arise in the fault space not accounted for in their local dev environment (they may write code that passes tons of unit tests, but as soon as ZK goes down their stuff ungracefully amplifies unavailability even though maybe using cached configuration would be totally reasonable, etc…)
Now you capacity plan by increasing simulated load against the service until a component fails, which was the first step in your plan no matter how the code was written anyway. What was your other plan? Guess?
On what kind of machine would you run the load test? How many cores? Would it behave differently on a different machine? Hard to tell. With a single threaded server it’s pretty straightforward.
Wait what? Are you saying you capacity plan by running your single threaded server on the dev box and hoping for the best?
This should probably include “(2008)” in the title, it’s old enough that “the old way” has been new and then old again, again.
Perhaps you have already done this but you can suggest titles for the story. Here is a direct link.
Writing straight-line code vs callbacks, listenable promises, or other eventing models seems like a no brainer to me. It’s a shame that OS threads are so heavy. If you need a higher number of connections per server, I’d switch to a language runtime with lightweight threads long before even considering manually applying a continuation passing transform to my code!
Somewhere along the line, someone got “scalable” and “fast” mixed up – and it stuck
Seriously. You only need to drop threads if you need more connections than you can have threads. In this day and age, you will only have more connections than threads if the average work per connection is none. In other words, nginx / HAProxy / stud type proxy software, or other fringe cases, not application code.
I had missed at first that this was from 2008. I wonder what the current state of these approaches in Java is?
Java is about at the level of Node.js, I would say. Generally people use Futures/promises (and there are like 3 - 4 different kinds of Future’s for some reason) and you have to do all these transform callbacks (which is made slightly better with Java8 and lambdas.
“Much like generics you can’t avoid it” was thrown into a slide I guess as a given but I’m curious what the support is for this statement. DRY?