After having discovered these primitives in Elixir, it’s weird to go back to other languages where they are not a thing.
These days, I work in a complex python codebase that spins a dozen of thread for various things. Each of these threads have their own “when to stop” logic, making it hard to gracefully handle ctrl-c. Communication is done by method access which edit a shared memory, meaning that sometimes error happen when locking is insufficient (when a list get edited while it’s being looped on by another thread). And most importantly, the lack of supervision makes it hard to understand what is running and even harder to do something when one thread dies because of an error.
I find myself often googling for some form of these Erlang primitives in python, without success so far.
I wrote a lot of web scrapers in Python. Tedious and error prone stuff dealing with threads and failure.
Then I wrote one scraper in Elixir and it amazed me. Controlling a pool of clients, handling failure in connections, resizing the pool, balancing work between workers, running Async tasks.
At best one might be able to claim that lightweight processes and supervisors are the key mechanisms at play3, but I think it would be more honest to recognise the structure that behaviours provide and how that ultimately leads to reliable software.
I think there’s a little bit more to it - specifically a runtime for managing links and other special features like links. But otherwise I think this really is a case of “Erlang’s foundational primitives let you take something fairly simple like an interface and leverage that in a way more powerful way”. The fundamental idea is really that isolated processes and message passing are the primitives you need for reliable systems. When you layer things on top of that that would otherwise be “good patterns” in other languages you get a lot of stuff “for free” in Erlang. Plus its syntax (especially the multiple handlers + destructuring in function overloads) makes it really easy to express states across behaviors.
So basically I maybe disagree a bit with the title but I thought this was a very nice writeup.
I would suggest that you check out the P programming language, I think you’d enjoy it.
So basically I maybe disagree a bit with the title but I thought this was a very nice writeup.
I chose the title partly because I don’t think you actually need lightweight processes to implement behaviours (I sketch an alternative towards the end of the post).
Another reason is that I feel a lot of people associate Erlang with lightweight processes and message passing already, and in order to “make room” for the new association of Erlang/OTP and behaviours it perhaps makes sense to weaken the existing association first.
Finally, I wanted to create some tension to lure the reader into reading the introduction (I’m not sure if this is a good tactic or if it worked).
I also considered something along the lines of: “explicit processes and message passing is the goto of concurrent programming” (as opposed to the structured approach that behaviours encourage).
I would suggest that you check out the P programming language, I think you’d enjoy it.
I enjoyed your article. I also liked seeing Jim Gray in your links on the bottom. To build on some things, here’s three gifts that you might enjoy (all PDF’s):
An Architectural Overview of QNX which describes a microkernel with lightweight processing, message passing, and high reliability. That’s been powering all kinds of things for decades. Probably lessons to learn from it.
Microreboots applied the restart concept a bit further in a Java stack. The paper does a good job of covering the many failures and components one might want to consider.
After having discovered these primitives in Elixir, it’s weird to go back to other languages where they are not a thing.
These days, I work in a complex python codebase that spins a dozen of thread for various things. Each of these threads have their own “when to stop” logic, making it hard to gracefully handle ctrl-c. Communication is done by method access which edit a shared memory, meaning that sometimes error happen when locking is insufficient (when a list get edited while it’s being looped on by another thread). And most importantly, the lack of supervision makes it hard to understand what is running and even harder to do something when one thread dies because of an error.
I find myself often googling for some form of these Erlang primitives in python, without success so far.
I wrote a lot of web scrapers in Python. Tedious and error prone stuff dealing with threads and failure.
Then I wrote one scraper in Elixir and it amazed me. Controlling a pool of clients, handling failure in connections, resizing the pool, balancing work between workers, running Async tasks.
Have you tried writing Python code structured-concurrency-style using e.g. trio with httpx? It brings back a lot of sanity into async Python.
Comment from the Erlang creator staging that permissions of this article is actually not true
It’s also about treating failures uniformly. Hardware failures show up as software failures. This is in Joes thesis somewhere also.
I think there’s a little bit more to it - specifically a runtime for managing links and other special features like links. But otherwise I think this really is a case of “Erlang’s foundational primitives let you take something fairly simple like an interface and leverage that in a way more powerful way”. The fundamental idea is really that isolated processes and message passing are the primitives you need for reliable systems. When you layer things on top of that that would otherwise be “good patterns” in other languages you get a lot of stuff “for free” in Erlang. Plus its syntax (especially the multiple handlers + destructuring in function overloads) makes it really easy to express states across behaviors.
So basically I maybe disagree a bit with the title but I thought this was a very nice writeup.
I would suggest that you check out the P programming language, I think you’d enjoy it.
I chose the title partly because I don’t think you actually need lightweight processes to implement behaviours (I sketch an alternative towards the end of the post).
Another reason is that I feel a lot of people associate Erlang with lightweight processes and message passing already, and in order to “make room” for the new association of Erlang/OTP and behaviours it perhaps makes sense to weaken the existing association first.
Finally, I wanted to create some tension to lure the reader into reading the introduction (I’m not sure if this is a good tactic or if it worked).
I also considered something along the lines of: “explicit processes and message passing is the goto of concurrent programming” (as opposed to the structured approach that behaviours encourage).
Yeah, P’s cool.
I enjoyed your article. I also liked seeing Jim Gray in your links on the bottom. To build on some things, here’s three gifts that you might enjoy (all PDF’s):
Fault-Tolerance in Computer Systems which thoroughly describes the architecture Gray et al built.
An Architectural Overview of QNX which describes a microkernel with lightweight processing, message passing, and high reliability. That’s been powering all kinds of things for decades. Probably lessons to learn from it.
Microreboots applied the restart concept a bit further in a Java stack. The paper does a good job of covering the many failures and components one might want to consider.
Please don’t do this.