I don’t know much about Erlang but I’m curious. What measures are taken to get a million processes to run together?
Do you just have a task pointer that iterates between all one million processes? Does each process have a pointer to the processes it calls? Is there any effort to avoid cache hits? What happens if some of the 1.5 gb of processes winds-up cached-out by the OS or is some effort made to avoid this?
What measures are taken to get a million processes to run together? Do you just have a task pointer that iterates between all one million processes?
An Erlang “process” is not a thread or a process in the operating-system sense; they’re structs with a chunk of heap, a pointer to some code, etc. These processes are run on a set of schedulers (in the modern SMP BEAM), often mapped 1:1 with operating system thread; each scheduler maintains a queue of processes waiting to execute. Schedulers may steal processes from one another if work is unbalanced.
Does each process have a pointer to the processes it calls?
There aren’t any inter-process “calls”, exactly - processes communicate via asynchronous message passing. Each process has a “mailbox” of messages to process and can process them as it pleases. You can imagine that it’s pretty trivial to implement an RPC-like mechanism on top of such a system.
Is there any effort to avoid cache hits? What happens if some of the 1.5 gb of processes winds-up cached-out by the OS or is some effort made to avoid this?
I’m not totally sure what you’re asking here. If you’re asking if the BEAM actively attempts to manipulate the operating system’s page cache, I’m pretty sure the answer is no.
I once asked whether Apache Storm might be considered “language independent Erlang BEAM/OTP”.
I didn’t know enough about Erlang to answer the question, though I know Storm pretty well. Some friends who had coded Erlang awhile didn’t think there was much of a relationship, since Storm doesn’t offer a functional programming model.
But this mailing list post does make me think there are similarities. Storm tries to offer a model for reliable message passing with fail fast, automatic retries, and horizontal scalability at the process level. It does this atop the JVM by spinning up LMAX Disruptors and Netty async event loops, and bridges outside the JVM with a JSON protocol using UNIX process I/O. So it may not be as performant in terms of computation per node, but it seems operationally and architecturally similar.