The diagram of “you can do everything in Erlang” really stood out to me. If true, this seems incredible. Does anyone actually write production Erlang systems this way e.g. writing data persistence in their Erlang application rather than delegating to Postgres? That sounds really scary.
Technically, Erlang can do persistable data via DETS (disk-based Erlang term storage) or Mnesia (a database written on top of ETS/DETS). In practice, I’ve heard this approach used much more as a control-plane database than as a data-plane, high-throughput DB. Specializing to an outboard DB like Postgres or MongoDB is much more common than persisting data within Erlang.
For ephemeral data, though, Erlang ETS offers some compelling advantages to Redis if you don’t want a network hop between you and the data you seek, and can pay the cost of replicating some of the same data across server nodes. Performance is good in control- and data-plane applications.
But these days, you often end up bringing Redis in anyway, as a backing store for Phoenix.PubSub when you don’t want to cluster your Erlang servers together (which is a whole can of worms in itself).
The rest of the diagram? More or less true as-is. Erlang/OTP is magic, a real batteries-included toolbox for soft-realtime distributed systems.
Klarna was storing their data in a large Mnesia cluster. They migrated to Postgres in the end, but that was only recently. It did save them a lot of hassle after the long migration was done. (I hope I remembered correctly from the talk at the Amsterdam meetup).
The last thing that I ran with Erlang in production was eJabberd. It had a bunch of options for the data storage but I used mnesia because I’m lazy. It only had about a hundred users (probably an average of 20 active ones, down to one by the time I gave up on XMPP), but generally sat happily at <1% of CPU load on a single-core 1 GHz machine. I think it used well under 128 MiB of RAM (the machine had 1 GiB, but eJabberd was a small part of what it did).
writing data persistence in their Erlang application rather than delegating to Postgres? That sounds really scary.
Why do you find it scary?
No offense to you, but I think part of what makes a lot of people weirded out by Erlang/Elixir–and Lord knows during peak refugee migration from Rails–is that it exposes competent, low-level tooling to solve problems like durability without needing to pull in other stuff and cargo-cult “best practices”.
Truly bizarre and ancient magicks (like matchspecs) can let you do a whole lot of stuff right out of the box.
There’s a lot of comfort in having your app servers be stateless, and pushing mutable state into a particular system designed specifically for that. It would scare me to have to think about disk state on my app servers again – that’s 20-years-ago stuff. (DHH is not inviting me to Thanksgiving.)
That’s what you get with Erlang and
Mnesia. The main difference is that you’re using BEAM for (very) lightweight process isolation, rather than the MMU for heavy OS process isolation.
Every Erlang process can be an independent failure domain. You structure your code so that the app server runs as a set of Erlang processes with monitors that restart them in case of failure. These push persistent state onto a system designed for it: Mnesia. That pushing is very cheap (ref counted immutable data within the same OS process), and Mnesia then handles pushing it to disk. You can run Mnesia in a separate BEAM VM if you want to (on another machine, if you want) and use the distributed Erlang bits to talk to it and then the overheads are closer to talking to an RDBMS over a socket.
It’s not the app-level logic that I’d be afraid of. Far from it, it’s as easy as using an external DB. The ops-level simplicity I’d be giving up, though, if I had to care about when every single app server dies and what happens to its disks? That gives me pause.
And at the point you’re shoving persistent state into a smaller pool of machines, IMO you are probably better served using a DB that is itself the main product, and not a side feature of a different product. Or go one step further, and use a managed DB like RDS Postgres, and pay for a robot to worry about it.
The ops-level simplicity I’d be giving up, though, if I had to care about when every single app server dies and what happens to its disks? That gives me pause.
What do you do when the disks in your RDBMS host die? One of:
Cry
Restore from backups and lose recent data.
Shrug and online a new replica to replace the one that died
Mnesia supports all of these models. You can persist data to a single disk. You can (separately) back that up. Or you can build live replicas. Like everything else in Erlang, Mnesia is a distributed system and so you can run nodes in different failure domains for redundancy. For small deployments, you can run it on the same machine (and in the same OS process) as your app logic. For a larger deployment, you can run live replicas with dedicated machines running Mnesia and the others running the app logic. Oh, and you can put some RAM-only replicas on the same nodes as your app logic for fast reads.
Erland and Mnesia were designed for telecoms systems, where five nines of uptime is considered unacceptably unreliable. You almost certainly don’t have the same reliability requirements as a major telco.
What do you do when the disks in your RDBMS host die?
Not a goddamned thing! RDS Postgres with automatic failover: pay 2x to have a replica you can’t touch, but that replica can take over in a couple of seconds following a primary DB crash. It’s the same as your #3 option but I save the shrug for the next day, when I read an email that there was a DB failover in the middle of the night and no one noticed a thing.
So why would you not do precisely the same thing with Mnesia, which is designed for live replication from the outset, unlike PostgreSQL which added it as an optional extension?
Because I consider it the wrong technical decision, most of the time. But I’m not building telecom stuff, I’m generally building stuff with much heavier requirements around database throughput.
Mnesia is not designed for the world of large scale data-plane requirements; it is an appropriate solution for the control-plane needs of a telecom switch. The live replication was designed for systems where each node is a daughterboard connected to the same backplane (which is likely responsible for Mnesia’s famous inattention to network partition safety).
Erlang escaped the lab successfully, and as a whole it is an incredibly robust toolbox for building systems, but not every single part of Erlang is a best-in-class tool for all use cases.
it exposes competent, low-level tooling to solve problems like durability without needing to pull in other stuff
What does this look like? I’ve heard vaguely about supervisors and “let it crash” and worker restarts, but I don’t understand how great monitoring and restart facilities would help me recover from e.g. disk or network or node failure.
It feels like designing distributed systems that store data ACIDly across multiple machines is hard regardless of language, and that seems scary to implement within my own application because not losing my customer’s data is mission-critical for me.
What does this look like? I’ve heard vaguely about supervisors and “let it crash” and worker restarts, but I don’t understand how great monitoring and restart facilities would help me recover from e.g. disk or network or node failure.
My guess is accepting that most applications simply don’t blow past what a single node with sqlite3 (in write-ahead-log mode) can handle comfortably. Have some kind of durable disk setup and aws s3 cp production.sqlite3 s3://backups/production-$(date +%s).sqlite3 running every few hours/minutes and you might be fine.
That said, managed offerings (I’ve used and not hated Heroku Postgres and Amazon Web Services Relational Database Service Aurora Serverless Version 2) start at like $50/mo, which is pretty cheap if your product generates revenue or your customer’s paying.
But at that point you’re using SQLite, and I don’t think that’s what the diagram means when it says “everything in Erlang”. If it were, then this argument would hold for data persistence for any language that has a SQLite binding.
Well, for a durable persistence layer we could use dets for a start, right? Or erl_tar, or zip. Or just file or io.
(Of course, if we wanted to have easy interop with BI software or dashboards, or wanted to do things that focus more on relations than documents, we absolutely should reach for an RDBMS like Postgres or sqlite).
My point is more that we shouldn’t be scared of using what’s built into the language if we don’t need more.
The diagram of “you can do everything in Erlang” really stood out to me. If true, this seems incredible. Does anyone actually write production Erlang systems this way e.g. writing data persistence in their Erlang application rather than delegating to Postgres? That sounds really scary.
Technically, Erlang can do persistable data via DETS (disk-based Erlang term storage) or Mnesia (a database written on top of ETS/DETS). In practice, I’ve heard this approach used much more as a control-plane database than as a data-plane, high-throughput DB. Specializing to an outboard DB like Postgres or MongoDB is much more common than persisting data within Erlang.
For ephemeral data, though, Erlang ETS offers some compelling advantages to Redis if you don’t want a network hop between you and the data you seek, and can pay the cost of replicating some of the same data across server nodes. Performance is good in control- and data-plane applications.
But these days, you often end up bringing Redis in anyway, as a backing store for Phoenix.PubSub when you don’t want to cluster your Erlang servers together (which is a whole can of worms in itself).
The rest of the diagram? More or less true as-is. Erlang/OTP is magic, a real batteries-included toolbox for soft-realtime distributed systems.
Klarna was storing their data in a large Mnesia cluster. They migrated to Postgres in the end, but that was only recently. It did save them a lot of hassle after the long migration was done. (I hope I remembered correctly from the talk at the Amsterdam meetup).
Edit: I found the presentation: https://m.youtube.com/watch?v=j7_T6R3uJSo
It seems that Whatsapp is also using Mnesia: https://algodaily.com/lessons/mnesia-whatsapp
The last thing that I ran with Erlang in production was eJabberd. It had a bunch of options for the data storage but I used mnesia because I’m lazy. It only had about a hundred users (probably an average of 20 active ones, down to one by the time I gave up on XMPP), but generally sat happily at <1% of CPU load on a single-core 1 GHz machine. I think it used well under 128 MiB of RAM (the machine had 1 GiB, but eJabberd was a small part of what it did).
Why do you find it scary?
No offense to you, but I think part of what makes a lot of people weirded out by Erlang/Elixir–and Lord knows during peak refugee migration from Rails–is that it exposes competent, low-level tooling to solve problems like durability without needing to pull in other stuff and cargo-cult “best practices”.
Truly bizarre and ancient magicks (like matchspecs) can let you do a whole lot of stuff right out of the box.
There’s a lot of comfort in having your app servers be stateless, and pushing mutable state into a particular system designed specifically for that. It would scare me to have to think about disk state on my app servers again – that’s 20-years-ago stuff. (DHH is not inviting me to Thanksgiving.)
That’s what you get with Erlang and Mnesia. The main difference is that you’re using BEAM for (very) lightweight process isolation, rather than the MMU for heavy OS process isolation.
Every Erlang process can be an independent failure domain. You structure your code so that the app server runs as a set of Erlang processes with monitors that restart them in case of failure. These push persistent state onto a system designed for it: Mnesia. That pushing is very cheap (ref counted immutable data within the same OS process), and Mnesia then handles pushing it to disk. You can run Mnesia in a separate BEAM VM if you want to (on another machine, if you want) and use the distributed Erlang bits to talk to it and then the overheads are closer to talking to an RDBMS over a socket.
It’s not the app-level logic that I’d be afraid of. Far from it, it’s as easy as using an external DB. The ops-level simplicity I’d be giving up, though, if I had to care about when every single app server dies and what happens to its disks? That gives me pause.
And at the point you’re shoving persistent state into a smaller pool of machines, IMO you are probably better served using a DB that is itself the main product, and not a side feature of a different product. Or go one step further, and use a managed DB like RDS Postgres, and pay for a robot to worry about it.
What do you do when the disks in your RDBMS host die? One of:
Mnesia supports all of these models. You can persist data to a single disk. You can (separately) back that up. Or you can build live replicas. Like everything else in Erlang, Mnesia is a distributed system and so you can run nodes in different failure domains for redundancy. For small deployments, you can run it on the same machine (and in the same OS process) as your app logic. For a larger deployment, you can run live replicas with dedicated machines running Mnesia and the others running the app logic. Oh, and you can put some RAM-only replicas on the same nodes as your app logic for fast reads.
Erland and Mnesia were designed for telecoms systems, where five nines of uptime is considered unacceptably unreliable. You almost certainly don’t have the same reliability requirements as a major telco.
Not a goddamned thing! RDS Postgres with automatic failover: pay 2x to have a replica you can’t touch, but that replica can take over in a couple of seconds following a primary DB crash. It’s the same as your #3 option but I save the shrug for the next day, when I read an email that there was a DB failover in the middle of the night and no one noticed a thing.
So why would you not do precisely the same thing with Mnesia, which is designed for live replication from the outset, unlike PostgreSQL which added it as an optional extension?
Because I consider it the wrong technical decision, most of the time. But I’m not building telecom stuff, I’m generally building stuff with much heavier requirements around database throughput.
Mnesia is not designed for the world of large scale data-plane requirements; it is an appropriate solution for the control-plane needs of a telecom switch. The live replication was designed for systems where each node is a daughterboard connected to the same backplane (which is likely responsible for Mnesia’s famous inattention to network partition safety).
Erlang escaped the lab successfully, and as a whole it is an incredibly robust toolbox for building systems, but not every single part of Erlang is a best-in-class tool for all use cases.
What does this look like? I’ve heard vaguely about supervisors and “let it crash” and worker restarts, but I don’t understand how great monitoring and restart facilities would help me recover from e.g. disk or network or node failure.
It feels like designing distributed systems that store data ACIDly across multiple machines is hard regardless of language, and that seems scary to implement within my own application because not losing my customer’s data is mission-critical for me.
My guess is accepting that most applications simply don’t blow past what a single node with sqlite3 (in write-ahead-log mode) can handle comfortably. Have some kind of durable disk setup and
aws s3 cp production.sqlite3 s3://backups/production-$(date +%s).sqlite3running every few hours/minutes and you might be fine.That said, managed offerings (I’ve used and not hated Heroku Postgres and Amazon Web Services Relational Database Service Aurora Serverless Version 2) start at like $50/mo, which is pretty cheap if your product generates revenue or your customer’s paying.
But at that point you’re using SQLite, and I don’t think that’s what the diagram means when it says “everything in Erlang”. If it were, then this argument would hold for data persistence for any language that has a SQLite binding.
Well, for a durable persistence layer we could use dets for a start, right? Or erl_tar, or zip. Or just file or io.
(Of course, if we wanted to have easy interop with BI software or dashboards, or wanted to do things that focus more on relations than documents, we absolutely should reach for an RDBMS like Postgres or sqlite).
My point is more that we shouldn’t be scared of using what’s built into the language if we don’t need more.
[Comment removed by author]