I think its probably worth noting that this doesn’t really solve the main pain points with distributing OTP releases, namely dynamically linked dependencies and packaging the VM which tends to be highly sensitive to changes in the host platform. It is a similar approach to what Distillery used to provide “executable” releases, i.e. a self-extracting tarball, just a shell script with the tarball embedded in a section at the end which the script would handle extracting and then exec’ing into the release start script. This is definitely a nicer variation on that trick, but from what I can tell, it isn’t fundamentally different, and not what people would associate with deploying statically linked executables ala C/Rust/Go/etc.
This looks rather cool. The 0.5 sec startup time means it probably won’t be the best option for cli tools that get called frequently (in general, the BEAM doesn’t seem like a good fit for short-run programs), but still.
I wonder how big a Phoenix app would end up being compiled like this? (Or if it’s even reasonable to try doing that)
This looks rather cool. The 0.5 sec startup time means it probably won’t be the best option for cli tools that get called frequently (in general, the BEAM doesn’t seem like a good fit for short-run programs), but still.
Yup, similar to the JVM, there’s a big delay during VM launch. Given Erlang’s niche in long-lived services, I don’t see this going away soon.
I wonder how big a Phoenix app would end up being compiled like this? (Or if it’s even reasonable to try doing that)
Building examples/simple_app results in a 13MB executable, whereas examples/phoenix_app is 15MB. That’s with whatever JS Phoenix pulls in too. Phoenix is apparently not a major factor in executable size.
For sure. There are times that I’d like something with the syntax of Erlang (or maybe elixir), but better suited to short-lived programs/scripts. It’s not strong enough of a desire to make something like that, I just remember thinking how nice and concise a lot of Erlang functions ended up being when I was writing them before.
I suppose Haskell could be used there, though, though I’ve not given it much thought, and Haskell isn’t something I’ve really wanted to write of late.
Yes and no. It’s a nice syntax for the BEAM, to be fair, but the lack of the pervasive pattern matching in Ruby means you’re writing a fundamentally different thing. It’s the pattern matching and/or type system I’d like to see outside of the BEAM VM, and in a smaller/lighter language.
Yeah this is definitely a use case for Lumen (disclosure: I’m on the core team), and is one of the main goals I personally had for the project.
I’d say 6 months is pretty on point for when things will be in a solid place, but we’re already super close to being able to use it for command line tooling and other ad-hoc programs, mainly just working out some kinks with codegen right now, and figuring out the best way to ship things like OTP.
I’d be happy to! I’m sure Luke and Hans would as well :)
To your point about the BEAM being impenetrable: that’s definitely one thing we think of as a benefit not only of Lumen, but other projects like it, and we’ve certainly collected a lot of notes while reimplementing the runtime, in particular around garbage collection/memory management and scheduling. I still need to spend time turning all of that ad-hoc note collecting into some proper documentation, but its absolutely something we’ll be doing!
The runtime portions of the project are currently in a state where they contain both the “real” implementation of various components as well as a bunch of legacy cruft we accumulated early on while the compiler was still very much not ready, so there are some places that are kinda ugly which will be cleaned up as we continue to strip those things out. Most of the core runtime functionality aside from the process internals is broken up pretty well though and would make for a good intro to how the BEAM does things (I hope!). Luke has put a vast amount of effort into building a test suite for the runtime, which is one area I think the BEAM lacks tests (it has a ton of tests, but they are basically entwined with Erlang, whereas Lumen can test a lot of its runtime completely independent of Erlang code).
Anyway, don’t want to try and answer everything all at once here, I would be happy to do some kind of AMA here, though I don’t actually know if they are considered acceptable posts or not (I don’t have any reason to think they aren’t, just haven’t seen any that I can recall). Feel free to shoot me an email though or hit us up in the #lumen channel in the Elixir slack, and I can at least answer any specific burning questions you have :)
I’m in a similar boat. As an Elixir programmer it’s hard to live without pattern matching in my scripting and CLI applications. Currently I’m settling on Lisp.
I think its probably worth noting that this doesn’t really solve the main pain points with distributing OTP releases, namely dynamically linked dependencies and packaging the VM which tends to be highly sensitive to changes in the host platform. It is a similar approach to what Distillery used to provide “executable” releases, i.e. a self-extracting tarball, just a shell script with the tarball embedded in a section at the end which the script would handle extracting and then exec’ing into the release start script. This is definitely a nicer variation on that trick, but from what I can tell, it isn’t fundamentally different, and not what people would associate with deploying statically linked executables ala C/Rust/Go/etc.
That said, its still very cool!
This looks rather cool. The 0.5 sec startup time means it probably won’t be the best option for cli tools that get called frequently (in general, the BEAM doesn’t seem like a good fit for short-run programs), but still.
I wonder how big a Phoenix app would end up being compiled like this? (Or if it’s even reasonable to try doing that)
Yup, similar to the JVM, there’s a big delay during VM launch. Given Erlang’s niche in long-lived services, I don’t see this going away soon.
Building
examples/simple_appresults in a 13MB executable, whereasexamples/phoenix_appis 15MB. That’s with whatever JS Phoenix pulls in too. Phoenix is apparently not a major factor in executable size.For sure. There are times that I’d like something with the syntax of Erlang (or maybe elixir), but better suited to short-lived programs/scripts. It’s not strong enough of a desire to make something like that, I just remember thinking how nice and concise a lot of Erlang functions ended up being when I was writing them before.
I suppose Haskell could be used there, though, though I’ve not given it much thought, and Haskell isn’t something I’ve really wanted to write of late.
If you’d like the syntax of Elixir, then you’d want to try Ruby, because Elixir’s syntax is inspired by Ruby.
Yes and no. It’s a nice syntax for the BEAM, to be fair, but the lack of the pervasive pattern matching in Ruby means you’re writing a fundamentally different thing. It’s the pattern matching and/or type system I’d like to see outside of the BEAM VM, and in a smaller/lighter language.
[Comment removed by author]
Lumen, a Beam VM written in Rust that targets Wasm might give u exactly what you want, but it is probably at least 6 months out.
Yeah this is definitely a use case for Lumen (disclosure: I’m on the core team), and is one of the main goals I personally had for the project.
I’d say 6 months is pretty on point for when things will be in a solid place, but we’re already super close to being able to use it for command line tooling and other ad-hoc programs, mainly just working out some kinks with codegen right now, and figuring out the best way to ship things like OTP.
What a wonderfully small world, :)
It would be interesting to do an AMA with you and KronicDeth on Lumen with questions around
As well as an architectural overview of Lumen. The Erlang VM is impenetrable to lots of folks with ideas for experiments that hopefully Lumen is not.
I’d be happy to! I’m sure Luke and Hans would as well :)
To your point about the BEAM being impenetrable: that’s definitely one thing we think of as a benefit not only of Lumen, but other projects like it, and we’ve certainly collected a lot of notes while reimplementing the runtime, in particular around garbage collection/memory management and scheduling. I still need to spend time turning all of that ad-hoc note collecting into some proper documentation, but its absolutely something we’ll be doing!
The runtime portions of the project are currently in a state where they contain both the “real” implementation of various components as well as a bunch of legacy cruft we accumulated early on while the compiler was still very much not ready, so there are some places that are kinda ugly which will be cleaned up as we continue to strip those things out. Most of the core runtime functionality aside from the process internals is broken up pretty well though and would make for a good intro to how the BEAM does things (I hope!). Luke has put a vast amount of effort into building a test suite for the runtime, which is one area I think the BEAM lacks tests (it has a ton of tests, but they are basically entwined with Erlang, whereas Lumen can test a lot of its runtime completely independent of Erlang code).
Anyway, don’t want to try and answer everything all at once here, I would be happy to do some kind of AMA here, though I don’t actually know if they are considered acceptable posts or not (I don’t have any reason to think they aren’t, just haven’t seen any that I can recall). Feel free to shoot me an email though or hit us up in the #lumen channel in the Elixir slack, and I can at least answer any specific burning questions you have :)
Great! It sounds like things are in motion. Priming is 30% of it. I’ll join the lumen slack channel is the next week or so.
That sounds like a very interesting project, especially with CLI tooling as a target.
I’ll have to keep an eye on it
Prolog? :) That’s what Erlang’s syntax originally evolved from!
I’m in a similar boat. As an Elixir programmer it’s hard to live without pattern matching in my scripting and CLI applications. Currently I’m settling on Lisp.