Lua isn’t so much “write once, run anywhere” (there is a Lua bytecode, but it’s not a very common AOT compilation target, and in any case most Lua is meant to run in exactly one environment) — a better descriptor would be “it’s everywhere”. Its intention was always to be used as an embedded “scripting” language inside of some bigger system, for convenience (not having to write all of your functionality in C or whatever) and extensibility (since you’re shipping an interpreter, it’s easy to let the users drop in their own code). Much like ended up happening with JavaScript, except a bit less accidental.
But yeah, now wasm is starting to fill that niche, and people can compile to wasm from (more or less) any language they want. Which is nice, although it does rob the idea of a lot of its simplicity.
This for me is the main interesting use-case for WASM - not running C code in the browser (though there are some great examples of that, like Figma) but instead using WASM as a kind of embedded VM in which you can run guest plugins.
I’m working on some new forum-like software with an API written in Go at the moment and one question I always got from people using old style PHP forums was: how are you going to make plugins work like the old days? And wazero is our solution to that!
Seeing stuff like this makes me really excited for the future where every language has easy access to a WASM interpreter, and easily compiles to WASM/can generate a WASM module. I’d love to be able to use libraries written in C# in my Go programs, or import functions from a JavaScript library in my Python programs.
I had read that WASM didn’t work well with garbage collection based languages like C#, but I don’t know how true that is these days. Anyway C# can be embedded in other languages if you have access to COM, and on Linux you could always embed Mono (probably you can do this with .net core, but I haven’t tried). You can definitely already use JavaScript in Python without involving WASM at all.
WASM runs inside a GC virtual machine, so yeah you can compile a GC language to WASM but then you will have two GCs. So cross-compilation like rust can do has a natural advantage.
GC just reached phase three a month ago, so it’ll take time for languages like .NET and Go to tune their WASM implementations to work better in tandem with the built-in GC.
People have found interesting and clever ways to make code written in one language available in another since “programming” was a thing. The thing that excites me is a possible future where cross-lnlanguage portability is something that’s built into every language, with WASM being the bridge between everything, not hacky, possibly unsupported forks like Mono, TinyGo, and IronPython.
I love how you need to explicitly supply environment interaction functions to the WASI runtime. If you don’t give it file access, it won’t have file access…
I have this dream of using WASM + WASI as a library/application mechanism for a Spark-like system. Build your processing code in whatever you want (Rust, Go, C++, anything that can compile to WASM) and then provide it to a coordinator system that launches the job and moves the app to the data like Hadoop, Spark, etc. Basically, I want Spark (Scala, Python) but with WASM as the compile target.
I knew that carcinization meant things evolving into crab-like forms, but I was confused what that had to do with the post. Now I have learned that Ferris the crab (a crustacean) is the unofficial mascot of Rust, so the post title is referencing the calling of Rust from Go.
It does work, but overall the developer experience is poor. Your build is no longer one simple go build. Now you have to remember to run cargo build --release and ensure that the resulting .so, .dll, or .dylib is in the right path for the OS’ dynamic linker to read from.
(Note: I know nothing about cgo.)
Can the library not be linked statically with cgo? It seems like this would be possible with the right comment directives. Or is there some (obvious?) issue with that that I’m missing?
Also: I wonder if there’s a reason that the HTML parsing/rewriting portion was written in Rust (other than the code reuse from the CLI tool, I think?), since there’s at least a few HTML parsing libraries in Go.
Yes but you still have to build the library per OS and CPU architecture. go build doesn’t have any facilities for dynamically doing this at compile time. You’d get Rust to spit out the moral equivalent of a .a file and have cgo link against that.
The main reason I didn’t do that is that is boring and not cursed at all. This tactic is cursed and I love it.
This makes me wonder if you can do fat libraries the same way fat binaries are done? Can’t find any obvious answers for cross-platform cross-architecture solutions. Darwin supports multi-architecture ones of course.
You can, but afaict you kinda end up with all the disadvantages of native code and fat binaries together. You need complicated build tools and you will inevitably end up with platform or API version mismatches anyway.
I like the distribute-bytecode-libs solution a lot, conceptually. No reason you can’t AOT compile them to native code on the platform they’re running on when the program is installed.
Why not just write the original program in Go? The consequence of an incorrect program or memory leak is marginal since the invocation is short lived…?
I don’t have lol_html in go: https://docs.rs/lol_html. There are go bindings for it, but that requires you to ship a Rust .so file, which I don’t want to do.
Deeply cool stuff. It’s back to CGI, this time wasm flavored
Everything old is new again. Compile once run everywhere is back in fashion!
but this time with WASM, not Lua or Java
Never heard this credo applied to Lua before. Seems like an interesting target for it. Do you have a link with more info?
sadly I don’t, but lua is pretty much universal for many scripting additions, especially many game engines
Lua isn’t so much “write once, run anywhere” (there is a Lua bytecode, but it’s not a very common AOT compilation target, and in any case most Lua is meant to run in exactly one environment) — a better descriptor would be “it’s everywhere”. Its intention was always to be used as an embedded “scripting” language inside of some bigger system, for convenience (not having to write all of your functionality in C or whatever) and extensibility (since you’re shipping an interpreter, it’s easy to let the users drop in their own code). Much like ended up happening with JavaScript, except a bit less accidental.
But yeah, now wasm is starting to fill that niche, and people can compile to wasm from (more or less) any language they want. Which is nice, although it does rob the idea of a lot of its simplicity.
This for me is the main interesting use-case for WASM - not running C code in the browser (though there are some great examples of that, like Figma) but instead using WASM as a kind of embedded VM in which you can run guest plugins.
I’m working on some new forum-like software with an API written in Go at the moment and one question I always got from people using old style PHP forums was: how are you going to make plugins work like the old days? And wazero is our solution to that!
Seeing stuff like this makes me really excited for the future where every language has easy access to a WASM interpreter, and easily compiles to WASM/can generate a WASM module. I’d love to be able to use libraries written in C# in my Go programs, or import functions from a JavaScript library in my Python programs.
I had read that WASM didn’t work well with garbage collection based languages like C#, but I don’t know how true that is these days. Anyway C# can be embedded in other languages if you have access to COM, and on Linux you could always embed Mono (probably you can do this with .net core, but I haven’t tried). You can definitely already use JavaScript in Python without involving WASM at all.
WASM runs inside a GC virtual machine, so yeah you can compile a GC language to WASM but then you will have two GCs. So cross-compilation like rust can do has a natural advantage.
GC just reached phase three a month ago, so it’ll take time for languages like .NET and Go to tune their WASM implementations to work better in tandem with the built-in GC.
People have found interesting and clever ways to make code written in one language available in another since “programming” was a thing. The thing that excites me is a possible future where cross-lnlanguage portability is something that’s built into every language, with WASM being the bridge between everything, not hacky, possibly unsupported forks like Mono, TinyGo, and IronPython.
I love how you need to explicitly supply environment interaction functions to the WASI runtime. If you don’t give it file access, it won’t have file access…
Yeah! I also didn’t give that a filesystem so any file I/O save stdin/stdout just doesn’t work. It’s kinda great!
I have this dream of using WASM + WASI as a library/application mechanism for a Spark-like system. Build your processing code in whatever you want (Rust, Go, C++, anything that can compile to WASM) and then provide it to a coordinator system that launches the job and moves the app to the data like Hadoop, Spark, etc. Basically, I want Spark (Scala, Python) but with WASM as the compile target.
I knew that carcinization meant things evolving into crab-like forms, but I was confused what that had to do with the post. Now I have learned that Ferris the crab (a crustacean) is the unofficial mascot of Rust, so the post title is referencing the calling of Rust from Go.
(Note: I know nothing about cgo.)
Can the library not be linked statically with cgo? It seems like this would be possible with the right comment directives. Or is there some (obvious?) issue with that that I’m missing?
Also: I wonder if there’s a reason that the HTML parsing/rewriting portion was written in Rust (other than the code reuse from the CLI tool, I think?), since there’s at least a few HTML parsing libraries in Go.
Yes but you still have to build the library per OS and CPU architecture.
go builddoesn’t have any facilities for dynamically doing this at compile time. You’d get Rust to spit out the moral equivalent of a.afile and have cgo link against that.The main reason I didn’t do that is that is boring and not cursed at all. This tactic is cursed and I love it.
It doesn’t sound like you want it? I don’t understand why you would introduce this problem knowingly. Accidentally I understand.
I committed the .wasm file to the repo. This means consumers don’t need the Rust toolchain installed.
This makes me wonder if you can do fat libraries the same way fat binaries are done? Can’t find any obvious answers for cross-platform cross-architecture solutions. Darwin supports multi-architecture ones of course.
You can, but afaict you kinda end up with all the disadvantages of native code and fat binaries together. You need complicated build tools and you will inevitably end up with platform or API version mismatches anyway.
I like the distribute-bytecode-libs solution a lot, conceptually. No reason you can’t AOT compile them to native code on the platform they’re running on when the program is installed.
Why not just write the original program in Go? The consequence of an incorrect program or memory leak is marginal since the invocation is short lived…?
I don’t have lol_html in go: https://docs.rs/lol_html. There are go bindings for it, but that requires you to ship a Rust .so file, which I don’t want to do.
Damn, go is really behind not having a similar package to parse and modify html…
Realistically, the HTML package in the x repos probably would work fine, lol_html just has an API I’m used to.
This is really cool, but the premise made me think of using a Makefile.