1. 26
  1. 11

    This is the wrong layer. WebAssembly doesn’t replace Kubernetes, it replaces OCI containers. It is a mechanism for packaging a self-contained program and running it in an isolated environment. It is not a mechanism for orchestrating a fleet of such deployments. ‘Is WebAssembly the new Docker?’ is a valid question.

    1. 10

      So one answer to the question as initially posed is that no, WebAssembly is not the next Kubernetes; that next thing is waiting to be built, though I know of a few organizations that have started already.

      I agree there is something of an “inner platform effect” going on here…

      https://en.wikipedia.org/wiki/Inner-platform_effect

      That is, WebAssembly is useful, but we don’t need it to encompass all of server side computing … That’s just rebuilding an operating system on top of an operating system, which basically doubles the number of problems your system has.

      I imagine it’s going to be a similarly useful technology as the JVM … but the JVM is by and large “just another Unix process” in a typical server workload. It didn’t end up being as general or universal as was once thought.

      1. 2

        I haven’t started using webassembly in practice yet, but isn’t one benefit of WASI et al that the enable more advanced capabilities systems than (most) traditional server operating systems? That could be a practical benefit, especially for multi-tenant services i think.

        1. 7

          Having default-deny of a wasm sandbox vs. default-allow of Unix is definitely better in principle. However it does make it harder to port existing software, which was one benefit of WASM.

          I think what @eyberg listed is basically what I’m getting at. WASM is a small part of a server side platform; WASI adds more, but you will end up rebuilding most of an OS. That is exactly what the JVM tried to do.

          And you will have downsides like lack of heap protection, since WASM is a flat address space:

          https://lobste.rs/s/pzr5ip/everything_old_is_new_again_binary

          And performance issues like lack of SIMD. I think they are working on adding SIMD, but again it’s another layer / point of indirection, i.e. an “inner platform”. An ISA is already an abstraction layer over physical hardware :) This may be a cost you want to pay in some circumstances, but not all.

          Here I question if WebAssembly is really “polyglot”: https://news.ycombinator.com/item?id=28581634

          i.e. programmers underestimate the degree to which languages and VMs are coupled. Any VM design decision creates some “losers”, and the losers are the languages that will experience a 2x-10x slowdown over their native environment.

          1. 1

            Having default-deny of a wasm sandbox vs. default-allow of Unix is definitely better in principle.

            Snap’s --classic flag comes to mind here.

            1. 1

              And you will have downsides like lack of heap protection, since WASM is a flat address space

              There is https://github.com/WebAssembly/multi-memory, which is basically memory segmentation for WebAssembly.

            2. 1

              The capability system is incredible, almost comparable to that of lua - with the advantage that nearly all WASM implementations are coded defensively and expect hostile bytecode.

          2. 7

            Something I’ve really wanted to do in my Copious Free Time is swap out the Dis virtual machine in Inferno with WASM. I feel like it could be doable.

            1. 5

              This has gotten brought up quite a lot in the past few years but every time I check there still seems to be a whole host of issues.

              Whoever is actually following the scene can they comment on the following (lack of) support? To me these are absolutely huge blockers to become a new generic app runtime ala k8s:

              true threads?

              64bit - https://github.com/WebAssembly/memory64

              dynamic linking - https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md

              real sockets? - https://github.com/WebAssembly/WASI/pull/312

              aslr/stack canaries?

              TLS? - https://github.com/bytecodealliance/wasmtime/issues/71

              1. 2

                A summary from my knowledge

                • True threads - do you mean pthread like shared state?
                • 64bit: I know nothing beyond that proposal
                • dynamic linking - no idea
                • Raw sockets: Not going to happen in a web facing environment (e.g. browser), but there’s no reason the host environment can’t provide them elsewhere. It is easy enough to expose such an API, the only real issue is designing the API interface.
                • aslr/stack canaries: WASM is memory safe w.r.t the execution stack and internal machine state so wayward code can’t attack control flow via the stack. ASLR is also less helpful as wasm doesn’t allow jumping to a location other than an entry point so many of the non-vm problems aren’t applicable. There are of course cases where you might not want an attacker to control the function being called, so it might be worth it. Because they’re already opaque implementations might already be randomizing?
                • TLS: same as raw sockets the host environment would provide it, and so the host would need to define things. Not likely to be visible in a browser context, as it is of limited value in the absence of raw socket access.
                1. 2

                  re: threads: No, shared array buffers and web workers are not what I would consider true threads.

                  re: 64bit: this effectively rules out all databases, most ‘enterprise’ software and many many other applications such as machine learning applications and other assorted ‘big data’ projects because of memory addressment requirements; also I haven’t deployed any 32bit applications in prod for what 10 years now?

                  re: aslr /stack canaries: I might’ve just labeled this ‘generic security’ as wasm doesn’t even have the notion of read-only memory - just because it’s in a ‘sandbox’ doesn’t mean anything as is pointed out in this paper: https://www.usenix.org/system/files/sec20-lehmann.pdf

                  As you pointed out - I guess that’s my point. There are definitely interesting use-cases with WASM and WASI but to answer the parent’s article claim that it is the next k8s or next application run-time - without addressing most of this, which I agree some of these will never be addressed - it is a resounding ‘no’. WASI was supposed to be the host environment but it also fails to address most of this. I suppose one could code up WASI++ and make it a server-side only implementation but then it’s not WASM (or even w/WASI) anymore.

                  1. 1

                    Re address space: it’s not quite as dire as it sounds. WASM does not quite operate in a 32bit address space. Effectively it has a core heap which is indeed bound to 32bits, but typed arrays exist outside of that and are indexed by size, so for example a wasm binary can access the 2^32-1th element of an array of doubles without a problem, and it can have many such typed arrays. That’s arguably useful in some like ML as they could be arranged to exist primarily in GPU memory for example.

                    The read/write memory issue the usenix paper spends time discussing aren’t super useful as there are very few cases where I have seen any program, security sensitive or not, use to protect the heap.

                    Most of the other issues in the paper boils down to the host environment blindly trusting the VM. Which is a no go for any containerisation mechanism.

                    I think a core issue with the usenix paper is that it is trying to pretend that wasm is meant to provide a memory safe environment. But that is by design not what wasm is. Wasm must be able to run code from a non-memory or type safe language, which means that the VM cannot, in the general case, place limits on how people interact with memory.

                    I do agree that it could have been achieved in a way that was harder to exploit, but the folk who started it were more interested in compiling arbitrary C into a form it could be executed at near native speed, while remaining safe in a browser context, so this is the model we got :-/

                    1. 1

                      Are you saying you can address more than 4 gig of memory? It is my understanding that you can’t and if you can’t that excludes a ton of applications - most importantly databases. I’m not saying that you can’t run other programs but the initial claim in the parent article makes it sound like wasm is going to be the new underlying runtime which is what I don’t agree with because of limitations like this.

                      which means that the VM cannot, in the general case, place limits on how people interact with memory.

                      This is a fundamental problem though.

                      Let me paint a picture for you. You have a function called “is_correct_password” or “is_admin” or “can_rm_rf_this_entire_partition”. The fact that you can’t mark certain pages as read only means I can overwrite that with whatever I want (eg: return true). It is a major security issue and the paper clearly shows real life examples of how dangerous it is. The whole “it is in a sandbox” claim that wasm proponents continuously make makes no sense as everything inside the sandbox - eg: the program logic at large is at risk.

                      To put it another way if you take an ordinary linux application running as an elf - say a go program, or a “memory-safe” rust program you considerably downgrade the security of it by compiling and running it as wasm.

                      1. 3

                        Are you saying you can address more than 4 gig of memory?

                        From WebAssembly? Yes. From C code? No. WebAssembly has memory objects. Each memory object is bounds checked. Every memory access is a memory object identifier + some offset (which must be within the size of the memory object). In 32-bit WAsm (the only version fully specified so far) each memory object is limited to a 32-bit size.

                        C code compiled to WebAssembly uses memory object 0 and lowers pointers to integer values that represent an offset within the allocation. You could probably create a different lowering that used 64-bit integers to encode a memory ID and an offset (though most implementations heavily optimise for accesses in object 0 and so this would probably be slow).

                        I personally hate this part of WebAssembly. Providing a PDP-11 memory model in 2017 is inexcusable. Especially since this was two years after I’d published the paper showing that most C could be lowered to CHERI with object-granularity spatial memory safety with zero or few changes (we’re currently seeing around 0.02% line of code changes large C/C++ codebases for CHERI - smaller changes than porting to an environment without exceptions, for example).

                        Let me paint a picture for you. You have a function called “is_correct_password” or “is_admin” or “can_rm_rf_this_entire_partition”. The fact that you can’t mark certain pages as read only means I can overwrite that with whatever I want (eg: return true).

                        WebAssembly doesn’t allow you to mark pages read only, but it also doesn’t allow you to mark pages as executable. Code, data, and stacks, are completely distinct memory regions in WebAssembly. You cannot modify code, you cannot take the address of the stack (address-taken stack allocations are lowered to allocations within data memory).

                        There are a lot of legitimate reasons to criticise WebAssembly but this is not one of them.

                        1. 1

                          You can overwrite constants in what traditionally would be read-only. You can overwrite portions of the stack. You can overwrite the heap. You can overwrite function pointers.

                          In particular with your suggestion that you can’t redirect calls is incorrect. The paper clearly shows examples of them redirecting indirect calls.

                          1. 1

                            A core difference is that you don’t get the blind overwrites you have on a real machine.

                            A “function pointer” in the wasm runtime is either an index into the global function table, or a funcref. When a function is called through that indirection, the call site must be calling with the correct signature. So not only must your substitute function be an existing entry point, it must also have the correct type. Further limiting the ability to attack said code.

                            I don’t know what you’re talking about when you say you can overwrite constants that would be traditionally readonly. The obvious one would be global variables, but globals are marked as being mutable or immutable at individual declaration granularity - they aren’t part of the primary memory region.

                            1. 1

                              In the paper they clearly show overwriting read-only data (page 11, figure 9) and they clearly show overwriting function pointers that are implemented as indirect calls (page 11, figure 8).

                              1. 1

                                Figure 9) ok, having read and reread I think I understand. By the VM spec it seems like this really should be read only, however the VM spec also allows the wasm vm instantiator the ability to place data at arbitrary locations, so they can’t be made read only. Coolcoolcool. That’s before you even consider sharing the same region with other vms and js or whatever the host environment is. In an ideal world the spec would mirror at least the basic idea of having an ro text section. In an ideal world it would probably work more similarly to shader languages in which you don’t get a priori knowledge of where in your linear address space things can go - you have to query the vm. That would also allow the vm to do aslr if it did end up being necessary.

                                That would also let the VM arrange things more sanely and so allow multimapping, etc to work with shared memory.

                                Figure 8) exactly as I said, they were able to change a call from one function of a specific type, to another function of that same type

                        2. 1

                          Sorry, I am a moron, for some reason I recalled being able to share typed arrays with the VM - I don’t know if that’s something that came up early on in wasm planning land, or something my memory completely screwed :D

                          For the second problem, your example can’t happen: the wasm vm is a Harvard architecture: program code does not exist in the data heap, and instructions in the wasm VM can only read and write to the data heap.

                          The WASM VM cannot do run time codegen at the moment, and any future APIs have to consider the safety of the VM which means they’re not going to allow rewriting of existing code in arbitrary ways.

                          Strictly of course a WASM program could in principle generate new wasm code, pass that to the host, and then have the host instantiate that in a new VM, but I suspect the end performance may be suboptimal :D

                          [edit: immediately having posted this I realized that I probably mixed up people historically complaining about the address space in JS, because [typed]arrays could (at the time) only have a maximum length of 2^32-1 elements. Obviously this isn’t “4gb” as it takes more than one byte per element*, and of course you can have many such arrays.

                          • The actual size per element varies, speaking for JSC: Historically it would be a pointer/element, but it’s been 8bytes on 32 and 64 bit for maybe a decade now? possibly longer. TypedArrays are obviously sizeof(type) per element
                          1. 1

                            So just to be clear - can you point me to an example of a database that needs 16 or 32 gig of ram to store indices in memory - not on disk - as wasm?

                            1. 1

                              I’m not a DB person, but I know many many system that work with large data use MMIO which means you need a large address space, even though you don’t necessarily have sufficient actual memory.

                              1. 1

                                Sure - but when that pages out to disk there goes your performance. Again I’m not talking about keeping a working set in memory. I’m just talking about keeping a large index in memory. If you can’t do that then the database grinds hard.

                                1. 1

                                  Yes, but the databases control their access very carefully.

                                  IIRC oracle dbs used to have their own file system so that they could get even better control of caching, etc

                                  Whether that’s still necessary I don’t know - you can get “consumer” hardware with ~200gb+ of ram these days so maybe it’s leas important?

                                  1. 1

                                    Not sure we’re on the same page here.

                                    I literally can’t spin up, say a mysql database, with more than 4gig of ram using wasm today and expect it to use that ram to store things like indices in memory. So if I have a small database that has 32 gig of ram that’s a no go for wasm today.

                                    1. 1

                                      I agree - we aren’t

                                      I think you missed my earlier correction (I thought memory blobs were in the spec) and I missed the “in wasm” in your subsequent request for examples :D

                  2. 1

                    true threads?

                    This is a tricky one. There are some proposals floating around to allow fine-grained memory safety in Wasm. These are very cheap to implement if you don’t allow shared-memory concurrency, much harder if you do (unless you have CHERI hardware). Disallowing pointers in shared buffers (requiring explicit message sending of pointers between workers) would work.

                    64bit

                    As far as I’m aware, this is easy but not a priority because it lacks a motivating use case.

                    dynamic linking

                    This would bring a lot of problems. What is the scope of sharing? WebAssembly is intrinsically a sandboxed deployment model and a lot of the value comes from having a complete inventory of the code running in a given sandbox.

                    real sockets?

                    This is not a limitation of WebAssembly, it’s a bit of WASI that isn’t fully spec’d. An embedding can provide sockets.

                    aslr/stack canaries?

                    Stack canaries are less necessary in WAsm because the return address is always stored on a separate stack to any address-taken allocations. ASLR is of limited use these days (there are lots of bypasses that are integrated into exploit toolkits) and it’s also difficult to get a useful amount of entropy in a 32-bit address space. That said, the WAsm program has complete control over its 32-bit address space and so a malloc implementation can easily add randomisation without needing any extensions to WAsm.

                    TLS?

                    Same answer as sockets. An embedding can expose KTLS.

                    1. 1

                      64 bit

                      The motivating use-case for 64bit is simple. I need to load more than 4gig into memory - that’s pretty much any ‘enterprise’ software project or anything that touches a small amount of data.

                      dynamic linking

                      I agree that this is not a huge problem at all, however, I will point out that the vast majority of software (99%+?) is all dynamically linked.

                      real sockets/TLS

                      I guess my point here is that while individual programs have had this functionality tacked onto them there is no common spec for it yet and probably won’t be? I linked directly to the issue to the WASI proposal that has been dead for at least a year. Until that extension exists that’s a fairly hard limitation. Most programs that expect use of native sockets and by extension TLS are not setup to delegate that responsibility over to a shim which is how some people are dealing with this now.

                      1. 1

                        The motivating use-case for 64bit is simple. I need to load more than 4gig into memory - that’s pretty much any ‘enterprise’ software project or anything that touches a small amount of data.

                        There’s a difference between more than 4 GiB of memory in a system and more than 4 GiB of memory in a single security context. I would suggest that any ‘enterprise’ software that puts 4 GiB in a single security context is something that the red team should focus on in their next audit.

                        I agree that this is not a huge problem at all, however, I will point out that the vast majority of software (99%+?) is all dynamically linked.

                        This is increasingly untrue for cloud deployments. Increasingly, VMs run a single container, which contains a single process. There is no benefit from dynamic linking and there’s a performance overhead.

                        1. 1

                          ? Are you saying I can map more than 4gig of memory in wasm? I was under the distinct impression that is totally impossible right now. As I’ve pointed out almost every single prod db will be doing this and many many java applications will be doing this.

                          As for statically linking in a container - that is still nowhere close to being prevalent. One small security benefit of dynamic linking is randomization of addresses from the libraries. I stand by my comment that over 99% of software is dynamically linked. If you have actual stats to show otherwise happy to see that.

                  3. 3

                    is K8s really an evolution of OpenStack? I thought it was an evolution of / reimagining of Borg.

                    1. 5

                      K8s itself is an evolution on a previous architecture, OpenStack. OpenStack had each container be a full virtual machine, with a

                      Yeah this is WRONG. Roughly speaking, OpenStack is an open source AWS, while Kubernetes was inspired by Borg (and both came out of Google).

                      And OpenStack/AWS are based on VMs, while Borg/Kubernetes are based on containers.

                      What eyberg posted from Hashicorp (a competittor) is slightly biased but generally correct. Google advertises Kubernetes as like Borg but in my opinion it’s worse. (I used Borg for many years at Google across several domains.)

                      I quote a first hand and informed opinion in this blog post:

                      http://www.oilshell.org/blog/2021/07/blog-backlog-2.html

                      MetalLB taught me that it’s not possible to build robust software that integrates with Kubernetes.

                      GKE SRE taught me that even the foremost Kubernetes experts cannot safely operate Kubernetes at scale.

                      1. 3

                        Armon isn’t stupid. He knows full well openstack is built for vms and k8s is built for containers.

                        What he is saying here is the vendor/marketing/kitchen sink approach is the same - not the tech.

                        1. 3

                          I’m quoting the author of the blog post, which says “architecture”, which is wrong.

                      2. 4

                        Armon captures this common sentiment pretty well in this interview:

                        https://www.nextplatform.com/2021/06/15/forget-mesos-and-openstack-hashi-stack-is-the-new-next-platform/

                        Armon Dadgar: For me, it comes down to three really simple things. One is just the elegance of experience. Kubernetes is OpenStack 2.0. It’s just as complicated. It’s just as vendor controlled. It’s just as foundation led.

                        Armon Dadgar: Let’s ignore the usability of Kubernetes, which is a mess. Let’s talk about its actual operational scalability. It’s also a joke. Borg runs on 10 million nodes, Kubernetes falls over if you have a few hundred. And so in what sense did Google learn from Borg when Kubernetes only scales to 1/1000th of what Borg does?

                        1. 4

                          The sentiment is generally right, although it’s not a great source since it’s a competitor.

                          But a worthwhile correction: A borg cluster doesn’t span 10M nodes. When I left >5 years ago they were 10K to 100K nodes each, and they don’t really talk to each other, and there is a lot of manual “toil” to keep everything up and synchronized and manage downtime. (I think this information is readily available in the Borg paper.)

                          Still I agree Kubernetes is worse than Borg along many dimensions, and I quoted another blog in a sibling comment giving lots of detail than that.

                      3. 2

                        webassembly is not at all about distributed orchestration, so it seems pretty obvious to me that no, wasm isn’t the new kubernetes.

                        It could however be a very good fit as a CRI implementation (or apparently as a kubelet).

                        There could totally be an alternative distributed orchestration project that would directly be bound to wasm, but I don’t know any yet (except maybe wasmcloud?)