1. 7

  2. 3

    A unikernel is simply an application that has been boiled down to a small, secure, light-weight virtual machine. The resulting virtual machine image does not contain an operating system like Linux or Windows.

    A unikernel still has a kernel to my understanding… so what kernel is this running?

    1. 4

      According to https://github.com/nanovms/ops/blob/master/CHARTER.md it’s the NANOS unikernel, which AFAICT is not open source.

      Appears to be straight-up spam; this is the only thing @eyberg has posted here and it’s promoting their commercial service without further comment.

      1. 3


        How do we know it is secure if we do not see the code?

        Come on! Show us the code, we want to have fun too! :)

        1. 1

          I wouldn’t call that spam cause you can use it for free via OPS. Also, I asked about this on IRC today. What would the difference between posting something about cockroachdb or elasticsearch or redis or any other ‘open source’ tools out there?

          1. 3

            Where’s the source for NANOS? I couldn’t find it, which lead me to assume this is not actually an open source tool. If it’s open source and OPS happens to offer a hosting service for it, that’s fine by me.

            If your sole interaction with the community is to drop a link to your employers stuff, that’s fairly spammy. When a newcomer does that, it feels like they’ve joined for the sole purpose of promoting their company (rather than for the purpose of being involved in the community).

            When people who are active participants post links to their work, it feels more like a friend saying “I did this cool thing at work, check it out”.

            Does that distinction make sense?

            1. 1

              Nanos hasn’t been officially released yet (look at the version - 0.1.x), no and I don’t know what form (read: license) it will take. However it is free as in ‘free beer’ today and will continue to be so.

              I didn’t see anything about the rules on the site that mentioned software needs to pertain to a given license. I see a lot of articles that have nothing to do with open source here. I popped on here cause it was mentioned in an article the other day here on /t/osdev -> https://dev.to/yonkeltron/the-coming-era-of-os-kernel-modularity-5emc and I noticed there was no link to the source code of the tool that was mentioned. I’m not an active member of this community as I frankly don’t have the time to be but the tool is open and you can download it for free. Are all parts open? No and neither are most ‘open source’ tools like the ones I mentioned - definitely not according to OSI.

              If a tool needs to be GPL only on this site please let me know if not I’m sorry you are offended and I apologize for breaking any rules on the site if that’s the case.

              I’m not trying to fight anyone.

              1. 5

                I don’t care about the licensing. Is the source available? Where?

                I read through the repo that was linked, and didn’t find anything that looked like it implements a kernel.

        2. 2

          Yes all ‘unikernels’ do have the notion of a kernel even though they might not call it such. We’ve developed a new one that is built with unikernel concepts in mind (eg: no users/permissions/etc., single process, etc.) It clearly has a lot of work to go but it runs quite a lot today in a much safer environment than stock linux.

          1. 1

            We’ve developed a new one that is built with unikernel concepts in mind

            From scratch?

            1. 4

              From the bootloader on up yes. It’s quite a lot of work but we felt it was necessary. We used to use Rumprun and OSv quite a bit but kept running into various limitations/issues and decided it wasn’t worth the effort to maintain forks since neither of those are super active (although Waldek is doing a great job w/OSv lately).

              We’re asked a lot why we didn’t just patch/fork linux and the short of it is because of the same reasons. Linux is measured in the >10M LOCs and it just has way too much stuff in it that is antithetical to the unikernel philosophy. However it’s important to realize that we’re not trying to replace Linux either - for instance we never plan on not running as a vm so half of what linux is (drivers) we don’t need to code/support allowing us to focus in on what really matters.

              1. 5

                Great job!

                This response should be the first thing I see when I look at the readme / and/or your business website. I spent ~10 minutes trying to find out what the hell your service does and why I would want to use it - this is the answer I was looking for.

                When you open source the kernel, please do post it here also, as it’d be of tremendous interest to the community.

                1. 2

                  “but kept running into various limitations/issues and decided it wasn’t worth the effort to maintain forks”

                  What issues and limitations were you running into?

                  1. 3

                    Well the big one with rump was 32bit only which is a non-starter for most java apps and databases and that opens up a large swath of technical decisions that have to be made (like virtual memory and paging). Also rump is composed with netbsd and so carries that load.

                    The big one with OSv was PIE only (which is not quite the case anymore but in our exp still has quite a few gotchas).

                    There are quite a handful of other technical issues but at the end of the day building our own allowed us to do whatever we wanted to.

                    1. 2

                      I could see you wanting to slim it down or use 64-bit. I assumed rump was 64-bit given the market. Only 32-bit? I couldn’t find something on the rumpkernel.org site saying that. You got a page from them or a detailed review I can link to as a reference in these conversations?

                      1. 3

                        apologies - I might be conflating issues - there was one for the 4G of ram here - https://github.com/rumpkernel/rumprun/issues/74 && for the past few years there was no aarch64 support (arm 64) but the nabla containers folks (ibm) have recently added support for that in the past few months -> https://github.com/nabla-containers/rumprun/pull/21

                  2. 2

                    That’s a rather incredible achievement, congrats! (And btw, yes, I think it would benefit you to open source that, since devops people aren’t going to run proprietary kernels if they can help it - at least I certainly have no plans to!). EDIT: consider the AGPL, it’s friendly to server-side stuff.

                2. 1

                  So, given an unikernel is running on a hypervisor (qemu in the case of ops), it is basicallg a highly privdropped restricted process (a qemu process) security-wise?

                  If a process has no exec(), and runs as non-root in a chroot, with further restriction on which syscall it can do, it might still be more complex from the inside (it listens for signals, interacts with virtual filesystem, have the large kernel mapped in memory…).

                  But a unikernel runs on top of an hypervisor, that is complex itself!

                  So, the point is not to remove the system complexity, but rather to move the complexity away from the application being developped downto something not in user’s (or attacker’s) control?

                  On the other hand, running application onto a docker host OS onto a hypervisor /is/ more overhead than just the unikernel directly onto the hypervisor with maybe some orchestrator that tell the hypervisor to start and stop unikernel VMs.

                  What about re-writing Ops as an unikernel with an authentycated TLS/SSH protocol?

                  So, do you expect people to run Ops on a bare-metal Linux (in order to reduce the stack) ? Otherwise, it is adding a layer on top and not removing one. ;)

                  I like this quote:

                  layers are only ever added, never removed

                  – apenwarr’s colleague

                  1. 3

                    Qemu is utilized by ops cause it’s the most common machine monitor out there, however there is no reason why we couldn’t roll in support for something else and intend to do so. Google made their own monitor that interacts with KVM that provides GCE mainly cause the qemu codebase is huge and supports everything under the sun.

                    As for the stack, no, OPS is intended mostly for those who would wish to deploy their applications to the public cloud - AWS, GCE, Azure, etc. although any ‘cloud’ (virtualized) environment will work. For instance when you spin up an instance in Google with this it creates a raw GCE image that does not have linux in it and we assume the majority of these users would be using these platforms.

                    I guess to be more explicit - this tool is designed more for everyday ‘work use’ like terraform or puppet for compute environments that have many machines to handle that you would find in companies versus a hobbyist use (ops -> devops). I wouldn’t run this locally for dev purposes but for production deploys. Not sure if that makes sense?

                    1. 2

                      “But a unikernel runs on top of an hypervisor, that is complex itself!

                      So, the point is not to remove the system complexity, but rather to move the complexity away from the application being developped downto something not in user’s (or attacker’s) control?”

                      If hardware support exists, hypervisors are more fixed purpose with way less complexity and code than monolithic kernels most people use. They just don’t have to do as much stuff. From there, they can be designed minimalist fashion easier to verify for correctness (eg Nova microhypervisor) or security (eg seL4). The cool thing is the isolation paradigm they implement can then pay off in all kinds of use cases.

                      On top of that, you then add the minimal amount of code necessary to support the application. Being small and clean slate, one might even use memory-safe language for it (eg MirageOS). That mitigates lots of attacks itself. Ideally, the API’s would also be made for developers to use them securely by default. From there, the extra failures show up in just the code the developer added in a nice box the hypervisor provides.

                      Disclaimer: Assumes hardware is secure. Little reason to think that. People should still try to get stuff above hardware right.

                      1. 2

                        Yes - you got it. In this security model the user and the attacker are treated much the same - again this assumes the infrastructure is either managed (aws/google cloud) or private (openstack/vmware/etc.). I think we crossed the rubicon a very long time ago when even small 20 person engineering teams have hundreds of vms they are managing. One linux is hard enough to secure.

                        As for hypervisors - we already have unikernel tuned hypervisors - IBM even has multiple patents on it. https://patents.google.com/patent/US9886303B2/en

                        As for hardware yes and no. You can “assume” it’s secure cause if someone can easily go around rooting vms on AWS we have some fairly serious issues to deal with. Having said that I agree that our current hardware is fundamentally flawed and this won’t fix that but I also contend we are stuck with it for a while.

                        1. 2

                          You can “assume” it’s secure cause if someone can easily go around rooting vms on AWS we have some fairly serious issues to deal with.

                          I assume they’re doing something different than most other groups that their resources allow them to do. It may or may not reduce their odds. They might also have better monitoring and response. What AWS deals with is not equal to or doesn’t extrapolate to what average company with virtualization can do. Maybe not even F/OSS virtualization since their local modifications might improve security.

                          So, I look at what the open products and many closed claim they do, the nature of the dependencies (esp shared resources), and the odds of leaks coming through them. The odds were high. It’s why I advocated for physical separation where possible on embedded-style boards. The leaks keep coming on server CPU’s.

                      2. 2

                        Many people have accepted the hypervisor as an implied part of their stack; in that case removing application/kernel separation means higher performance and less abstraction.

                        1. 1

                          Or even unikernel on bare metal!

                          I think that what we often ask for when we use a hypervisor is an array of similarly manageable machines that we can spin or stop.

                          It would be interesting to get unikernels spawned via PXE boot on a couple of managed hosts. Bare metal metal.

                          Spawn a unikernel on hardware like one would spawn a process. More, smaller machines.

                          No vertical scaling though…

                  2. 2

                    Another question I’m curious about: the website says that unikernels are faster because there’s no scheduler. But doesn’t the idea of running several unikernels still require for there to be a scheduler … somewhere? And so there must be a scheduler still, just moved up a level? Can you comment on the difference (if there is one)?

                    1. 4

                      There are two ‘schedulers’ here and one of them is something that is implementation specific (there are north of 10 different unikernels out there today).

                      At the app/kernel layer we definitely support multiple threads but it’s single process (since there is no intention to run other programs). There are also single threaded unikernels out there. Some would consider that more of a purist style, however, we find that limiting since most people coming from a linux world, even using a single-process, single threaded program would still have access to multiple threads inside the kernel. Whether you go cooperative vs pre-emptive threading is kind of a different argument but a great (recent’ish) paper on the subject discusses problems with the way Linux does things https://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf . I point it out because again - Linux was built with the implication that it would be running on bare metal supporting multiple programs ran by multiple users - these constraints are broken when it’s one application per vm.

                      At the orchestration/vm/cloud layer that is done with tooling around the hypervisor. If you spin up a unikernel using OPS on google cloud for instance you’ll spin up a GCE image and google cloud runs a modified/forked kvm. So they will schedule that vm as they see fit on their servers. You the user can then orchestrate several of them together (eg: a database, a webserver, a background worker, etc.) into the same vlan using higher level orchestration tools like ops or terraform or some other yet to exist tool.

                      So scheduling in these two senses can be like physics - what works in newtonian can be radically different at the quantum level.

                    2. 2

                      What gives it that security?

                      Single Process

                      Oh yes, process isolation is definitely not a security feature (/s). And of course no word about binary exploit. What about an attacker that can jmp into the whole kernel code ?

                      No Users

                      Oh no, users privilege separation is definitely not a security feature (/s). If we remove them we can’t bruteforce their password. We immediately log in as “god” instead (better than root, you’re the kernel).

                      [the shell,] It’s an antiqutated concept that only lends it’s hands towards those who want to do your company harm.

                      So you are adressing this speech to company-owners that have no idea of how UNIX-style computer works? I noticed that when you offered to expose kernel memory address rather than the system shell. But no shell means harder for an attacker to exploit an existing breach (or an admin to debug), true.

                      Reduced Attack Surface […] Less code == less exlloits

                      You made a point here. Kudos.

                      1. 2

                        P.S.: I still like the idea of unikernels.

                        1. 2

                          Not sure about the first point but to elaborate there is a single process and no facilities to create another process (no fork/exec). Also, unlike other unikernels we actually do have the concept of cpl0 and cpl3 and have opted to utilize vdso style ‘speedups’ where it makes sense. Also, we are explicitly addressing ‘binary exploits’ (eg: shellcode).

                          As for users - you can’t login to these - at all. There is no ssh/telnet. Half the user syscalls are merely stubs. You can’t su/sudo. The entire concept of a shell doesn’t make sense in an unikernel since it already means you intend to run multiple programs.

                          While this can execute linux programs I would be hard-pressed to call it a unix.

                          If you are curious about more of the security properties I wrote up a short blogpost on it that is more technical.


                          1. 2

                            From that blog entry (which I certainly wouldn’t accuse of being “technical”):

                            This is reminiscent of the “unikernels are un-debuggable” points brought up earlier. This is funny too considering one of the authors started working on an [sic] unikernel specific debugger…. For real.

                            Is that a reference to my blog entry on unikernels being unfit for production? On the one hand, I can’t imagine it is – I am certainly not engaged in any unikernel debugger, and my piece was clearly single-authored – but on the other, I don’t know what else this could be a reference to. Could you clarify?

                            1. 1

                              hi Bryan :)

                              It wasn’t directed at you. One of the NccGroup authors of the paper in question ended up writing a unikernel debugger because at the time there was no gdb stub to hook up to whatever implementation he was looking into.

                              1. 1

                                Is it not easier to debug when the source is fully available ? * hum hum * ;)

                            2. 1

                              Nice to know there is some privilege structure that remains. The article linked talks for itself.

                              But I get the idea : noone will ever log (or break) into the kernel and an attacker will have to think again (I still don’t consider SSH with a strong key and no password a large threat).

                              It looks like a reversed approach: instead of dropping privileges as the program goes (changing for a different user than root, chroots, pledge, capabilities, read-only filesystems…), do not expose the vunerable resources on the first place: filesystem privileges can be implemented by a fileserver in which the unikernel logs in, or that only expose (or have at all) the data the application is allowed to access.

                              But there are many cases where there is a need to parse complex strings (risky) theb do something, then send the result back.

                              To mitigate the risk, it is possible to split the job into multiple processes : one that parse, one that does and send back, so that the risky work of parsing the user input gets done in a separate memory space and process as the one that have full access to the protected data.

                              How to do this with unikernels? Many tiny unikernels with networking to communicate rather than a pipe()fork()? Then there is a need for strong firewalling between the two or implementing authentication and crypto for every pipe replaced. This adds complexity and TCP overhead and require more configuration so that the two point at each other.

                              Yet it permits a much more distributed system where it is possible to have a lot of distributed parsers and a few workers or the opposite.

                              1. 1

                                Threads! Maybe there is some way to limit privileges per thread (I’ll read the doc). Then while it is still the same memory space, one thread that have no network access, one thread that have no storage access. Woops, am I baking thread-ish processes? Not the good approach for unikernels…

                                I’ll think again…

                            3. 2

                              In my part of the industry most software is deployed into a vm where it is the only thing running. Having access to that one thing is functionally equivalent to getting root on the box.

                              1. 1

                                In my part of the industry, it’s all vms running debian running apache running php running a framework running a website. In my part of the industry clients get their wordpress plugin hacked in multiple ways every day. I’ll probably quit my part of the industry.

                                At works, things are managed from the inside of the system. Each have an agent that does the monitoring (a la puppet).

                                But for every hack I’ve seen (not that many, it’s not on me to spot them…) there were basically no attempt to get root.

                                1. 2

                                  If you’ve already gained control of the website, what point is there to getting root?

                                  1. 1

                                    Yes, exactly! Not that much point!

                                    Well still, attacking other machines, take over the infrastructure some more or attacking the hypervisor (VM escape)… But I bet the main target is taking over “whatever the client is doing” (taking over the application running as non-root).

                            4. 1

                              It gets me to wonder if it would be possible to use the kernel text console output from the hypervisor for the purpose of logging.

                              1. 2

                                by default that’s exactly what we do currently although we highly recommend for prod services to ship it to whatever log provider is being used (papertrail, elasticsearch, splunk, syslog, etc.)

                                1. 1


                                  I spawn daemon | s6-log, with the logger that does:

                                  • writing to a file with timestamp for each line,
                                  • pattern matching filter useless logs away, or post some lines for further processing (send alerts…),
                                  • rotate / compress logs as soon as the written size exceed the wanted size and/or date,

                                  This makes logging extremely simple for everyone, flexible, trifial, lightweight, easy to implement, to configure, uniform across all daemons (or qemu vms), ubiquitous (everything have a mode to log to stderr)…

                              2. 1

                                Uh, curl | sh

                                1. 2

                                  Well you can always build from github if you’re concerned but if you are downloading programs from a website and you don’t trust the website don’t run the program. We do have open issues for building debs/rpms and such but it’s one of those things that happens when enough people care.