1. 15

    Interesting project, but this marketing BS is not a good link for Lobsters. Here are the very basic questions I had about Nanos:

    • Does it descend from some pre-existing system?
    • What language is it implemented in?
    • How is it licensed?
    • Who supports its development, and what is their business model?

    To answer all but the last of these, I had to click around and find the github repo. The FAQ on this marketing site is pretty vacuous. “Syscalls: N/A” lol.

    Anyway, I think I’ll stick with https://mirage.io/ for my unikernel needs. But thanks for sharing!

    1. 5

      Anyway, I think I’ll stick with https://mirage.io/ for my unikernel needs.

      I only just looked into this stuff but… all of MirageOS’s peers are dead. So I think I welcome anyone new in the space. We need progress and competition!!!

      1. 3

        Good point, but it perhaps says something about the unikernel concept itself and how it’s played out in practice. Overall, I agree, and I’m happy to see Other People’s Money being spent on a technical subject I personally find interesting. I sincerely wish the nanos folks all kinds of success.

        At the same time, I must admit to having some doubts about the wisdom of implementing a unikernel with a completely fresh code base in a famously unsafe language, but hey, I guess we’ll see how it plays out.

        1. 2

          The money point is important to point out. It’s not something you can just whip together in a weekend and not something I could see happening without lots of $$ to employ full time engineers on. Even then the difference between hello world and production usability is a huge gulf to cross - most teams unfortunately die before they can cross it. That in my most humble opinion is the biggest problem in the ecosystem. Government grants can only take you so far.

          I hear your complaint against c. We’ve discussed doing other complementary projects in rust but the pkg/dep system is a major turn off and none of our engineers speak rust either other than hello world projects. Likewise $$ again is a prime concern for working in that ecosystem. When you have tiny little saas companies that raise tens of millions and employ tens to hundreds of software engineers you have to ask what is the true cost of a new language or a new operating system?

          1. 2

            All very cogent points, and be it far from me to armchair-quarterback anyone’s business strategy. Application support is probably the first priority for a real-world unikernel, and that drags in legacy APIs and other ecosystem factors that we can’t really control. Latent security problems only become critical when you actually have some customers.

            I’ve had a corner of an eye on this space for a little while, but I wouldn’t claim any expertise. To me, it seems like Mirage has carved out a bit of a niche and, like many open source projects, is sort of puttering along just fine with little in the way of sponsorship. But its adoption is of course limited to the (relatively tiny) Ocaml ecosystem. Then there’s the rump kernel approach, which largely piggybacks on NetBSD’s flexibility and mature-ish code base. Looks like there’s some consultancies promoting that. Probably works great for some applications. Doesn’t appear to need VC, at a glance.

            The only unikernel I have had any real first-hand experience with is HalVM. I can assure our readers that no realistic amount of government grants or eager first customers could have saved it from drowning in its own space leaks. Haskell itself is largely subsidized by the academic tenure system, and is thus well insulated from its own practical failings, but that’s another story entirely.

            1. 2

              Agree with most of this, however, Antti spent something like well over a decade fleshing out rump - and that was on top of netbsd which was forked 27 years ago so I wouldn’t agree that the ecosystem doesn’t need financial resources. The EU is trying to pump some money into the ecosystem thankfully. There indeed are a few consultancies but they tend to come and go (again lack of resources) - I keep a running list.

          2. 1

            in a famously unsafe language

            Oof. I only just noticed it’s in C. Okay, I’ll root for it, but hopefully they’re making up for that handicap.

        2. 5

          This site is a WIP - literally went up yesterday.

          Also, this is not intended to be a marketing site - it’s a community site. I was curious if the lobsters crowd would cast it as such but since I have seen plenty of other sites like ziglang with ‘donate now’ buttons figured one link on a community site wouldn’t be ‘marketing’.

          As for the questions:

          1. No. From the ground up it’s written in scratch - github.com/nanovms/nanos .
          2. C (mostly)
          3. Apache 2 - https://github.com/nanovms/nanos/blob/master/LICENSE
          4. NanoVMs - (the real marketing site ;)
          1. 5

            If it’s a WIP, why did you post it here?

            A donate button doesn’t make a marketing site, but there’s almost no substantial content — barely a synopsis — and it is indeed mostly trying to convince you to use it. That’s marketing.

            1. 6

              Are WIPs not allowed to be posted to Lobsters? Zig is a WIP, it’s at 0.6.0. Can we not link to a Zig release page?

              1. 5

                No, of course it’s fine; the point is someone said “weird that this page is missing these basic things” and the author replied “the site is a WIP, it only went up yesterday” to explain why these basic things are missing. Both are talking about the site, not the software itself, because they’re talking about the submission, not the product, which is a distinction so many people in this thread can seemingly not make.

                Lobsters is about discussing submissions, right? If you feel the need to defend your submission by saying “it’s a WIP, it only went up yesterday” when someone points out key information is missing, I don’t think you should have posted it yet. What’s the rush?

              2. 3

                If it is not appropriate I apologize, although, I feel lobste.rs should post a guidelines document as I see a lot of links posted that have commercial links back and I can’t think of a single OSS project that is commercially supported that doesn’t have a link back.

                Just now I found the following on lobste.rs - all that had more than one link going to a commercial post:

                https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/ https://android-developers.googleblog.com/2020/09/android11-final-release.html https://blog.ipfs.io/2020-09-08-nix-ipfs-milestone-1/

                1. 7

                  Like I said, the donate button isn’t what matters, and the examples you’ve given serve my point: they are chock full of technical content and detail, which is what we want to see here. This submission has almost no substance.

                  1. 4

                    A new operating system and a new file system aren’t technical enough?

                    This “submission” is backed up by multiple repositories of open source code:

                    https://github.com/nanovms/nanos

                    Where is this aggression coming from?

                    1. 15

                      There’s sincerely no aggression; you’re asking about what’s appropriate and I’m doing my best to help you understand why (from my point of view) I feel this isn’t an appropriate submission. I think my opinion is at least somewhat representative of community norms based on votes. I’m not attacking you, I’m sharing my context. You don’t have to agree with my assessment.

                      The page is light on detail and mostly serves to advertise. That’s what it comes down to. I don’t think this submission adds anything. The direct repo link would always be better, or a technical detail post.

                      And yes, you posted the repo a little while ago, and that’s okay. If there’s a new release, link to the patch notes instead. Reposting the same link isn’t forbidden either, especially if there’s been big changes.

                      1. -4

                        I think a lot of people would find these comments aggressive - maybe you can disclose your real name and who you work for. :) (That’s ok, you don’t have to.)

                        You are most definitely correct that it does to advertise the community aspect of the operating system. I’m sorry if you are looking for a corporate only POV. I won’t advertise it here but you can easily find it.

                        I’m glad you don’t find patch notes offensive - I’d find any patch notes offensive that are offensive. Nanos.org is a brand new site so sorry no new changes.

                        1. 14

                          I actually found your comments to be on the aggressive side, while others tried to explain their views on why your post is flagged. And the length of their posts implies it’s a sincere efforts.

                          1. 10

                            maybe you can disclose your real name and who you work for.

                            This is in bad faith. Additionally, my real name and employer are trivial to locate. I really think you should reevaluate your angle here.

                            I’m sorry if you are looking for a corporate only POV.

                            I have to believe you’re being intentionally obtuse in light of my suggesting “the direct repo link” or “a technical detail post”. I’ve really tried only to be kind in my comments here and represent my views to get to a shared understanding with you, but you seem to consistently engage in hostility and points raised while further stirring things up.

                        2. 4

                          Where is this aggression coming from?

                          There seems to be a misunderstanding here. I think absolutely nobody is objecting to submitting Nanos, that would be ridiculous. People are objecting (although I disagree) to submitting the URL https://nanos.org/, instead of (more relevant) URL https://github.com/nanovms/nanos.

                          I hope this makes things clear.

                          1. 1

                            The nanos.org website, is strictly for providing a place for the community to place a voice outside of the company. One of the reasons of having a community site was to move a lot of knowledge that was in engineers heads to the community in the hopes that it would not be lost.

                            The very fact that it is a .org and not a .com, should be painfully clear but in 2020 I realize that is just not something that works with everyone.

                            Just as kernel.org is an .org, yet retains logos from Fastly, Packet, Redhat, Google and many others, nanos.org is an open source site with source code found elsewhere.

                            Is this such a problem?

                            1. 5

                              Submitting a site structured like kernel.org would indeed meet some annoyance, because there is nothing to read on this page, and not clear which of the links you considered most worth following. You could pick one previously unposted of the many pages you have mentioned with technical substance and say in the comments that «We are currently working on building a community site on nanos.org and making the linked material available (or updating it) was a part of that work». That would look in a different way.

                              Note that a page that looks like a pure advertisement for a community is still pure advertisement, even if of a slightly different kind compared to commercial advertisement.

                              I guess you could imagine the following imaginary (or is it?) use case for Lobste.rs: automatically download the linked articles and separately comments automatically and read them offline. If changing what you link to and mentioning the original link in the first comment would increase the value for such a use case and not clearly decrease the value for the more typical use, it is likely that such a replacement would also improve the reception here.

                              1. 0

                                Submitting a site structured like kernel.org would indeed meet some annoyance

                                Are you kidding me?

                                I guess I was wrong in posting a free/open source community based website to lobste.rs - I’ll refrain from that in the future. So much hostility.

                                1. 3

                                  Strictly speaking, you were submitting a page, submissions are not really read as sites — there are surely many good pages on your site to submit.

                                  1. 1

                                    I still don’t understand the frustration here. Is the main page not the most direct way to announce a public free open source tool that did not have it before?

                                    If you had never heard of rust before aside from a few coworkers chatting about it and rust-lang.org didn’t exist but then it pops out of existence you wouldn’t find it appropriate to post?

                                    1. 5

                                      rust-lang.org goes in somewhat more details than the current state of nanos.org, and I guess a more detailed page about what choices Rust makes would be a better choice than the main page.

                                      Many of us want to click the link and find a page containing a text about something technical that directly convinces us to care. Currently nanos.org tells me «Nanos is a unikernel» and nothing else (sure, almost every unikernel will mention that avoiding context switches will improve performance in read()-bottlenecked tests), then there are some links. I can easily understand people preferring to get a submission where the main link goes to a substantial text, not to a place where you need to find out which link to click to get the text. (I personally did not flag this submission, though)

                                      1. 2

                                        I 100% agree there should be more detailed documentation and there most definitely will be, it’s a work in progress.

                                        I suppose all the frustration can be summed up with “come back when there is X bytes of technical documentation and link to that rather than the front page”. Noted.

                          2. 3

                            Literally nothing about the link you used for the submission is technical at all. It’s a marketing splash page.

                            1. 1

                              Literally? Nothing?

                              https://nanos.org/thebook isn’t technical at all? After one day? With links to pull requests of code? and binary serialization formats for a non extX filesystem?

                              Please.

                              1. 5

                                Then why didn’t you link to the book? Remember, I said the link you used for the submission.

                                1. 2

                                  It’s a link from the main site…

                  2. 1

                    This community has become so fucking toxic! It’s really sad that I now no longer enjoy reading the comments. I use to come here for the fun technical discussions. But once again the internet has ruined another great website. jcs should have kept everything invite only and small. No wonder he gave up and walked away…

                    1. 2

                      I think what you’re witnessing is the site adjusting to a course correction toward its original purpose. A few people were making a push toward getting more ‘culture’ related content on the front page, and that’s not what this site has ever been about. Some folks have been pushing back.

                      Fortunately, in the last few months I’ve seen more interesting material hitting the front page than in recent years, and the comments are still well worth exploring (in those threads). Anything resembling “toxicity” comes in ‘culture’ tagged articles.

                  1. 3

                    I was surprised to see it uses lwIP as a network stack. Contrary to my preconception, maybe lwIP is fast enough?

                    1. 3

                      Full disclosure: there’s (very little) code of mine in nanos, but I haven’t worked on it/used it in about 1.5 years now – I wanted to but there just weren’t enough hours in a day for it :(.

                      Back when I used it, IwIP wasn’t super-fast, but it was pretty easy to work with and pretty flexible. I haven’t ran any serious benchmarks (i.e. on dedicated networking/serving hardware), but I did run some basic tests on some of my development machines and it wasn’t dog-slow. I don’t expect it would be useless for production workloads, albeit there’s likely quite some room for improvement.

                      1. 2

                        Thanks for your contributions! Perf was not top of mind at that point in time and is actually one reason why nanos.org didn’t exist back then but a few of the ft’s have made some serious progress. When brian added ftrace it really revealed a lot that was hard to see before.

                        1. 2

                          FWIW, I think lwIP really is the best choice here, and as I mentioned above, I don’t doubt that it’s adequate for production workloads. There’s always room for improving any networking stack, but lwIP itself is the result of a lot of super solid work. Coming up with something better than it is, in and of itself, a project of considerable magnitude, comparable to developing a unikernel itself!

                          I think it gets a bad reputation because it’s used in tons of bad firmware, often running on top of bad hardware. I haven’t done it myself, but I know of at least one case where poor performance was traced not to lwIP, but a buggy Ethernet MAC implementation in the FPGA below it.

                        2. 1

                          wasn’t super-fast, but it was pretty easy to work with and pretty flexible.

                          This is fantastic reminder that often we need to balance usability (and ease of use in general) with other things. Kudos for being honest and open about trade-offs.

                          1. 2

                            Indeed! See my other reply for some additional details – like I said, my involvement with nanos has been minimal (a few weeks?) and it was more than an year ago, so it’s not really “my” trade-off to comment about (and I know things have improved considerably – see the parent of that post of mine). But in general – and, for what it’s worth, in this case, too – I think this is a very valid trade-off to make. lwIP offers adequate performance for many (most?) of the things for which you’d use a unikernel running on virtualized hardware, and it’s pretty solid and easy to integrate. It takes years for a network stack of any performance level to reach lwIP’s maturity, doing that in the early stages of building a new kernel would be both unmanageable and pretty much useless.

                        3. 2

                          lwIP was chosen for speed of adoption not necessarily for anything else - I fully expect us to evolve to something else in the future

                          1. 2

                            I don’t really know how to classify a network stack as fast, but I’ve worked with LwIP and we’ve pushed it to more than gigabit speeds in synthetic benchmarks (although we weren’t benchmarking LwIP but rather some other code around it).

                          1. 2

                            Nanos looks very interesting, thanks for sharing!

                            I don’t get why some folks are annoyed about you posting a link to the site, it gives me a good idea about what Nanos and the ops tool do, at a glance.

                            1. 3

                              Really? I have literally no idea what Nanos is or does except “runs applications” on a unikernel. Would you mind sharing what it is Nanos actually does?

                              1. 5

                                Nanos is a Linux syscall compatible unikernel. It seems to be a less mature OSv, another Linux syscall compatible unikernel.

                                1. 3

                                  Thank you, the OSv site was really helpful.

                                  1. 2

                                    I always forget about OSv but it does look pretty awesome. Do you also know about UniK?

                                2. 3

                                  Mind sharing your good idea? What does Nanos do, and what does the ops tool do? I couldn’t tell, at a glance. I didn’t find “run code faster than the speed of light” to be informative, the FAQ didn’t say much, and all 600-ish words of “the book” tell me it’s a unikernal that’s opinionated about security but doesn’t say what it’s good for or why I might want to use it.

                                  I could tell from “getting started” that they wanted me to pipe some script from curl to sh, but I didn’t want to do that, especially before I understood what those things do and why I might want to use them.

                                  1. 3

                                    Nanos is not Linux (it is an independent implementation, so for example, as noted above, it uses lwIP as a network stack, not the Linux network stack), but it can run Linux binaries, by emulating Linux’s ABI and syscall interface. It claims to run Linux binaries better than Linux in some sense, for example by being faster. Nanos probably can’t ever be better in Linux binary compatibility than Linux, but it claims to be good enough.

                                    1. 2

                                      But it claims it can’t run on bare hardware, right, so I’d need to run it on top of KVM or Xen, which means I’d need to run Linux also? Even just to fire up a VM on VMWare workstation and try it out?

                                      And “the book” claims it has no syscalls…

                                      How is it faster than running Linux binaries than Linux is, if I need to fire up KVM or Xen in order to bootstrap it? Why wouldn’t I just statically link my binaries and directly run them on the host, sandboxed?

                                      I’ve spent 5 minutes looking at the github site, which is a little more helpful than the site OP linked, but I’m still not sure what my use case is for this yet, even though it does sound interesting.

                                      1. 2

                                        Let’s say you were running Linux on AWS virtualized (very common scenario). Since you weren’t running on bare hardware anyway, there is no disadvantage to switching to Nanos on that scenario, from that reason.

                                        It has no syscalls in a sense that there is no transition from user mode to kernel mode. Your code runs in kernel mode, and Linux syscalls are emulated as function calls.

                                        I haven’t done benchmarks myself, but it is very believable Nanos runs some Linux binaries faster than Linux, since it does less than Linux. As I noted above, there is no syscall transition, which is often a significant overhead. There is also no users and no processes: you are the only user, and your code is the only process. This does greatly simplify what the “kernel” (quoted, since technically your code is kernel) needs to do.

                                        1. 2

                                          We actually do keep CPL for no other reason that if you have certain pages that are say only readable you don’t want an attacker to make them writeable (and that still gives us a speed boost). Context switching can imply a few different scenarios depending on context (heh!). Kernel -> User, kthread to kthread, user -> user, etc. We’ve found that a lot of the heavy switching is actually a result of the fact that modern GPOS like linux have hundreds of user processes upon boot even if the intention is to only run a single application (which is very common in server-side deployments).

                                          1. 2

                                            Thanks for the correction.

                                            Since you are here, I want to ask why should I run Nanos instead of OSv. Since two seem to be in a similar niche, a good comparison would be welcome.

                                            1. 3

                                              First off, I very much respect the OSv engineers, however, the corporate side of that has moved on to https://www.scylladb.com/ which makes governance a bit of an issue (for us). There are a handful of architectural differences. We have strived to be as binary compatible towards linux as possible (elf loading, using whatever libc the app is linked to, etc.) We also approached the niche from a security POV, versus the performance view so that’s where we’ve focused. It should be noted that all of these are very much operating systems just not general purpose operating systems and so each one has a lot of engineer hours that go into it. So each system excels in various areas where others might be deficient. This is compounded by the fact that most unikernel enthusiasts are motivated by different goals/needs (NFV, edge, performance, security, etc.).

                                          2. 1

                                            I think where I’m getting lost is looking at the proposed getting started workflow captured in this image:

                                            https://nanos.org/static/img/terminal-2.png

                                            That looks to me like it’s running on a linux host, and I think that is contributing to my confusion, if the real idea is that I should be replacing Linux with this on some public cloud hypervisor. Which leaves me needing to go understand the ops tool, I think, in order to assess whether this might fit into my workflow or offer any benefit.

                                            Thanks for jumping in and explaining… I hadn’t spotted where in the stack this fits, and that’s helpful.

                                            1. 3

                                              The image indeed shows running on Linux host. As I understand, it is for convenience, and it launches KVM behind the scene.

                                              The real idea is indeed replacing Linux with Nanos on public cloud. This chapter on how to create AWS image should make things clearer.

                                              1. 1

                                                The ops tool is complementary. You can envision it as something comparable to terraform/chef/puppet (but not really). The idea is that most end-users provision their web application software (eg: all websites) to various public cloud providers which we support every single popular one. There’s a lot of api involved that is not proper to put into the core kernel nanos and if you dive into nanos further you’ll find things like the filesystem manifest which for real applications can get rather large (think jvm or python applications). Ops also helps in this regard by generating that on demand.

                                                ops can be ran on linux and mac today and we have someone working on windows as we speak.

                                            2. 1

                                              It very much does have syscalls. That section is empty because it needs to be filled out. There are some syscalls we support 100%. Some are stubbed out (on purpose). Some are implemented differently.

                                              You are right it most definitely is not written for bare metal. It is written for most production web application environments which run predominately on the public cloud (aws, gcloud, etc.) which use 2 layers of linux. One for the hypervisor layer and one for guest. This replaces the guest layer.

                                              1. 2

                                                Thanks. That clarifies it some. I think I need to understand some details about the ops tool in order to evaluate further. If you’re working on the site, that might be a good thing to expand on.

                                                1. 1

                                                  For sure - thank you for the suggestion!

                                          3. 1

                                            One of the whole reasons for the site is to provide more documentation that would be hard to grok from the code alone.

                                            Nanos is meant to be a server-side only system of running linux applications with a focus on performance and security. For the install script - it points to https://github.com/nanovms/ops/blob/master/install.sh which is viewable/editable on github. You don’t have to use it and can opt to build from source instead.

                                            1. 2

                                              I think I understood that the intention of the site is to provide info that would be hard to absorb from just looking at a git repository. The feedback I was offering is that the “what” and “why” information that might make me want to try it out isn’t robust enough for me to understand yet, and the “getting started” information that is on the site leaves me needing to go pick through a git repository and read the source code for an install script/orchestration tool in order to feel comfortable that I understand what it claims it will do before I run it.

                                              1. 2

                                                I fully agree that we need/want more documentation/information - that’s the whole purpose of the site. We have decent documentation for OPS https://nanovms.gitbook.io/ops/ that is fully PR’able on github but that provides no justice for Nanos itself which is way more technical and since OPS isn’t the only horse in town we wanted a central place to detail nanos much like a rust-lang.org .

                                                1. 1

                                                  Nice overview, thanks

                                        1. 5

                                          Nanos serves static content almost twice as fast as Linux

                                          Tested with Go (net/http)

                                          If you’re testing static content, is there a reason to use something other than nginx?

                                          1. 2

                                            We have nginx and many many other applications available as packages. (think apt-get).

                                            The focus on go is that we have a lot of random infrastructure tooling written in go and so is probably the best supported. Both ops.city and nanos.org are go unikernels running nanos. ops has been up since more than a year now?

                                            We have a project to do automated performance regression analysis and reporting so it’s our hope that this becomes extremely transparent/reproducible in the future.

                                          1. 1

                                            Run code faster than the speed of light

                                            That’s literally impossible.

                                            1. 2

                                              it’s a superlative (inserted by the designer) ;)

                                              1. 1

                                                Why so narrow-minded? https://en.wikipedia.org/wiki/Quantum_dot_cellular_automaton (for the humor-deficient among us, I’d like to clarify: that’s “humor”)

                                              1. 3

                                                The security arguments against static linking aren’t about managing vulnerabilities. It’s about things like the lack of ASLR:

                                                https://www.leviathansecurity.com/blog/aslr-protection-for-statically-linked-executables

                                                1. 5

                                                  You can do ASLR / PIE executables with statically linked programs. According to this article, it’s statically linked glibc that’s the issue, not statically linked programs in general. Here’s a proof of concept of statically linked PIE executables with zig. It didn’t land upstream yet, but it works fine.

                                                1. 8

                                                  It’s silly to assume Linux.

                                                  There’s great options such as Netbsd rump kernels, to build unikernels on.

                                                  1. 8

                                                    It’s on tens of millions of machines, powers the clouds, and who knows how many smartphones. It makes sense to build on such momentum. In contrast, building risky projects on other projects with low popularity usually leads to the new project not going anywhere.

                                                    1. 4

                                                      At the end, marketing reasons, not technical ones.

                                                      1. 8

                                                        Partly marketing. The massive codebase, API, drivers, and Linux-compatible code are definitely technical reasons to build on it. Even vendors of separation kernels all did Linux VMs for such reasons.

                                                        1. 11

                                                          I think it is worthy to remember that a lot of unikernel stuff current in the wild is not Linux. Rumpkernel, mirage, and others are pursuing different approaches and they are being successful. Unikernels are very specialized and sometimes the things you listed as advantages are not where the focus of a given project it. It might be the case that mirage is more attractive to a project than some linux based solution.

                                                          1. 5

                                                            I think this point is under-appreciated. There are something like over 10 unikernel projects out there and almost every single research paper on them (which is not a small set) forks/tweaks existing projects into new ones.

                                                            1. 4

                                                              This kind of discussion is why I love this site. I’ve learned so much from the comments.

                                                            2. 5

                                                              they are being successful.

                                                              I think this would be the main point of contention. The stuff I mentioned has massive uptake, reuse, contributions, and tooling. Unikernels themselves are barely a blip on the radar in comparison. The projects using them probably have nowhere near as much revenue and contributions as Linux-based products and projects. Successful seems a stretch unless they’re aiming for gradual progress in their tiny niche. It’s also possible I missed major developments while I was away from security research.

                                                              This isn’t to discount any of their work. I like each project you mentioned. Anti’s work even happened with the system incentivizing him not to do it and griping at him for it. That’s an extra-respectable sacrifice. To further illustrate, if this was languages, this would be one group saying they were building their own platform gradually vs another saying it integrated with one or more of C/C++/.NET/Java/JavaScript’s runtime, libraries, and tooling. I’d see the latter getting more uptake for technical and marketing reasons even if there were painful consequences to that choice.

                                                              All that said, I don’t know that I’d build a unikernel out of Linux. The Poly2 and OKL4 approaches of using pieces that help in isolating ways makes more sense if I was digging deep. OKL4 let you do native drivers/code, stuff in VM’s, and virtual drivers connecting to actual drivers in Linux VM’s. That last thing for drivers, syscalls, or apps is how I’d use Linux code on top of lightweight hardening for it. The unikernel or whatever could run beside it isolated somehow. IIRC, Poly2 folks were straight-up deleting code they didn’t need out of the OS. An eCOS-like, configurable fork of Linux that made that easy always seemed like a good idea.

                                                              Too sleepy to say more. Just some random/organized thoughts on all that before I pass out. :)

                                                              1. 2

                                                                Reading your first post again, I now get a better sense of what you call technical aspects.

                                                                So I stand corrected: then are not “more like economical then technical arguments”, but more akin to “technical arguments entangled with an economical factor”.

                                                                Related:

                                                                There are a few strong indicators that Linux is entering a new era and slowly but steady get more aspects of a microkernel

                                                                https://fosdem.org/2020/schedule/event/linux_kernel/

                                                                1. 4

                                                                  “technical arguments entangled with an economical factor”.

                                                                  You can say that. It works for unikernels a bit if you look at any startups taking VC money to make one happen.

                                                                  Thanks for the link. I’ll check it out when I can.

                                                                  1. 2

                                                                    There are a few strong indicators that Linux is entering a new era and slowly but steady get more aspects of a microkernel

                                                                    That talk did rub me the wrong way, for its vast ignorance of the state of the art.

                                                                    They didn’t even seem to understand what a microkernel is nor what the point of the approach is, which is quite unacceptable past 1995, when the famed Microkernel Construction paper by Liedtke was published.

                                                                    This is a trait I found to be very common among ignorant Linux fanbois, who mostly do believe that the so-called Tanenbaum-Torvalds Debate was somehow “won” by Linus. The reality of the situation is that, unfortunately, while technically competent in other areas, Linus himself is almost as out of touch with decades of microkernel advancement as the fanbois are.

                                                                    1. 2

                                                                      At some point, Linux taking over everything and anything hides smaller jewels around.

                                                                      1. 2

                                                                        Also in FOSDEM and much more interesting was this track.

                                                                        I particularly enjoyed Gernot Heiser and Norman Feske’s talks.

                                                              2. 3

                                                                A massive driver and code base and support for everything under the sun feels more like “economical” than technical arguments. Also, a large part of that code base is written for POSIX rather than specifically Linux.

                                                                1. 9

                                                                  Linux lacks stable driver APIs. That’s a strong argument against it.

                                                                  1. 4

                                                                    Linux has drivers, that is a strong argument for it. You don’t seem to like it very much I’ve noticed in earlier comments, I don’t really see why given that the thing is open and free and as such can be used as a source of inspiration at least, a source of code in the end for a myriad of applications. Nobody is forcing the use of Linux, if a BSD or one of those fancy verified systems I’ve seen you mention in earlier posts fits the bill those can be used as well. As long as the source is available for tinkering and the licence is amenable to that I’d say hallelujah and get hacking.

                                                                    1. 4

                                                                      The point of Unikernels is that you do not need that much drivers anyway, as most of the devices that will be available will be virtualised anyway. So the amount of existing drivers is not an argument. And as you will be working on the lowest level, lack of stable kernel API is a stronger argument than the amount of available drivers.

                                                                      1. 1

                                                                        Good point on virtualized hardware vs drivers.

                                                                        So unikernels become about changing the API with the ecosystem from syscalls to generic virtualized driver interfaces (to wrap akin a libc wraps syscalls).

                                                                        There is more work to do, more overhead to talk to neighbors (network, network everywhere, but also makes all IPC “network transparent”), but also not as much as if a whole kernel with the whole set of driver had to be written.

                                                                        This depends on programming style also: do you use multiple small programs that get composed together (no, incompatible with unikernels), or big daemons that communicate with other big daemons (yes, compatible with unikernels)?

                                                                        “You” is generic here, not particularly hauleth.

                                                                        1. 1

                                                                          Also, I do not think you can use an unikernel to act as a driver? Maybe though you can use an unikernel to be focused on hosting a driver for some hardware, then exporting a service (over the network? what else to communicate to an unikernel?) to the rest of the systems (thus not requiring the network).

                                                                          This can also be achieved with a process though: for instance, plan9 mostly supports one type of filesystem: 9p, and for all other fs types, a daemon reads the /dev/… block device and export an 9p filesystem stream (let’s say an UNIX socket*) to mount as 9p.

                                                                          • Plan 9 does not have UNIX sockets per-se
                                                                          1. 2

                                                                            Also, I do not think you can use an unikernel to act as a driver?

                                                                            Refer to Netbsd’s RUMP kernels.

                                                                            then exporting a service

                                                                            You’ll find the more you advance in these thoughts, the closer you get to a pure microkernel multiserver design.

                                                                            From my perspective, unikernels are the technical representation of realizing something is wrong with UNIX’s design then trying to reinvent the wheel, without doing the necessary prior research on what’s available out there.

                                                                    2. 2

                                                                      Now that gets technical and interesting! :)

                                                                      Happy if you can hint me somewhere I can read about it, otherwise I’ll end-up facing it after some time…

                                                                      1. 2

                                                                        I don’t have any link at hand to some great article about the (bad, there’s also good) side effects Linux’s lack of driver APIs, but here’s some advantages of having driver APIs:

                                                                        • As long as the driver API is still supported, the driver that works today will continue to work in the future. This is very visible on e.g. the BSDs, Illumos and Haiku (where BeOS drivers do work).
                                                                        • As the interface between the driver and the operating system is clearly defined, which avoids families of bugs to do with misunderstandings there.
                                                                        • This does also facilitate portability with other OSs, both by facilitating the porting to another API, or by a different OS providing the same APIs.
                                                                        • And it does facilitate POLA and thus security, as with the right system design, the driver API can act as a barrier between the driver and the rest of the system, including hardware itself.

                                                                        And a disadvantage that gets mentioned a lot is that having driver APIs supposedly facilitates closed drivers and removes the incentive for open ones. As BSDs have excellent hardware support, there’s much doubt about whether this is true, but it’s also possible that this is in some form facilitated by the popularity of Linux, which does not implement driver APIs.

                                                                  2. 3

                                                                    Because a thing is everywhere does not mean it is good. Because it has a massive codebase does not mean it is good code.

                                                                    1. 5

                                                                      You’re talking abstractly instead of concretely about the specific project. It runs in everything from top-of-the-line smartphones to desktops to supercomputers. It must be good for something. The alternative(s) may be better in these use cases depending on what they’re trying to achieve, though. For instance, NetBSD’s codebase is supposedly designed to be very portable and easier to understand. Pieces of its code have been added to alternative OS’s before. I think QNX used its networking stack.

                                                                      On the unikernel end, one person brought up NetBSD with several mentioning projects are building on other projects. This isn’t probably technical so much as the usual, social factors. In an interview, Anti just happened to be working on stuff for NetBSD, ran into problems, and built something that helped with those problems. Led to more interesting capabilities. Others built on what he already put out because folks often take the easy or obvious path. Others focused on leveraging a certain language building for theirs with that front and center. And so on.

                                                                      The discussion we’re having is probably a different one than the unikernel authors had when each made their design decisions. ;)

                                                                  3. 6

                                                                    There are a lot of advantages to using a popular technology that have nothing to do with marketing.
                                                                    Here is a partial list:

                                                                    • Books, courses, and training are more readily available.
                                                                    • Support is easier to obtain.
                                                                    • Experts and consultants are easier to hire.
                                                                    • It is less likely to disappear even if another technology replaces it.
                                                                      COBOL, Fortran, IBM mainframes, MUMPS, OpenVMS, and XML are still available even though their popularity has declined.
                                                                    1. -2

                                                                      Die, XML, die! :P

                                                                      1. 1

                                                                        I downvoted myself on that one.

                                                              1. 1

                                                                I can spin up a vm with 416 vCPUs on Google Cloud - https://cloud.google.com/compute/docs/machine-types .

                                                                These are really old arguments that were postulated by various interpreted languages in the late 90s to early 2000s on single core machines when linux threads were not in the shape they are today.

                                                                I find this argument completely out of sync with the capabilities of 2020.

                                                                1. -4

                                                                  Before the reinstallation I did a backup of my entire /home directory. So after I reinstalled all the Flatpak applications I had installed before, I could simply restore almost all files and configurations of the Flatpak apps by copying the .var folder from the backup without having to set up everything from scratch. Flatpak creates a virtual file system in this folder for each application.

                                                                  If I had installed all programs manually via the package management of the Linux distribution, it would not have been enough to simply copy a folder to restore all configurations. I would have had to copy many different folders.

                                                                  How is it possible to be so wrong in every single sentence? :-/

                                                                  1. 10

                                                                    Is it really necessary to phrase your criticism like that? @jlelse is sharing their opinion, and you replying in this way does little more than stop newcomers in the industry from being comfortable to contribute.

                                                                    You can see on their bio that they’re a student, would you say something like this to a junior or grad hire where you work? I really hope not.

                                                                    1. 1

                                                                      Thanks!

                                                                      1. 2

                                                                        No worries! Don’t let negative comments stop you writing and sharing.

                                                                        Nice to see your post about Gitea, I’ll probably be setting something like that up for myself soon!

                                                                        1. 2

                                                                          I’ve been sharing my opinions online for several years now and have received enough criticism (some of it more constructive than this) and it hasn’t stopped me, but rather had the opposite effect. 😅

                                                                          Gitea is really a nice tool, have fun! 😌

                                                                          1. 2

                                                                            some of it more constructive than this

                                                                            lol sounds like you’ve been pretty lucky then 🤣

                                                                            Thanks! It is! I had one before, then replaced it with self hosted GitLab. Kinda want to go back now haha

                                                                    2. 6

                                                                      Could you please explain what’s wrong?

                                                                      1. 4

                                                                        Before the reinstallation I did a backup of my entire /home directory.

                                                                        Needless backup of caches – there is no point in backing up all of $HOME.

                                                                        So after I reinstalled all the Flatpak applications I had installed before, I could simply restore almost all files and configurations of the Flatpak apps by copying the .var folder from the backup without having to set up everything from scratch.

                                                                        So the caches just got copied back again, thanks to Flatpak’s insane violation of the XDG base directory standard.

                                                                        Flatpak creates a virtual file system in this folder for each application.

                                                                        Flatpak needs to stop violating the XDG base directory rules.

                                                                        If I had installed all programs manually via the package management of the Linux distribution, it would not have been enough to simply copy a folder to restore all configurations.

                                                                        There would have been no difference between “normal” applications and Flatpak apps, considering the whole $HOME dir was backed up. Which is needless, as backing up .config (and perhaps .local) should be enough for non-Flatpak applications.

                                                                        I would have had to copy many different folders.

                                                                        You would have to copy .config (and perhaps .local) – which you did anyway by backing up all of $HOME.

                                                                        Even if you did only back up .var, you would still have needlessly copied cache folders, as Flatpak’s make it impractical to avoid copying cache directories, thanks to their “virtual file system”.


                                                                        So it comes to either

                                                                        • backing up .config (and perhaps .local) for “normal applications”, or
                                                                        • backing up .var, then manually excluding cache directories inside it.
                                                                        1. 3

                                                                          What about ~/.mozilla for example?

                                                                          1. 1

                                                                            There is a bug report for that.

                                                                            Generally I try to minimize the amount of applications that ignore the rules of the operating system they run on.

                                                                            I did this by making $HOME read-only and picking alternatives that follow the rules.

                                                                          2. 5

                                                                            Not to mention the bit at the bottom about distros’ package distributions.

                                                                            Sure thing, lets move everything to incredibly slow, memory and cpu hungry containers so that we can rely on people we barely trust not to be malicious whose main priority is having the fanciest new features to keep on top of security vulnerabilities, rather than the people whose sole interest is producing a stable and secure system.

                                                                            This bad-reinvention-of-static-linking fad boggles the mind.

                                                                            1. 1

                                                                              What’s the cpu overhead of a docker container? It’s not literally zero (in most there’s an extra check executed) but that still seems like a bizarrely claim.

                                                                              Memory overhead makes slightly more sense as there’s less scope to reuse dynamic library pages, bit that applies equally to non-container static linking.

                                                                              This leaves me with the impression that you don’t actually understand the things you talk about. Containers have plenty of flaws without inventing problems that don’t exist.

                                                                              1. -1

                                                                                ? I think anyone that has ever touched a ‘container’ find them to be cpu and memory hogs. Not sure how that is even debatable.

                                                                                The comment about them being malicious is imo understated. Most containers in popular registries are filled with vulnerabilities if not outright malware.

                                                                                1. 5

                                                                                  That’s pretty debatable actually, and depends heavily of your host OS. On Linux, a container is nothing more than a process with a bunch of security policies set. Performance wise, they behave exactly like a normal process.

                                                                                  1. 1

                                                                                    not really - anyone using containers in prod will be using an orchestrator such as k8s which adds serious overheard

                                                                                    1. 1

                                                                                      Right, but that’s the orchestration and networking abstractions being heavy. Containers themselves are pretty lightweight. For development, or for running single applications on your machine, they shouldn’t be much worse than running the application directly (other than the non-shared memory, as someone else pointed out).

                                                                                      1. 0

                                                                                        That’s a very reductive argument you are making. What’s the point of saying that a single container on a laptop with no orchestration is light-weight? Most people who have performance concerns are concerned with running in production.

                                                                                        However if you wish to continue down this road let’s talk about the fact that even in a dev environment you are using nat for something like a local webserver.

                                                                                        Finally - I just don’t understand the apologetics in the container world. Why be so defensive?

                                                                                        1. 1

                                                                                          That’s a very reductive argument you are making.

                                                                                          Well, the argument is reductive, because it’s being made in the reduced context of using containerized solutions for software deployment on one’s computer, or a small server. You were the first to mention performance concerns in the thread.

                                                                                          Most people who have performance concerns are concerned with running in production.

                                                                                          I’m sure there’s an argument to be had about how much overhead, but seeing as Google has successfully created a company worth hundreds of billions of dollars using containerized deployments, I’m a bit skeptical about them being business killers. In fact, I’d argue 99.99% of companies out there would be financially wiser by spending another $400/mo server to deal with any performance degradation, than having expensive developers waste time managing dependency hell and convoluted deployment stories.

                                                                                          And I say that as someone running a company with exactly 0 containers in production.

                                                                                          However if you wish to continue down this road let’s talk about the fact that even in a dev environment you are using nat for something like a local webserver.

                                                                                          I’d be incredibly impressed by any development environment pushing so much data between containers that NATing between them becomes a concern. My company deals with video streaming; our development environment streams video between containers; I’ve never noticed a measurable impact on the development experience, other than (maybe?) whatever tiny increase in CPU usage might make my battery life shorter by a few minutes per charge.

                                                                                          Finally - I just don’t understand the apologetics in the container world. Why be so defensive?

                                                                                          Containers are a tool. The right tool, at the right time, might make a huge difference (spoon vs. hammer). I think people aren’t defensive, they are just relaying how containers improved their particular experience.

                                                                                  2. 4

                                                                                    The comment about them being malicious is imo understated. Most containers in popular registries are filled with vulnerabilities if not outright malware.

                                                                                    Not disputing this, but could you give some examples of containers filled with malware? Especially in popular containers?

                                                                                    (Filled with vulnerabilities I can completely believe.)

                                                                                    1. 1

                                                                                      Well a simple google search will show hundreds of examples - here’s on from less than a few weeks ago:

                                                                                      https://blog.aquasec.com/threat-alert-kinsing-malware-container-vulnerability

                                                                                      1. 1

                                                                                        That’s an attack on unprotected docker ports. Not examples of Most containers in popular registries are filled with vulnerabilities if not outright malware. I’d like to see a support for the claim that many containers (I assume you meant images) contain malware.

                                                                                        1. 2

                                                                                          Docker was called out for this behavior a while back ago because they refused to remove the malware after it being reported for over eight months.

                                                                                          https://arstechnica.com/information-technology/2018/06/backdoored-images-downloaded-5-million-times-finally-removed-from-docker-hub/

                                                                                          So people were like ok - we’ll set up our own registry and then you see things like this:

                                                                                          https://threatpost.com/docker-registries-malware-data-theft/152734/

                                                                                          Also - let’s not forget that Docker Hub itself was hacked last year:

                                                                                          https://threatpost.com/docker-hub-hack/144176/

                                                                                    2. 3

                                                                                      Are you sure you aren’t confusing container overhead with the VM overhead in Docker for Mac? I’ve never seen a container imposed significant CPU overhead and I work with them day in and day out.

                                                                                      1. 1

                                                                                        In local dev envs yes you’re correct you get a massive overhead from running them in a vm such as on a mac. In prod envs the overhead comes from the fact that they duplicate existing networking and storage layers (eg: overlay networks).

                                                                                        If it’s just one container on a linux laptop maybe you are right but the people I know don’t run one. If you wish you can qualify my statement with “production use”.

                                                                                        1. 2

                                                                                          Fair enough, but what’s the alternative? Throw out Fargate, K8s, etc and manually operate VMs (bin packing applications, configuring logging/ssh/process-management/monitoring/etc)? This seems penny wise and pound foolish.

                                                                                          1. 1

                                                                                            I’ve never found a need for containers and I know I’m definitely not alone. Also the argument that containers somehow simplify systems administration I find to be completely contrary to my experience of companies that have adopted them.

                                                                                            1. 1

                                                                                              What is your experience? In my experience, it’s much easier to wrap an application in a Docker image and send it off to Fargate than to solve all of those problems yourself with VMs. We were able to transition our ops team into a Cloud Engineering team–rather than operating our individual applications, coordinating quarterly deployments, babysitting VMs, etc, we now focus on showing the dev teams how to use CloudFormation and CircleCI to build out their own pipelines and infrastructure that they in turn operate, monitor, etc. They deploy their own applications as often as 12 times per day. Obviously not all of this was containers–containers aren’t a silver bullet–but there’s no way we could have pushed the overhead of managing VMs onto our dev teams.

                                                                                              1. 1

                                                                                                Quarterly deployments?

                                                                                                Maybe this is just a difference of work environments rather than tooling.

                                                                                                I was a very early adopter of ec2 back in 2006? and have heavily used public cloud vms since in production deployments (I’ve worked at SF Bay companies for the past 10+ years) and was a heavy bare metal user for even longer prior.

                                                                                                In all these years of both bare metal && public cloud vms I’ve always worked at companies that were pushing deploys in excess of ten times/day. I would not work at a company that didn’t. As for the circleci comments - I’d also not work at a company that doesn’t have CI. That’s kind of table stakes.

                                                                                                As for baby-sitting servers - I suppose that depends on the task. It’s my experience that companies in the bay that have embraced containers have much larger (dev)/ops teams than those that don’t.

                                                                          1. 3

                                                                            Overall I do like the idea of using unikernels instead of containers.

                                                                            There are several arguments in this article which don’t make sense for me.

                                                                            Unikernels Avoid Vendor Serverless Lock-In

                                                                            You could replace the word “unikernel” with “container” in this whole section and it would equally apply. I think that the author is specifically talking about poor container usage where people are running many processes instead of just one. Which, to be fair, is all too common, but that’s a different discussion altogether. You could also design bad bloated unikernels.

                                                                            Unikernels Avoid Kubernetes Complexity

                                                                            I agree with much of the points in this section being critical about the complexity of the big container orchestration tools… but again this paragraph sounds like the author has only experienced badly crafted containers:

                                                                            Containers are notorious for eating databases alive. With unikernels, you can pause, stop, restart, and even live migrate with ease with no need to use external software to do any of this because, at the end of the day, they are still just plain old VMs, albeit specially-crafted ones.

                                                                            1. 7

                                                                              unikernels get rid of the notion of the “server” and embrace a serverless mindset. That is not to say there is no server, but it’s not one you can ssh into and install a bunch of crap

                                                                              The “bunch of crap” are crucial tools for monitoring, logging, security (e.g. HIDS), debugging (e.g. strace).

                                                                              Attackers are still equally able to attack a vulnerability in your application and inject an in-memory payload as before.

                                                                              Essentially you gave up detection and forensic capabilities.

                                                                              It’s either working or it’s not.

                                                                              Until you want to investigate an occasional glitch or performance issue that depends on any non-trivial interaction between application, kernel, drivers, firmware, hardware.

                                                                              Unikernels Avoid Kubernetes Complexity

                                                                              Also traditional UNIX processes and OS packages. Nowadays they provide very good sandboxing without the additional layers of filesystems and virtual networking.

                                                                              1. 0

                                                                                The logging/monitoring/debugging points are just flat out wrong. This gets brought up way too often and it’s just not correct. I’d appreciate it if people would stop repeating things that simply aren’t true.

                                                                                Logging works out of the box. Ship it to papertrail/elastic/splunk/whatever. Monitoring works out of the box. Newrelic, prometheus, lightstep, etc. works out of the box.

                                                                                HIDS? For what? You can’t login to it - there aren’t the notion of users.

                                                                                As for in-memory payloads go for it - it becomes much much harder cause you can’t spawn new processes and at least Nanos employs ASLR and common page protections found in Linux so imo that becomes way harder to attack.

                                                                                As for ‘debugging’ firmware/kernel/hardware - a) none of that would be debugged in a unikernel cause it’s all virtual and b) why would you do that on a prod system? That’s the sort of work you do in a development environment - not a live system.

                                                                                1. 2

                                                                                  That’s the sort of work you do in a development environment - not a live system.

                                                                                  Until your repro is in a live system and the company is losing money so you need to debug it now.

                                                                                  1. 1

                                                                                    yeh, I’d completely disagree here - most ops people don’t code at all and frankly if you are going to “debug” “kernels, drivers, firmware, hardware” on a live prod system - that’s a firing offense in my most humble opinion

                                                                                    1. 2

                                                                                      Usually, the issue is in userspace. But when downtime is costing thousands of dollars a minute, and your repro slipped past all of your testing, QA, staged rollouts, and SREs, you go into the vault, get the SSH keys, and debug wherever you can.

                                                                                      1. 1

                                                                                        I’d agree with the sentiment that it is usually in userspace 100%.

                                                                                        Thousands/min? Was this measured over the course of a week or over the course of a few hours? If over the week why did it suddenly become a problem? If over the course of a hour was it because of a bad deploy that could simply be rolled back?

                                                                                        Also, why would a single instance cause thousands/min damage and not more than a few servers? If it was a single instance perhaps can just kill that one. If it’s infectious than that points something non-instance specific.

                                                                                        As someone that has had to personally deal with shitty code blowing up servers at odd hours of the night repeatedly for years I get the feeling - I get it :) . I just don’t think the practice of sshin’g into random servers is a sustainable process anymore. Definitely not when it’s just splashing water on forest fires vs fixing the root problem.

                                                                                        1. 1

                                                                                          So, in the instance I am thinking of, someone else’s rollout changed the query being sent to our service, which broke accounting for a significant portion of the ads served by YouTube.

                                                                                  2. 1

                                                                                    You cannot possibly instrument/log every parameter to every function in both user and kernel space for every call. It is way, way too expensive. The same thing goes for per-processor utilisation with subsecond resolution. It has to be done in a targeted way, when it is required.

                                                                                    I feel like people who object against live diagnostics have never heard of perf and BPF, or at least not used them to solve hard problems.

                                                                                    Centralised collection of metrics is great for discovering trends and forecasting. It is not of sufficient resolution to diagnose issues in production.

                                                                                    1. 1

                                                                                      We’ve heard. :)

                                                                                      Nanos even has support for both ftrace ( https://github.com/nanovms/nanos/tree/master/tools/trace-utilities ) like functionality and strace like functionality.

                                                                                      I still hold to the case that if you are resorting to running something like perf in prod you’ve failed to instrument or profile your application beforehand. The author explicitly warns in the docs that you might want to even test before using production as it has had bugs resulting in kernel panics in the past. Many of the problems that people “can’t replicate” in a dev/test environment actually can be and should be.

                                                                                  3. 0

                                                                                    The “bunch of crap” are crucial tools for monitoring, logging, security (e.g. HIDS), debugging (e.g. strace).

                                                                                    Even though I agree this is typically the case, this is not the way “DevOps” is supposed to work. Containers should not have all this stuff in them either. Containers should run 1 process. They can’t always work this way, without rewriting a program, but we should strive for that.

                                                                                    Attackers are still equally able to attack a vulnerability in your application and inject an in-memory payload as before.

                                                                                    Yes, but the attack surface is greatly reduced.

                                                                                    Also traditional UNIX processes and OS packages. Nowadays they provide very good sandboxing without the additional layers of filesystems and virtual networking.

                                                                                    I can’t help but largely agree with this. However, there are many benefits to the application isolation from VMs, unikernels, and containers, which allow for sysadmin models that are not possible with just processes and packages on a single host.

                                                                                    1. 4

                                                                                      So if a container should only run 1 process, what’s the difference between a container and a statically linked program running in a jail?

                                                                                      1. 1

                                                                                        Containers should not have all this stuff in them either. Containers should run 1 process.

                                                                                        Indeed. All the tooling is on the base OS and often works in roughly the same way against normal unix processes and containers.

                                                                                        Attackers are still equally able to attack a vulnerability in your application and inject an in-memory payload as before.

                                                                                        Yes, but the attack surface is greatly reduced.

                                                                                        Not if you consider the point above. The application is still the same codebase (regardless if it’s running as a sandboxed process, a container and so on). You still have to either deploy and run all the ancillary tooling at some point in the stack or have unmanaged black boxes around.

                                                                                        1. 0

                                                                                          The reduction in attack surface, while true, is not the main security selling point imo. It’s the fact that most common exploits wish to execute other programs, usually many others. If I want to take advantage of your weak wordpress setup and install a cryptominer not only is it a different program - it’s probably not written in php and thus I won’t be able to install it in a unikernel environment and run it. I’m forced to inject my payload in-memory as you state and that makes things rather hard very very fast. Most payloads simply try to exec /bin/sh as fast as possible. Can you code a cryptominer with ROP alone? Can you code a mysqlclient with ROP alone?

                                                                                          I’d love to see this.

                                                                                    2. 0
                                                                                      • Your argument here doesn’t match the header you pasted (serverless) - so not sure what you are trying to point out here - the security side of things or serverless lock-in? If you can elaborate I can help provide pointers. If it’s the security side of things - the fact that it is single process by architectural design deals a pretty hefty blow to all the malware found in containers today like the stuff mentioned in https://unit42.paloaltonetworks.com/graboid-first-ever-cryptojacking-worm-found-in-images-on-docker-hub/ . Otherwise, that header is about serverless offerings like lambda or google cloud run.

                                                                                      • There’s a quite a lot of “orchestration” tooling that is arguably necessary in the container ecosystem precisely because they insist on duplicating networking/disk on top of existing virtualization. The point I was trying to make here was that since at the end of the day these are virtual machines - you get all that “orchestration” for free. Make sense?

                                                                                      1. 1

                                                                                        From Unikernels Avoid Vendor Serverless Lock-In

                                                                                        However, unlike most serverless offerings that are entirely proprietary to each individual public cloud provider, unikernels stand apart because instead of some half-baked CGI process that is different from provider to provider, they assume the basic unit of compute is a VM that can be shuttled from one cloud to the next and the cloud provider is only responsible from the virtual machine down.

                                                                                        I don’t understand how this is unique to unikernels. You can shuttle containers from one server (or cloud provider) to the next.

                                                                                        Security- Yes, containers and their tooling are terrible here. Unikernels, especially well designed unikernels really shine in this regard.

                                                                                        There’s a quite a lot of “orchestration” tooling that is arguably necessary in the container ecosystem precisely because they insist on duplicating networking/disk on top of existing virtualization. The point I was trying to make here was that since at the end of the day these are virtual machines - you get all that “orchestration” for free. Make sense?

                                                                                        Ok I think I’m getting your perspective now. You’re assuming the container orchestration software is running inside of VMs managed in-turn by VM orchestration software. More abstraction, more code, more attack surface, more problems. Totally agree here, if I’ve got it right now. Maybe you could spell it out a bit more at the start of the article, even though it’s quite clear now on re-read once I’ve already grabbed onto the perspective.

                                                                                        As an aside, container orchestration software could run directly on a hypervisor, and that’s where things seem to be moving to in the cloud-native industry. Which is going to be… exciting, from a security perspective.

                                                                                        1. 2
                                                                                          • afaik live migration is still not a production quality feature for containers although it has existed in the vm world for over a decade now? and the container ecosystem has invented an entire language to describe the whole concept of persistence which is trivial in vms and non-trivial in containers, but this section wasn’t really taking aim at containers

                                                                                          • yeh - the security argument wasn’t really for that - it’s the fact that if you are spinning up a k8s stack on gcloud or aws you are inherently already on top of an existing vm stack and there’s a ton of duplication going on here which makes no sense from a performance or complexity standpoint - the security arguments are a bit different, as far as the containers on hypervisor - I know kube-virt is there and I see this concept talked about a lot on twitter but I don’t see much movement there, regardless - that’s essentially just stuffing a linux inside of a vm and using the container mechanisms for orchestration - part of the security story here is not just ‘container suck at security’ - it’s deeper than that - it’s the fact that linux is ~30 years old and was built in a time before heavy commercialized virtualization (ala vmware) and before the “cloud” - these two abstractions give us a chance to deal with long-standing architectural issues of the general purpose operating system

                                                                                          1. 1

                                                                                            Regarding live migration, isn’t it rare for public cloud providers to actually support this?

                                                                                            1. 1

                                                                                              That’s not a primitive that they traditionally expose as-is you are correct. To do so in a free-for-all environment would probably have some serious security/scheduling ramifications, however, you see it pop up in quite a few places regardless. For instance if you go boot a database on google cloud right now and give it say 8 gig, insert a few hundred thousand rows, you can instantly upgrade that database to 16 gig or 32 gig ram without it actually going down. Behind the scenes they are spinning up a new vm, transparently migrating page by page to the new vm without destroying the live database and then shutting down the old vm. Also, AWS uses it to migrate vms from faulty hardware as well.

                                                                                              Of course in private cloud situations this is routinely used for backup/DR.

                                                                                              This is all to say that there are many many features that are simply not possible under a container paradigm.

                                                                                            2. 1

                                                                                              it’s the fact that linux is ~30 years old and was built in a time before heavy commercialized virtualization (ala vmware) and before the “cloud”

                                                                                              What’s the connection? UNIX is older, virtualization exists since 1960ies and hardware virtualization since 1970s.

                                                                                              1. 2

                                                                                                The keyword here being “commercialized virtualization”. We are talking about ESX from VMWare here. Anyone using large sets of servers in north america and not using AWS (api on kvm/xen) or Google Cloud (api on kvm) is using xen from Citrix or esx from VMWare.

                                                                                                We didn’t have style of offering in the 90s. Everyone had to use real servers and actually use the unixisms of multiple processes with multiple users on the same box. You’d pop into a sun box and use commands like who and see a half dozen users. You could be the BOFH and wall them a message stating you were kicking them off.

                                                                                                Times have changed. Because we not only have access to virtualization in private datacenters but entire APIs against the ‘cloud’ we can finally drop the chains that have existed in unix land for so long and solve a lot of the problems like performance and security that exist.

                                                                                                1. 1

                                                                                                  Isn’t making Linux more fitted for virtualization one of the motivations for systemd?

                                                                                                  1. 2

                                                                                                    When I say virtualization I’m speaking about classical machine level virtualization not carving up a linux system with namespaces. The OSTEP book, while one of the more accessible books on operating systems, uses virtualization in a very liberal sense. Ever since then we’ve seen the containerati abuse the term as well and this is where the confusion sets in.

                                                                                                    Machine level virtualization has actual hardware capabilities to carve up the iron underneath. Machine level virtualization doesn’t care what the operating system is (linux, macos, windows, etc.), systemd otoh is clearly linux only (and a very specific set of linux at that). In fact today’s virtualization (vs that of 2001) is so sophisticated from things like passthrough and nic multiplexing it’s possible to run vms faster than an average linux on the hardware itself - that’s how good it is today.

                                                                                                    That is why I’m very hesitant to label namespaces and their friends ‘virtualization’. To me that’s a very different thing.

                                                                                                    1. 1

                                                                                                      Thanks for taking the time to clarify!

                                                                                                  2. 1

                                                                                                    Is there also an implication with “ Linux is 30 years old” that Linux has not been developed since its inception? That something is old is not an automatic disqualification if it has active development.

                                                                                                    1. 3

                                                                                                      It’s not about the age. It’s the fact that the environment has changed and core concepts are not appropriate anymore. We had to use operating systems that supported multiple processes by multiple users in the 90s. Linus’ computer was a 286/386 in 93. We didn’t have commercialized virtualization or the cloud. Back in the 70s when the concept was delivered originally in Unix they were using computers like the pdp-7 and pdp-11 that took up entire walls and cost half a million dollars. Clearly back then that architecture had to be that way.

                                                                                                      Contrast/compare today when even small 20 person companies can be using tens or even hundreds of vms because they have so much software. We need not mention the big players that pay AWS hundreds of millions of dollars/year or banks that wholly own entire sets of datacenters completely virtualized.

                                                                                                      So it’s not the fact that linux is 30 years old or unix is 50 years old - it’s the concepts of being able to say run a database and a webserver on the same machine or the fact that you even want the concept of a interactive ‘user’ when you have a fleet of thousands of vms. Most people’s databases don’t even fit on a single machine anymore, a lot of their web services don’t either - anyone with a load balancer can show you that. We’ve consumed so much software that the operating system has largely inversed itself yet we are still using the same architecture that was designed for a completely different time period and it’s not something you can just yank out or seccomp away.

                                                                                                      1. 2

                                                                                                        That’s a very interesting perspective. Thanks for explaining in greater detail!

                                                                                        1. 1

                                                                                          are rumpkernels dead? at least the website isn’t resolving anymore.

                                                                                          1. 1

                                                                                            Last word is that Antti took up brewing beer.

                                                                                            1. 1

                                                                                              good for him i guess :)

                                                                                              1. 1

                                                                                                That reminds me, I’m curious about why you chose to write your own unikernel rather than continuing his work. With the latter approach, you’d have the solid foundation of NetBSD’s drivers and TCP/IP stack, while keeping the advantages of unikernels.

                                                                                                1. 1

                                                                                                  Half of the tcp/ip stack has been temporarily borrowed from LWIP (although there’s a very high chance it’s replaced in the next year depending on time/resources).

                                                                                                  Drivers are actually a non-issue if you presume that things will only be deployed in a vm (we have 0% intention to run on real hardware - it’s a part of the architectural model). Then you just need to support the disk, network, clock, etc. and those we obviously re-use for whatever the hypervisor is.

                                                                                                  It also might help that Antti had no intention of making a ‘unikernel’. His thesis revolved around how one could debug/switch out/fix drivers in a live system.

                                                                                                  Lastly, we saw early on that forcing people to re-compile software they didn’t write was a non-starter.

                                                                                                  1. 1

                                                                                                    I agree with you that being able to use existing binaries compiled for Linux is a very useful feature. So rumprun certainly isn’t the full solution. I also understand that you don’t need a large selection of drivers. But having NetBSD’s TCP/IP stack and filesystem, if nothing else, would have been useful, right? You could then have implemented Linux emulation on top of that. So I wonder if you considered that approach, and if so, why you rejected it. Alternatively, did you consider building on OSv? I’m sure that what you want to sell isn’t the kernel per se, but higher-level tools on top of it. So why not build on existing work?

                                                                                                    1. 1

                                                                                                      Not as useful as you might think. Whatever you port you have to support.

                                                                                                      Emulating linux from bsd would be an entirely different ballgame that we don’t wish to expend resources on. Building a systems company is a bit different from building a SaaS company. First off, it’s not ‘super hot’ and second it takes a long time (engineering years) to get even the smallest stuff going. There’s this idea of innovation tokens and you can only spend so many at a time.

                                                                                                      Also, why would we build on dead (or not financially enabled) projects that we don’t control?

                                                                                                    2. 1

                                                                                                      Drivers are actually a non-issue if you presume that things will only be deployed in a vm (we have 0% intention to run on real hardware - it’s a part of the architectural model). Then you just need to support the disk, network, clock, etc. and those we obviously re-use for whatever the hypervisor is.

                                                                                                      i think a really interesting use case for unikernels are cheap arm boards though. if you have the whole system for yourself, a 15$ board might go a long way.

                                                                                                      1. 1

                                                                                                        agreed, espc w/the confluence of 5g/ML/edge - nanos doesn’t do ARM today but it might in the future, wink wink nudge nudge

                                                                                                        edit: i might’ve responded too fast - even if we support ARM in the future it’d still be only as a vm - not flashing the device (we’re def. not the only ones that think this way)

                                                                                              1. 6

                                                                                                I think unikernels are a really interesting and great idea; I love their simplicity over complexity approach to problem solving.

                                                                                                But I’d love to have a more ingenuous discussion about them, e.g.

                                                                                                This is why we keep on seeing articles like the one from AirBnb complaining that kubernetes has serious performance problems.

                                                                                                This quote links to an article that, more or less, discusses how Airbnb approached the common problems of multi-tenancy the Kubernetes way. Unikernels, I assure you, have the same, similar, or even new multi-tenancy problems. And they have their own unikernel flavor of solutions.

                                                                                                The takeway from that article should not be that “Kubernetes has serious performance problems” so much as “multi-tenancy is hard and each platform has its own solutions”. It doesn’t feel like an honest takeaway and makes the rest of the writeup feel disingenuous, i.e. it lost my trust in the author’s objectivity.

                                                                                                1. 1

                                                                                                  The whole point here is that multi-tenancy is much easier to deal with considering they are deployed as vms so they don’t have to reinvent the wheel and the scheduling and the isolation problems are resolved by the cloud providers (ergo no k8s necessary).

                                                                                                  If we are talking about having an ingenuous discussion can you explain why you think it would be harder?

                                                                                                1. 3

                                                                                                  The article should probably disclose that the author is the CEO of NanoVMs, which develops the Nanos unikernel and Ops deployment tool.

                                                                                                  Having said that, I agree about the complexity of Kubernetes and related things. But rather than completely dump Linux inside the VM, I think I’d go with something like LinuxKit, which can build minimal, immutable Linux-based VM images, using Docker (or is it OCI) images. IMO, this achieves many of the benefits of unikernels, without throwing away the proven, mature foundation of Linux.

                                                                                                  1. 1

                                                                                                    The author profile discloses it but yes I’m at NanoVMs.

                                                                                                    As for the other - you are still stuck with linux then right? Why not just use a linux vm then? Cause you have all the same problems that linux has. We’re not just trying to fix container issues here. We’re trying to fix longstanding linux issues. Again, Linux is 30 years old and is largely the same as Unix. I’ve seen a lot of ‘windows 7 EOL’ lately but guess what? Windows 95 was released after Linux was. There’s an awful lot of really bad software in the FOSS world and unikernels offer the chance to start fresh and really wipe the slate clean.

                                                                                                    1. 7

                                                                                                      There’s an awful lot of really bad software in the FOSS world and unikernels offer the chance to start fresh and really wipe the slate clean.

                                                                                                      There’s also a lot of really good, featureful, well-debugged software in the FOSS world, including the Linux kernel. You mention the age of the kernel as if it’s a bad thing, but building upon mature technology, rather than starting over just for the sake of wiping the slate clean, is how we make progress. To put it bluntly, I have a lot more confidence in the reliability and performance of the Linux kernel than I have in your two-year-old unikernel. At the same time, using something like LinuxKit, we can drop the user-space components that really are redundant while keeping the good parts.

                                                                                                      1. 1

                                                                                                        To be clear - I’m not denying your right of faith. :)

                                                                                                        For this argument it’s not really about the kernel per-se - it’s the constructs it has that allows software to run rampant and ripping that out of linux is not as easy as applying an overly restrictive seccomp to it. Multiple processes and all the interesting SYSV IPC stuff you can do with them is a very large chunk of code. Think about the implications that has to say the scheduler. Or think how this applies to security.

                                                                                                        Likewise, half of linux is merely device drivers. Just last fall there was a discussion on the LKML to finally remove floppy disk support and yet someone wanted to keep supporting it for ‘posterity’. Ripping some of this out is more political than anything else. However, some very famous and loved software projects such as postgresql (which would never fit in an unikernel) are inherently built around some of these primitives. That’s why you just can’t jettison the bathwater. At some point you have to state what you are going to keep and what you are going to throw.

                                                                                                        As for userspace - that’s an entirely different mess. Case in point - try and install an interpreted language such as python or ruby or something of that nature. You’ll end up installing a half-dozen other compilers each depending on a whole legion of libraries that have been around for a few dozen years. Some like libxml (compiled into every single interpreter I know of) have a fully functioning ftp server in them - why?

                                                                                                  1. 9

                                                                                                    This article is a bit of a rambling rant about threads, setting up a bunch of dubious strawmen, and doesn’t say:

                                                                                                    1. Anything about what KeyDB is or why it has a better architecture than Redis, unless you accept the implied assertion that multithreaded == better architecture.
                                                                                                    2. Anything about unikernels (the word is not even in the article)

                                                                                                    Honestly the title feels like clickbait given how low little it’s related to the article text.

                                                                                                    1. 3

                                                                                                      Thanks for the feedback.

                                                                                                      1. It wasn’t really supposed to be about keydb - that was just used as an example of what would work better (versus redis) - the links and such are edits dzone did. KeyDB was used as an example of showing what can/should be done for future facing software.

                                                                                                      2. Interesting! I wrote an entire article without saying that word - I didn’t plan that - the premise here is that a lot of older nix software (eg: let’s pick on postgresql) is old and not written to excel in a single process (but many threaded) system. I could’ve easily picked apart postgres instead but the keydb/redis was much easier to illustrate the point. Not only that, but developers have been pushed not to advance this style of architecture. My argument here is that if people would adopt this over something that embraces sys-v ipc style we could open up a much better ecosystem.

                                                                                                      1. 1

                                                                                                        There was work even way back when on making concurrency and/or parallelism easier to get right. Here’s some for you to check out in case you find them interesting: Concurrent Pascal (among others), Eiffel’s SCOOP, Clay’s type-safe locks (pdf), ParaSail, and DTHREADS (pdf).

                                                                                                        Since you mentioned Python, I’ll throw in that a few people told me they just used GNU Parallel with their scripts. I’m not sure how they do the integration step if there is one. I imaging pulling stuff out of text files into collections is common for them. GNU Parallel solves the harder problem with a one-liner in the terminal.

                                                                                                        1. 2

                                                                                                          In an unikernel context Parallel wouldn’t be an option. It solves the main job of executing N things yet still in the old create a new process way - that’s what I’m arguing to get rid of because of it’s numerous security and performance issues. Python has many non-trivial limitations (beyond the GIL) that would need to be addressed to have true multithreading. Jython kinda meets in the middle but introduces other issues. I’m not saying any of this is easy - just something to aspire to.

                                                                                                          Going back to how all this relates to unikernels - many unikernels have support for multiple threads but not multiple processes by design. One of the things that needs to happen to make the ecosystem thrive is more keydbs and less redis’.

                                                                                                          1. 2

                                                                                                            Honestly, I didn’t get to the unikernel claims since the article was pretty ranty on the anti-Python, anti-Guido stuff. I’d have made it if you just did a technical breakdown of what older solutions did, what modern hardware does, how it doesn’t fit, and what you were promoting. Those kinds of articles also get more support in general on Lobsters. More potential uptake, too. I agree Parallel itself wouldn’t fit a unikernel but a Parallel syscall might.

                                                                                                            1. 2

                                                                                                              It didn’t really come off as anti-python and/or Guido to me, but that might just be the last year or two giving me a higher bar for “anti Guido” :(

                                                                                                              I don’t care for Python, but Guido always struck me as a really thoughtful, nice guy; those nerds who drove him out of Python should be tarred and feathered.

                                                                                                              1. 1

                                                                                                                for the record I’m neither anti-Guido nor anti-python - that section was really intended to highlight:

                                                                                                                a) the inherent struggle to take advantage of native threads in interpreted languages such as python (it affects most, all? of them)

                                                                                                                b) the time period that certain views get espoused (1998, 2003, even up to 2008) these arguments had technical merit but the advent of real good threads in linux in ’03 and the drive towards many cores/many socket machines during those years really removed those technical arguments imo

                                                                                                                as evidenced in the prior section Go did not suffer this same fate even though one or more of it’s creators had similar arguments for the ability of the average developer to take advantage of threads to the extent that they designed one of it’s core features goroutines as an abstraction around them

                                                                                                            2. 1

                                                                                                              Performance of fork() I can buy, but insecurity? That I do not. In a multithreaded program, one bad reference and the poof! There goes the entire program, but in a multiprocess program, only one process goes poof on a bad reference (remember my days of programming on the Amiga sans MMU). Also, there’s been studies that have down fine grained locking degrades performance just as much as a single global lock (perhaps even more). Also, multithreaded programming is hard to reason about, which is why I personally prefer the actor model (share nothing message passing) over accessing shared mutating state.

                                                                                                              1. 1

                                                                                                                insecurity for this definition means ‘thwarting bad actors’ not making your program crash from a bad memory reference due to programming error - i’d put that in the memory safety camp; also this is in the unikernel (single process) context not necessarily a linux system

                                                                                                                being constrained to whatever you can do in the memory layout of a single executable is quite different than being able to execute random other arbitrary programs because your system is inherently built for that (eg: your average monero cryptominer bot dropper is most assuredly going to come in binary format and prob. written in a diff. language); your average shellcode does just that - pops a shell; for example i’ve yet to see an entire mysql client written in pure shellcode w/no calls to fork/exec*

                                                                                                      1. 11

                                                                                                        because I can compile binaries statically

                                                                                                        1. 6

                                                                                                          This is repeated a lot but almost every time I hear this the people stating this mean something completely different than ‘static linking’ which is what the statement implies (‘compile’, ‘statically’). For instance I see quite a lot of gophers state this as well yet when you look at the binaries libc and a half dozen others are linked in. I’m not stating this is good/bad - that’s a different argument. I just don’t find this to be a factually correct statement and when repeated a lot it is very misleading. If you look at most of the php or ruby interpreters in use you’ll find the situation is even dramatically different since it’s written in c, older, and has quite a few libraries that are written in c and linked in (rmagick, libxml, etc.) Furthermore most of the container shops I’ve interacted with don’t compile from source to begin with. It’s ostensibly one of the main reasons they are using containers to begin with because they seem to find it hard to compile/install the specific versions/libraries.

                                                                                                          To eliminate confusion those that repeat this should probably clarify that they are creating filesystems with known fixed versions of libraries that are in most likelihood 99% dynamically linked (the vast majority of linux binaries are) - not statically linking them.

                                                                                                          1. 2

                                                                                                            So you don’t think people who espouse “static compilation” actually use static linking?

                                                                                                            1. 1

                                                                                                              no I don’t - not in the “single binary completely statically linked” sense - especially with the interpreted languages as I mentioned as they aren’t sitting there and compiling from source and statically linking in all the different libraries to the language interpreter itself - in fact many of the interpreted languages apis that use dynamic libraries aren’t even built that way - many of them are built explicitly to dynamically load - this is easily proven by code - it’s not even really debatable in cases like this - again - this isn’t a right/wrong conversation here - it’s more of a confusion that’s be promulgated and I don’t think OP or any of the upvotes were intentionally trying to deceive - I think it’s one of the many things that dev evangelists from camp container have misspoken quite too often and then posts with titles of “why are you not using containers” appear <– this is a huge reason why

                                                                                                              1. 2

                                                                                                                I certainly wouldn’t want people to use the phrase “static compilation” or “static linking” to refer to an interpreted program… In this case though it seems quite possible that they were referring to compiled programs, statically linked.

                                                                                                                1. 1

                                                                                                                  so the forest for the trees here is that if you are indeed statically linking via say go why would you need docker to facilitate that process? I think the go switches are something like “go build -tags netgo -ldflags ‘-extldflags “static”’” - as I said before, ostensibly the reason why people were using docker in the first place is “it works on my machine” and they weren’t having to guess what version of what libraries were being used not that it’s really common to link in random shared libs to go anyways - that’s not a go idiom

                                                                                                                  1. 6

                                                                                                                    so the forest for the trees here is that if you are indeed statically linking via say go why would you need docker to facilitate that process?

                                                                                                                    i think that’s precisely the point /u/coco was making

                                                                                                                2. 1

                                                                                                                  I do not really understand what you say… for me “static linking” and “static compilation” is the same, but I’m just a C programmer, and distributing standalone binaries that work everywhere usually amounts to adding “-static” to your makefile

                                                                                                                  1. 4

                                                                                                                    nah I was confused - caleb was kind enough to help sort things out; i hear a lot of container folks state this as a reason for using containers to begin with whereas you were stating that it’s a reason why you don’t need containers and that’s completely valid

                                                                                                                    as for this direct comment - generally speaking there is a compile and a linking stage but you’re right if it’s a simple program that can accomplish both at the same time

                                                                                                                    sorry for the confusion

                                                                                                              2. 1

                                                                                                                you look at the binaries libc and a half dozen others are linked in.

                                                                                                                though those libs tend to be fairly reliable and pre-installed on virtually any system, whereas other libraries that might require a separate installation step can be linked into the binary (or perhaps distributed with the executable zip, Windows style)

                                                                                                            1. 4

                                                                                                              Even against the hype machine that is docker/k8s I’d argue there is way more content on doing things “the boring way” - you can go a very long ways with a little bit of apt-get.

                                                                                                              In the past I have used puppet/chef/terraform/shell scripts/etc. to build aws images and just launched manually whatever is needed with plain old ci (jenkins) systems doing actual deploys/rollbacks. I think the concerns about native library compatibility are typically overblown. Yes you have to change base images every now and then but you’d have to do the same in containerland. I’ve never liked the configuration management software out there yet I can see value on larger teams. I think the ci space is pretty straight-forward yet think there is plenty of room for improvement.

                                                                                                              True immutability is rather hard to do unless you severely restrict ssh access. I find people (including myself) will install all sorts of crap if given time and a problem. This can go south when certain tools and I’ll pick on sysdig come with kernel modules or someone’s favorite editor decides to upgrade tls.

                                                                                                              However I truly have not had the horrors that I see with people when they choose containers/k8s. My personal opinion on containers is that if people think installing a given service is such a pain then perhaps they shouldn’t be administering that service (with anything).

                                                                                                              Also I’m heavily involved in a company that is working on unikernel tooling so there’s that. :)

                                                                                                              1. 1

                                                                                                                Well, that’s debatable. I’ve seen so many articles that discuss about the deployment process of a jar file and write in the middle of the the blog post “deploy jar” without any explanation on how they deploy it, where do they put it, how they allow rollback, etc… There’s probably no magic, but I feel that it’s the jungle and everybody has different conventions.

                                                                                                              1. 3

                                                                                                                The link in the description on GitHub: https://nanos.org/ seems to be dead right now.

                                                                                                                It would be nice to have some comparison to existing similar solutions like:

                                                                                                                BTW There is a nice list of unikernel projects:

                                                                                                                1. 2

                                                                                                                  Yeh sorry we’ve reserved the domain name but we don’t have a site for it yet - kinda too busy with other things. I’d imagine that site will be up within a month or so though.

                                                                                                                1. 1

                                                                                                                  It gets me to wonder if it would be possible to use the kernel text console output from the hypervisor for the purpose of logging.

                                                                                                                  1. 2

                                                                                                                    by default that’s exactly what we do currently although we highly recommend for prod services to ship it to whatever log provider is being used (papertrail, elasticsearch, splunk, syslog, etc.)

                                                                                                                    1. 1

                                                                                                                      Thanks!

                                                                                                                      I spawn daemon | s6-log, with the logger that does:

                                                                                                                      • writing to a file with timestamp for each line,
                                                                                                                      • pattern matching filter useless logs away, or post some lines for further processing (send alerts…),
                                                                                                                      • rotate / compress logs as soon as the written size exceed the wanted size and/or date,

                                                                                                                      This makes logging extremely simple for everyone, flexible, trifial, lightweight, easy to implement, to configure, uniform across all daemons (or qemu vms), ubiquitous (everything have a mode to log to stderr)…

                                                                                                                  1. 1

                                                                                                                    Uh, curl | sh

                                                                                                                    1. 2

                                                                                                                      Well you can always build from github if you’re concerned but if you are downloading programs from a website and you don’t trust the website don’t run the program. We do have open issues for building debs/rpms and such but it’s one of those things that happens when enough people care.

                                                                                                                    1. 3

                                                                                                                      A unikernel is simply an application that has been boiled down to a small, secure, light-weight virtual machine. The resulting virtual machine image does not contain an operating system like Linux or Windows.

                                                                                                                      A unikernel still has a kernel to my understanding… so what kernel is this running?

                                                                                                                      1. 4

                                                                                                                        According to https://github.com/nanovms/ops/blob/master/CHARTER.md it’s the NANOS unikernel, which AFAICT is not open source.

                                                                                                                        Appears to be straight-up spam; this is the only thing @eyberg has posted here and it’s promoting their commercial service without further comment.

                                                                                                                        1. 3

                                                                                                                          https://github.com/nanovms/ops/blob/master/lepton/image.go#L369

                                                                                                                          How do we know it is secure if we do not see the code?

                                                                                                                          Come on! Show us the code, we want to have fun too! :)

                                                                                                                          1. 1

                                                                                                                            I wouldn’t call that spam cause you can use it for free via OPS. Also, I asked about this on IRC today. What would the difference between posting something about cockroachdb or elasticsearch or redis or any other ‘open source’ tools out there?

                                                                                                                            1. 3

                                                                                                                              Where’s the source for NANOS? I couldn’t find it, which lead me to assume this is not actually an open source tool. If it’s open source and OPS happens to offer a hosting service for it, that’s fine by me.

                                                                                                                              If your sole interaction with the community is to drop a link to your employers stuff, that’s fairly spammy. When a newcomer does that, it feels like they’ve joined for the sole purpose of promoting their company (rather than for the purpose of being involved in the community).

                                                                                                                              When people who are active participants post links to their work, it feels more like a friend saying “I did this cool thing at work, check it out”.

                                                                                                                              Does that distinction make sense?

                                                                                                                              1. 1

                                                                                                                                Nanos hasn’t been officially released yet (look at the version - 0.1.x), no and I don’t know what form (read: license) it will take. However it is free as in ‘free beer’ today and will continue to be so.

                                                                                                                                I didn’t see anything about the rules on the site that mentioned software needs to pertain to a given license. I see a lot of articles that have nothing to do with open source here. I popped on here cause it was mentioned in an article the other day here on /t/osdev -> https://dev.to/yonkeltron/the-coming-era-of-os-kernel-modularity-5emc and I noticed there was no link to the source code of the tool that was mentioned. I’m not an active member of this community as I frankly don’t have the time to be but the tool is open and you can download it for free. Are all parts open? No and neither are most ‘open source’ tools like the ones I mentioned - definitely not according to OSI.

                                                                                                                                If a tool needs to be GPL only on this site please let me know if not I’m sorry you are offended and I apologize for breaking any rules on the site if that’s the case.

                                                                                                                                I’m not trying to fight anyone.

                                                                                                                                1. 5

                                                                                                                                  I don’t care about the licensing. Is the source available? Where?

                                                                                                                                  I read through the repo that was linked, and didn’t find anything that looked like it implements a kernel.

                                                                                                                          2. 2

                                                                                                                            Yes all ‘unikernels’ do have the notion of a kernel even though they might not call it such. We’ve developed a new one that is built with unikernel concepts in mind (eg: no users/permissions/etc., single process, etc.) It clearly has a lot of work to go but it runs quite a lot today in a much safer environment than stock linux.

                                                                                                                            1. 1

                                                                                                                              We’ve developed a new one that is built with unikernel concepts in mind

                                                                                                                              From scratch?

                                                                                                                              1. 4

                                                                                                                                From the bootloader on up yes. It’s quite a lot of work but we felt it was necessary. We used to use Rumprun and OSv quite a bit but kept running into various limitations/issues and decided it wasn’t worth the effort to maintain forks since neither of those are super active (although Waldek is doing a great job w/OSv lately).

                                                                                                                                We’re asked a lot why we didn’t just patch/fork linux and the short of it is because of the same reasons. Linux is measured in the >10M LOCs and it just has way too much stuff in it that is antithetical to the unikernel philosophy. However it’s important to realize that we’re not trying to replace Linux either - for instance we never plan on not running as a vm so half of what linux is (drivers) we don’t need to code/support allowing us to focus in on what really matters.

                                                                                                                                1. 5

                                                                                                                                  Great job!

                                                                                                                                  This response should be the first thing I see when I look at the readme / and/or your business website. I spent ~10 minutes trying to find out what the hell your service does and why I would want to use it - this is the answer I was looking for.

                                                                                                                                  When you open source the kernel, please do post it here also, as it’d be of tremendous interest to the community.

                                                                                                                                  1. 2

                                                                                                                                    “but kept running into various limitations/issues and decided it wasn’t worth the effort to maintain forks”

                                                                                                                                    What issues and limitations were you running into?

                                                                                                                                    1. 3

                                                                                                                                      Well the big one with rump was 32bit only which is a non-starter for most java apps and databases and that opens up a large swath of technical decisions that have to be made (like virtual memory and paging). Also rump is composed with netbsd and so carries that load.

                                                                                                                                      The big one with OSv was PIE only (which is not quite the case anymore but in our exp still has quite a few gotchas).

                                                                                                                                      There are quite a handful of other technical issues but at the end of the day building our own allowed us to do whatever we wanted to.

                                                                                                                                      1. 2

                                                                                                                                        I could see you wanting to slim it down or use 64-bit. I assumed rump was 64-bit given the market. Only 32-bit? I couldn’t find something on the rumpkernel.org site saying that. You got a page from them or a detailed review I can link to as a reference in these conversations?

                                                                                                                                        1. 3

                                                                                                                                          apologies - I might be conflating issues - there was one for the 4G of ram here - https://github.com/rumpkernel/rumprun/issues/74 && for the past few years there was no aarch64 support (arm 64) but the nabla containers folks (ibm) have recently added support for that in the past few months -> https://github.com/nabla-containers/rumprun/pull/21

                                                                                                                                    2. 2

                                                                                                                                      That’s a rather incredible achievement, congrats! (And btw, yes, I think it would benefit you to open source that, since devops people aren’t going to run proprietary kernels if they can help it - at least I certainly have no plans to!). EDIT: consider the AGPL, it’s friendly to server-side stuff.

                                                                                                                                  2. 1

                                                                                                                                    So, given an unikernel is running on a hypervisor (qemu in the case of ops), it is basicallg a highly privdropped restricted process (a qemu process) security-wise?

                                                                                                                                    If a process has no exec(), and runs as non-root in a chroot, with further restriction on which syscall it can do, it might still be more complex from the inside (it listens for signals, interacts with virtual filesystem, have the large kernel mapped in memory…).

                                                                                                                                    But a unikernel runs on top of an hypervisor, that is complex itself!

                                                                                                                                    So, the point is not to remove the system complexity, but rather to move the complexity away from the application being developped downto something not in user’s (or attacker’s) control?

                                                                                                                                    On the other hand, running application onto a docker host OS onto a hypervisor /is/ more overhead than just the unikernel directly onto the hypervisor with maybe some orchestrator that tell the hypervisor to start and stop unikernel VMs.

                                                                                                                                    What about re-writing Ops as an unikernel with an authentycated TLS/SSH protocol?

                                                                                                                                    So, do you expect people to run Ops on a bare-metal Linux (in order to reduce the stack) ? Otherwise, it is adding a layer on top and not removing one. ;)

                                                                                                                                    I like this quote:

                                                                                                                                    layers are only ever added, never removed

                                                                                                                                    – apenwarr’s colleague

                                                                                                                                    1. 3

                                                                                                                                      Qemu is utilized by ops cause it’s the most common machine monitor out there, however there is no reason why we couldn’t roll in support for something else and intend to do so. Google made their own monitor that interacts with KVM that provides GCE mainly cause the qemu codebase is huge and supports everything under the sun.

                                                                                                                                      As for the stack, no, OPS is intended mostly for those who would wish to deploy their applications to the public cloud - AWS, GCE, Azure, etc. although any ‘cloud’ (virtualized) environment will work. For instance when you spin up an instance in Google with this it creates a raw GCE image that does not have linux in it and we assume the majority of these users would be using these platforms.

                                                                                                                                      I guess to be more explicit - this tool is designed more for everyday ‘work use’ like terraform or puppet for compute environments that have many machines to handle that you would find in companies versus a hobbyist use (ops -> devops). I wouldn’t run this locally for dev purposes but for production deploys. Not sure if that makes sense?

                                                                                                                                      1. 2

                                                                                                                                        “But a unikernel runs on top of an hypervisor, that is complex itself!

                                                                                                                                        So, the point is not to remove the system complexity, but rather to move the complexity away from the application being developped downto something not in user’s (or attacker’s) control?”

                                                                                                                                        If hardware support exists, hypervisors are more fixed purpose with way less complexity and code than monolithic kernels most people use. They just don’t have to do as much stuff. From there, they can be designed minimalist fashion easier to verify for correctness (eg Nova microhypervisor) or security (eg seL4). The cool thing is the isolation paradigm they implement can then pay off in all kinds of use cases.

                                                                                                                                        On top of that, you then add the minimal amount of code necessary to support the application. Being small and clean slate, one might even use memory-safe language for it (eg MirageOS). That mitigates lots of attacks itself. Ideally, the API’s would also be made for developers to use them securely by default. From there, the extra failures show up in just the code the developer added in a nice box the hypervisor provides.

                                                                                                                                        Disclaimer: Assumes hardware is secure. Little reason to think that. People should still try to get stuff above hardware right.

                                                                                                                                        1. 2

                                                                                                                                          Yes - you got it. In this security model the user and the attacker are treated much the same - again this assumes the infrastructure is either managed (aws/google cloud) or private (openstack/vmware/etc.). I think we crossed the rubicon a very long time ago when even small 20 person engineering teams have hundreds of vms they are managing. One linux is hard enough to secure.

                                                                                                                                          As for hypervisors - we already have unikernel tuned hypervisors - IBM even has multiple patents on it. https://patents.google.com/patent/US9886303B2/en

                                                                                                                                          As for hardware yes and no. You can “assume” it’s secure cause if someone can easily go around rooting vms on AWS we have some fairly serious issues to deal with. Having said that I agree that our current hardware is fundamentally flawed and this won’t fix that but I also contend we are stuck with it for a while.

                                                                                                                                          1. 2

                                                                                                                                            You can “assume” it’s secure cause if someone can easily go around rooting vms on AWS we have some fairly serious issues to deal with.

                                                                                                                                            I assume they’re doing something different than most other groups that their resources allow them to do. It may or may not reduce their odds. They might also have better monitoring and response. What AWS deals with is not equal to or doesn’t extrapolate to what average company with virtualization can do. Maybe not even F/OSS virtualization since their local modifications might improve security.

                                                                                                                                            So, I look at what the open products and many closed claim they do, the nature of the dependencies (esp shared resources), and the odds of leaks coming through them. The odds were high. It’s why I advocated for physical separation where possible on embedded-style boards. The leaks keep coming on server CPU’s.

                                                                                                                                        2. 2

                                                                                                                                          Many people have accepted the hypervisor as an implied part of their stack; in that case removing application/kernel separation means higher performance and less abstraction.

                                                                                                                                          1. 1

                                                                                                                                            Or even unikernel on bare metal!

                                                                                                                                            I think that what we often ask for when we use a hypervisor is an array of similarly manageable machines that we can spin or stop.

                                                                                                                                            It would be interesting to get unikernels spawned via PXE boot on a couple of managed hosts. Bare metal metal.

                                                                                                                                            Spawn a unikernel on hardware like one would spawn a process. More, smaller machines.

                                                                                                                                            No vertical scaling though…