1. 5

    Found similar results where I work. We recently migrated from Erlang to Golang. Our ops team runs six instances of our Go server app on a 64-core EC2 server because each instance gets four cores/eight logical cores. Plus, monitoring and debugging seems easier with Go.

    1. 25

      Nice article. I must admin that I am a systemd fan. I much prefer it to the soup of raw text in rc.d folders. Finally, an init system system for the 1990s.

      1. 13

        I’ve never had a problem doing anything with systemd myself - I think a lot of the hate towards it stems from the attitude of the project owners, and how they don’t make any effort to cooperate with other projects (most notably, IMO, the Linux kernel folks). Here’s a couple of interesting mailing list messages that demonstrate that:

        1. 11


          I was initially skeptical about the debug ability of a systemd unit, but the documentation covers things to great depth, and I’m a convert to the technical merits. Declarative service files, particularly when you use a ‘drop-in’, are a definite step up from the shell scripts of sysvinit.

          The way the project tries to gobble up /everything/ is a concern though, given their interactions (or lack thereof) with other parts of the community.

          1. 2

            My impression is that the resistance to systemd stems from it not being unixy. Not being Debiany, even.

            I use for i in ..., sed, grep, awk, find, kill -SIGHUP, lsof, inotify, tee, and tr all damned day to mange my system, and systemd has left me blind and toothless.

            I’m still working on my LFS-based replacement for my various Debian desktops, vms, and laptop.

            1. 1

              Declarative service files, particularly when you use a ‘drop-in’, are a definite step up from the shell scripts of sysvinit.

              I’ve never found “systemd vs sysvinit shell scripts” to be a particularly compelling argument. “Don’t use sysvinit shell scripts” is a perfectly fine argument, but doesn’t say much about systemd. There are loads of init systems out there, and it seemed to me that systemd was never in competition with sysvinit scripts, it was in competition with other new-fangled init systems, especially upstart which was widely deployed by Ubuntu.

              1. 1

                In the case of Debian, it’s basically sysvinit or systemd.

                1. 2

                  That’s still not much of an argument for systemd; it’s just passing the buck to the Debian developers, and going with whichever they chose. That’s an excellent thing for users, sysadmins, etc. to do, but doesn’t address the actual question (i.e. why did the Debian devs make that choice?).

                  According to Wikipedia, the initial release of systemd was in 2010, at which point Ubuntu (a very widely-deployed Debian derivative) had been using upstart by default for 4 years.

                  Debian’s choice wasn’t so much between sysvinit or systemd, it was which non-sysvinit system to use; with the highest-profile contenders being systemd (backed by RedHat) and upstart (backed by Canonical). Sticking with sysvinit would have been an abstain, i.e. “we know it’s bad, but the alternatives aren’t better enough to justify a switch at the moment”. In other words sysvinit’s only “feature” is the fact that it is already in widespread use, with all of the benefits that brings (pre-existing code snippets, documentation, blogposts, troubleshooting forums, etc.).

                  These days systemd has that “feature” too, since it’s used by so many distros (including Debian, as you say), which was the last nail in sysvinit’s coffin: at this point sysvinit is mostly hanging on as a legacy option (Debian in particular cares very deeply about stability and compatibility). Choosing between Debian sysvinit and Debian systemd isn’t so much a choice of init system, it’s a choice of whether or not to agree with the Debian developers’ choice to switch init system. And that choice was between systemd, upstart, initng, runit, daemontools, dmd, etc. They abstained (stuck with sysvinit) for many years, until around 2015 when the systemd vs upstart competition was resoundingly won by systemd, with Ubuntu switching away from upstart and Debian switching away from sysvinit.

                  As I saw all of this going on, my interpretation was:

                  • Around 2005 every popular distro was using sysvinit because of its entrenched base, a few users advocated for alternatives like initng but the distros didn’t find the improvements to be worth the cost.
                  • Ubuntu switched to upstart, making init systems a hot topic: sysvinit became viewed as legacy, upstart was being looked at closely by other distros and it seemed like, once it got enough real-world usage, many might switch over.
                  • Systemd appeared, inspired by Apple’s launchd, and gradually gained users. At this point sysvinit was already seen as legacy and the question was what systemd offered that upstart didn’t.
                  • Debian debated switching init system, and with input from Ubuntu developers they both agreed that systemd was the better option (from my understanding, systemd’s “lazy” approach was a fundamentally better fit to the init problem than upstart’s “eager” approach). Upstart basically died at this point.
                  • All subsequent debates about systemd focus on how it’s better than sysvinit, which was never really in question.

                  To me, comparing systemd to sysvinit is like those shampoo adverts which claim their product gives an X% improvement, but the fine-print says that’s compared to not washing ;)

                  1. 1

                    OpenRC is drop-in and works perfectly fine. I dropped it in and I’m using it on all my installs with no issues.

                    1. 1

                      I think maybe you’ve misunderstood me.

                      I don’t mean you can install systemd and it will continue to work with sysvinit scripts.

                      I’m referring to systemd’s “drop-in” unit configurations. You can override specific parameters of a unit without having to replace the whole thing.


          1. 2

            What a great retrospective. All those hours of fiddling, frustration, and guesswork back in the early 1990s – now I kind of understand why.

            1. 1

              Link broken. Updated link seems to be: http://www.semibug.org/ipv6-handouts.pdf

                1. 8

                  I’m gonna mention qutebrowser which I love. It has vim-ish bindings and is a completely keyboard-driven browser. Just recently they switched the default backend to QtWebEngine which greatly increased performance and stability. https://www.qutebrowser.org/

                  1. 1

                    Thank-you for the link to QuteBrowser.org. It’s wonderful, so far. I’m planning to try nEXT also, but need to spend a bit more time getting it to run on Solus. QuteBrowser was already in the package repository.

                  1. 3

                    I love the comments at the bottom of the page. Especially the one from Doug Wyatt.

                    1. 2

                      Happy new year! “2013” still sounds futuristic to my old brain. But in the last few weeks of 2017, I had three goals form out of the blue:

                      1. Eat less (skip breakfast, and maybe eventually also lunch). Saw this on Lobsters.
                      2. As much as possible, migrate my personal systems from Intel x64 to ARM-based. Starting the year with Raspbian, hoping to end the year with OpenBSD. ARM is going to be huge?
                      3. Finally get to grips with VLANs, VPNs, host file blocking, and firewall rules in pfSense, to properly separate IoT and personal devices on my family home networks.
                      1. 10

                        This topic touches a nerve for me, after iOS 7 made the iPhone 4 so laggy that I felt that I had no alternative but to recycled it.

                        Apple could probably silence the critics with an iOS notification to tell users the state of their battery when the CPU starts to throttle due to battery issues. Though I would not place a bet on Apple doing that.

                        Fingers crossed that Microsoft’s alleged Andromeda device, and the Librem 5 device from Purism, can inject some fresh ideas into the market. The current mobile market needs a good shake-up.

                        1. 12

                          Fingers crossed that Microsoft’s alleged Andromeda device, and the Librem 5 device from Purism, can inject some fresh ideas into the market. The current mobile market needs a good shake-up.

                          I doubt people are willing to trust Microsoft again after they had backstabbed Windows Phone 8 users with lack of WM10 upgrades, and the WM10 users getting sunsetted, in a long line of backstabs of Windows Mobile users from them; and I severely doubt the Librem 5 will do much better than the FreeRunner, let alone N900, did. Unfortunate, but it’s based on precedent.

                          I think MS and others are waiting for the theoretical future form factor that will obsolete or at least put a dent into smartphone sales; the existing market is too entrenched, but a new one is fertile. The problem is guessing what’s going to actually take off.

                          1. 1

                            Honestly, if Apple started popping up notifications that say, “Your battery is old, and we had to slow the phone down,” they’d be ragged on for telling people to buy a new phone.

                            1. 1

                              Well my Parent’s Macbook Air is saying the battery “Needs Servicing” so it’s not like they aren’t warning their computer users…

                          1. 4

                            All the major desktop OSes seem to have ARM support now (Linuxes, BSDs, Windows) except for macOS. This has to be the future, and I’m ready to embrace any ARM laptop that could run Linux, BSD, or Windows with 22 hour battery life, with the potential to connect from anywhere. Even Vivaldi has an ARM build now.

                            I am just thinking out of my ass here, but I wonder if the reason macOS has felt so neglected over recent years is because some of Apple’s developer resources are being put into a new ARM-compatible desktop OS. Remember how Steve was all about the per-watt stats when Apple switched from PowerPC IBM to x86 Intel processors? It’s hard to imagine that Apple has forgotten that in the years since.

                            1. 16

                              I was just getting into GTD with Emacs org mode when I discovered Bullet Journals: http://bulletjournal.com/

                              With bullet journals, you keep everything in a small notebook in your pocket. It’s satisfyingly analogue, and less complex than GTD. I don’t do any of the fancy colouring or artistry. My journals are raw and scrawly, and don’t require batteries or a screen.

                              For everyday tech notes and writing, I still use org mode. But my personal and work stuff is now all tracked through bullet journals: a small pocket-sized Leuchturm 1917 for personal stuff, and a lined Blueline record book for work. I’ve been doing it for four months now, and it’s pretty decent. I think it’s worth a look-in if you would like an easy system to start with.

                              1. 5

                                I can’t enough good things about tracking my work with a bullet journal. I’ve been at it for almost three years and really appreciate the monthly (or weekly, as desired/needed) culling of unnecessary tasks.

                                1. 4

                                  I love the bullet journal approach, especially how it is specifically intended to be customized and improved upon. I discovered it about 3 months ago, and it’s the only productivity system I’ve ever used that I’ve managed to keep using for more than a couple of weeks.

                                  I personally use a dotted Moleskine notebook that is just small enough to stick in my back pocket so I can keep it with me everywhere I go.

                                  1. 2

                                    I use org-mode very heavily, but I don’t really like being tied to a computer 24/7. Given that you have experience with both, do you think there is a way to integrate Bullet Journals with org-mode? For now I have a pocket notebook that I will sometimes use to write lists of things that eventually just get transcribed to org-mode.

                                    1. 1

                                      After about a week of using a bullet journal I think org-mode serves a different but complimentary purpose. I’m using bullet journal for daily life tasks like dentist appointment and weekend plans with friends; org-mode for software, anything I do on the computer etc.

                                    2. 2

                                      Those who like bullet journals, but dislike the daily rewriting ritual / table of contents focus, should check out “final version perfected” by Mark Forster.


                                      This really helped me get out of a rut, and reboot my GTD workflow. Mind you, that happened in ~2012 or so, and only for a short period. I’m a full-time GTD person, and have been for a while. And i use org-mode and emacs to manage it.

                                      1. 1

                                        Been using a bullet journal here now for about 5 months, and absolutely agree! Mine’s not pocket sized, and I’ve recently teetered between using it for only work, or for work and other personal things. Seems to work best for just work, and I hadn’t thought of just getting another yet. Might give that a go!

                                      1. 1

                                        This sounds great on the surface, but it seems strange that the article mentions neither the total number of students taking the exam, nor the percentages of female or minority students. An increase in the number of females taking the APCS exam from 2,600 to 29,000 over ten years could be a statistical improvement, no improvement, or even a decline – it depends on the number of males. I don’t know what to think now.

                                        We desperately need more diversity in tech. Here’s hoping.

                                        1. 3

                                          Uh, yeah, it totally has charts of the percentage of female and underrepresented minorities. Did you read the whole thing!?

                                          1. 2

                                            Ah, you’re right and I am an idiot for not registering the y axis on the second graph. But glad to be wrong.

                                            Percentage up about ten percent in ten years. That’s decent.

                                            1. 1

                                              It’s 18.34% => 23.25% on the engineering-bound test: https://code.org/promote/ap

                                        1. 52

                                          I am on fastmail for my domain. Works fine, does everything I need.

                                          1. 7

                                            I am also a happy fastmail.com customer since about 2 years now. I used mailbox.org before, a german email provider, which is quite cheap (1€ per month) and allowed to use custom email domains but their spam filter sucked. Fastmail’s spam filter is also not perfect, in fact Gmail has still by far the best filtering, but their service is great and I can use custom email domain’s too. They also develop JMAP a JSON based IMAP replacement.

                                            1. 7

                                              I’d say the fact that JMAP is JSON based is only marginally-relevant; it’s got several significant design improvements over IMAP - e.g:

                                              • Folder renames no longer munge mail IDs (usually forces clients to re-download all messages).
                                              • No persistent connection (IMAP keeps your mobiles radio awake).
                                              • Flood control (some IMAP commands can send millions of identical lines in response).
                                              • Saving a draft with an attachment doesn’t make you re-send the attachment.
                                              • Subscribe to all changes in your mailbox via a single connection (vs one connection per folder)
                                              1. 1

                                                It’s more than IMAP replacement too, possibly better described as an alternative to Exchange ActiveSync.

                                              2. 3

                                                I’m with mailbox.org myself, with the 2.5EUR/month plan and a private domain. Mostly happy, I don’t have issues with spam. They seem to be quite opinionated on how to handle spam: https://www.heinlein-support.de/vortrag/spam-quarantaene-und-tagging-der-grosse-irrtum. But it seems classical spam tagging has been added recently, though I haven’t tested it: https://mailbox.org/update-des-webportals-bringt-nuetzliche-zusatzfunktionen-fuer-ihr-e-mail-postfach/

                                                I’m not that happy with the web interface though, it seems to be https://en.wikipedia.org/wiki/Open-Xchange.

                                                1. 1

                                                  Is JMAP even supported anywhere? Does anybody use it? Last I checked, not even Fastmail actually used this for anything. Seems like the project started with some energy but is mostly dead now? What a shame, as I’d love to use it somewhere… Please do correct me if I’m wrong.

                                                  1. 5

                                                    Hi, I’m some engineering guy at FastMail.

                                                    JMAP is currently going through the standardisation process at the IETF to become an RFC. Several companies have built or are building client and server implementations based on those drafts. We’re putting a lot of work into JMAP support in Cyrus.

                                                    At FM, we use it internally for some (but not yet all) of our UI-server interactions, and we’re working on converting the UI to use JMAP natively (once the standardisation work has stablised).

                                                    Finally, we’re just about to launch a new product that uses JMAP from top to bottom - Cyrus, Ix (a JMAP API generator) and Overture (a UI framework with a JMAP-backed storage layer).

                                                    So there’s lots happening on JMAP at FastMail and elsewhere.

                                                    1. 1

                                                      That’s really wonderful to hear. Once a year I email FastMail tech support asking them if there’s a JMAP thing, but the answer is always something like “no, and we don’t know when if ever.” And then I’m sad. This here is the first positive confirmation I’ve received, and I’m quite happy to hear it!

                                                      Hopefully once you release a fully JMAP designed system, you’ll have auto-exporters from existing tag-based systems like Gmail? Something like this would probably net you a massive user base.

                                                2. 7

                                                  I switched to fastmail last month and I am very happy with it. Before that, I had been self-hosting for 10 years, but I started seeing my emails listed as spam after I switched VPS providers (despite correct SPF etc), and I wasn’t motivated enough to fight for my IP reputation again.

                                                  1. 5

                                                    Also Fastmail, moved from Google Apps for domains 2 or 3 years ago. Besides the advantages others mentioned, subdomain addressing is also a cool feature. Some mail providers support plus addressing


                                                    subdomains addressing is a bit nicer. You can make disposable addresses in the form of:


                                                    makes it easier to write rules and to drop mail when the address is sold to some spammer.

                                                    Also their support is pretty good. I had a small feature/refinement request twice, in both cases they had the feature implemented in their beta site in a couple of days.

                                                    1. 5

                                                      I went to fastmail two years ago when the server on which I’d hosted my own email for about eight years died. I was happy to give a great company about $60 a year to host my family’s email. I was probably spending $60 a month of my own time just to administer the damn thing.

                                                      1. 4

                                                        I’m on Fastmail too, with my own domain, for about ten years. The web UI is focused and fast, and the iOS app is just a webview, but a decent one that’s quick. I use Fastmail aliases and inbox rules to send to multiple external addresses, like a basic private listserve. Tons of advanced features for mail users, DFA, and no advertising or shenanigans with your inbox.

                                                        They went through a purchase by Opera a while ago, then a few years later Opera sold the business back to the original Fastmail employees – not a single hiccup or business misstep the whole time. They are laser focused. They contribute back to the open source mail server community.

                                                        The only issue on my wishlist is that they still don’t support the full CardDAV protocol, which means I cannot fully sync my Fastmail addressbook with iOS, Mac, Windows, or *nix apps, but they’re working on it, and it’s due soon (early 2018?).

                                                        I think it’s cheap for what you get, if you’re into that sort of thing.

                                                        1. 1

                                                          What exactly is missing from CardDAV support? I’m happily using it to sync contacts to my iOS/Android devices.

                                                        2. 2

                                                          Same here. I use fastmail for every new domain that I need email for and it’s pretty great.

                                                          1. 1

                                                            Another vote for fastmail. Been a user for several years now. Has by far the best webui out of any provider. Very stable, and quick restoration of backups if you ever need them.

                                                            1. 1

                                                              Another +1 for Fastmail. I’ve used them for 3 years and have been pleased with all their services. Their documentation is clear, the system is not hard to use, and they answer questions promptly.

                                                              The only thing I’m waiting for is HTTPS support on their web hosting. But if you need serious web hosting, Fastmail probably shouldn’t be yout first choice.

                                                              1. 1

                                                                Yep, fastmail here too, it’s superb.

                                                              1. 9

                                                                Glad they’re progressing and improving UX. This…

                                                                “based on the “Security by Compartmentalization” principle”

                                                                …is remarketing of security terms. What they’ve actually built is a tiny subset of a Compartmented Mode Workstation (CMW) on top of the Multiple Independent Levels of Security (MILS) model w/ low-medium-assurance implementation. These are well-known concepts with tons of research into them and many commercial products. No need to reinvent terms. Interestingly, though, we saw the separation kernels certified as MILS rise around 2005, do great in pentesting, and ultimately be withdrawn as a central concept by NSA after isolation-only approaches didn’t prove out on Intel-style hardware that broke the model too much. Too many leaks and bypasses. Plus, you still have to have an extra component guarding information flows between security levels to enforce end-to-end security policy. Worked nicely in embedded systems, though.

                                                                In any case, such a design can at least isolate untrustworthy code from many types of attacks if the TCB is high-assurance. The CMW’s were also good at preventing accidental leaks of secrets by casual users since they checked labels. Here’s examples of both for those curious.




                                                                Note: Notice the colorful windows of CMW’s with bottom-stripe to make them unspoofable. Privilege architecture also tries to stop things like password grabbing. Stuff on bottom-right of INTEGRITY-178B gives a glance at all the kinds of stuff one might have to look into to create a secure TCB for MILS system. The DTIC document is likely the formal verification of it since it was only one done through Common Criteria w/ formal proof in those years.

                                                                1. 3

                                                                  I don’t think that the Qubes team has ever heard of SELinux. MILS is integrated into the sVirt solution specifically for isolating VM’s, but for some reason they absolutely refused to use KVM for almost that reason. The paravirtualization was always super off-putting to me, but the complete lack of doing prior-research bothers me far far more.

                                                                  1. 5

                                                                    I blasted them on the mailing list about not leveraging results of prior work. In that conversation, she didnt know about trusted paths, disagreed on Xen being security risk, didnt know benefits of user-mode drivers, and didn’t know about any of the competition that predated Qubes. She also countered my proposal to build on a security-focused microkernel saying Mac OS microkernel isnt secure. (Huh?)

                                                                    So, yeah, no confidence in that solution’s security. She later added trusted path and griped at Xen folks about their insecurity. No credit of course. ;) I relegated it to maybe a Linux hardening scheme with good usability. Still benefits to that for lay users.

                                                                    Note: Far as VMM’s, look up Nova microhypervisor’s design in the dissertation if you want to see the right kind of architecture for TCB reduction. Karger’s VAX VMM for security (esp layering).

                                                                    1. 2

                                                                      These are really great insights, Nick. Thank-you. I was looking forward to Qubes OS 4.0, but you’ve got me thinking deeper.

                                                                      Is Qubes OS providing a false sense of security, or does it still provide a genuine improvement over – say – running a standard Linux distribution with browsers in Firejail containers?

                                                                      1. 7

                                                                        I would take xen based security over linux cgroups or whatever in a heartbeat.

                                                                        1. 4

                                                                          I think in general you are accepting a certain degree of false sense of security when you are using Linux in general, it is not a high assurance operating system. I think Nick might agree that Linux is not a solution to the VMM layer and a better step might be to actually have an alternative that has formal methods for proofs or at the very least the VMM layer has formal methods. It was recently brought to my attention by Nick (thanks by the way) that the Xenon project exists just for the use case with Xen. I think if Qubes was dedicated to using Xen they would go after the Xenon project and attempt to get help from them to improve their stature instead of relying on solutions that have no strong guarentees.

                                                                          As for firejail I will point out that firejail is no more formally verifiable than Xen, in fact here is them fixing CVE-2016-7545. In my opinion it’s just a more fancy chroot, that doesn’t provide any leaps and bounds ahead in security improvements.

                                                                          Also as another piece of fuel for the fire just read this piece of documentation from the CubesOS team. Yes that is them actually suggesting to run everything as passwordless root sudo.

                                                                          In Qubes VMs there is no point in isolating the root account from the user account

                                                                          1. 2

                                                                            Re that last part: what’s actually wrong with letting an attacker have root in a VM where all interesting data is user-owned and fs changes aren’t persistent?

                                                                            1. 1

                                                                              Due to memory loss, I can’t answer that question in detail for this use case. What I do remember is I’d never give anything root in a UNIX-based deployment due to the Principle of Least Privilege or Authority. That’s a security pattern that says every component gets as little access and ability as possible to do its job. This is especially important if specific privileges (i.e. root) have led to privilege escalations in the past due to sloppy coding, sloppy configuration, or complex interactions between components nobody saw coming. Matter of fact, one of main reasons for POLA is to block attacks you don’t see coming or increase odds of detection mid-attack as they try to pivot through the system.

                                                                              So, it’s a bad idea in general. Anyone explaining why it’s safe when they give attackers a ladder toward the higher-hanging fruit is probably doing the wrong thing. Only time it’s sensible is when users demanded extra performance, features, integrations, or so on that made it absolutely necessary to add risk. Even then, better have a recovery strategy.

                                                                              Fun version of Saltzer and Shroeder’s security principles:


                                                                          2. 1

                                                                            Xen w/ Dom0 is a smaller attack surface than the average Linux distro. Clean separation of activities into VM’s with that attack surface is an improvement over other designs. That’s why I recommend it as a hardening technique. The better options are unfortunately going to be commercial and pricey if they’ll even pick up the phone. Those use a tiny kernel at the bottom that’s designed for security with minimal components and attack surface. Alternatively, you might use an OS such as OpenBSD that at least does a lot of code review and mitigations. There’s Linux solutions like grsecurity or SELinux to provide extra protections but people tend to find lots of bugs in the privileged code of stuff like Linux.

                                                                            I always used and still recommend physical separation with controlled sharing. I used cheap, embedded computers behind a KVM. Each one was a different security level with at least one air-gapped with no way to communicate with other stuff. There’s a lot of manual, technical work in such a setup. Yet, the virtualized stuff usually had problems or was unaffordable to me. (shrugs)