1. 3

    Some facts that remove all the sensationalism:

    • This person was an IT contractor,. not an employee
    • They weren’t “fired” per se; their contract simply wasn’t renewed due to a silly oversight and bad processes
    • The “machine” had nothing to do with the termination, except for implementing some apparently-naive ticket scripting to revoke access

    None of this is shocking to me EXCEPT it’s unconscionable his staffing company (and “recruiter” as he refers to them) didn’t call to say plainly “your contract was sadly not renewed, do not go to work as of date X until a new contract may or may not be signed”. The fact his staffing company and host manager either didn’t know or didn’t tell him, and allowed him into the building to do work after the contract was up, are very problematic and could be legally actionable.

    1. 1

      This reads like a proper sci-fi short story and I suspect that a LOT of it was highly embellished. It makes sense to me that his direct managers would be confused as to what was going on, but it does not make any sense that it took them three weeks to figure it out. “The thing that I’m responsible for is on fire and my contractor can’t access anything” is a managerial complaint that should make its way up to the fucking CEO in a matter of hours if all anyone else wants to do is point fingers. So either important details were omitted to make it a better story or this big business is exceptionally bad at being a business.

      1. 3

        I think you have too much faith in the efficiency of business. There are many reasons people might choose not to escalate, for example - even if it doesn’t help the business. Most managers are in the business of protecting their little patch and their own career prospects, nothing else. Unfortunately, I found the story quite plausible (although it could be exaggerated as you say).

        1. 1

          You’re probably right. I’ve had only 10 months experience in a truly bureaucratic business myself (not counting the military which is actually suprisingly efficient, just slow) and that wasn’t even a big company. I do find it plausible, just the details far-fetched.

    1. 2

      FWIW, for Lets Encrypt I use acme.sh in a daily cronjob that renews the cert every 30 days (i.e. if it’s within 60 days of expiration). Then I have alerts that fire if the time runs down. It’s kind of aggressive but it protects against non-outage risks like subtle API changes or bugs.

      1. 2

        Does your acme.sh automatically restart your webserver processes? If so, does it run as root or do you use a sudo or something else to restart them?

        1. 1

          In my case it feeds the new cert into the configuration management tool and it then pushes out the new config like normal. acme.sh itself is decoupled from the web server.

      1. 1

        Interesting architecture, but not impressed by the 25M API calls per month.

        1. 2

          It’s always confused me when “queries per month” is used as a headline. That’s an indicator of the user base, not the technical infrastructure. On average it’s about 10 qps, which for this feature requires possibly a large Arduino. :) BTW, no idea why a web site owner would accept a paid dependency just to do a btree lookup “as a service”.

          1. 1

            Because they’re solving hard problems that will change the world through ip geolocation as a service.

        1. 2

          For somebody on the sidelines of Node, what’s the latest recommendation to avoid this? I’ve heard NPM supports namespaced modules like Go does, but is that predominant? Can it reference a package checksum/signature? Is there any way to control 3rd and 4th tier deps?

          1. 6

            It’s pretty easy…don’t use Node or NOM.

            I’m not kidding.

            1. 6

              This is pretty much it. Working with NodeJS/NPM even at distance is like having a romantic dinner in a fucking dumpster fire.

              I’ve had the displeasure of working (backend/ops) on some projects where they insisted on flavour of the week frontend stuff. So there was node, npm. A dozen other interdependent build tools. I ended up helping the guy running the project track down a bug - basically a downstream library had changed something and it broke stuff. That same module was included somewhere like 11 or 12 times, all slightly different versions. The dependency tree for that project’s JS files (keeping in mind this was frontend only) was like one of those “European royalty” family tress: everything is somehow related, and just looking at it makes you want to scrub your eyes with tobasco.

          1. 3

            I’m really curious about the background of Nim users. I imagine its more folks coming from a Python/Ruby/Smalltalk background as compared to a C/C++ background but I have no actual basis for that assumption.

            If you are an active Nim user, I’d love to know your background and why Nim appeals to you.

            1. 2

              I’m a python user; slowly trying to replace python scripts with nim. Nim’s standard library is pretty decent. Unless i’m doing something highly specialized (eg: I’ve not tried to do any AWS automation in Nim), nim is often quick enough to write with the same amount of effort I would have spent writing Python.

              1. 5

                I have 3 primary questions:

                • why are you trying to replace Python with Nim?
                • what it is about Python that has you wanting to replace it?
                • what is it about Nim that lead you to trying it as your replacement?
                1. 3
                  • over the last few years, I’ve come to see the value in a compiler and statically type checked language. Especially one like nim, where type checker is your friend and not onerous.
                  • I’m tired of tracking dependencies in a virtualenv etc., having a single binary is soo much nicer.
                  • There is a huge amount of head room to grow as a programmer (eg: I’ve barely scratched the surface of what I can do with macros, templates and other “higher” language features).
              2. 1

                FWIW, I’m a polyglot programmer and I really like Nim but the only thing I’ve used it for is the Advent of Code puzzle solutions last December. It’s a lot of fun to program with and I like that it compiles to a single binary with no deps like Go.

              1. 8

                It’s a super easy mistake to make! They’re so tiny, and hard to find! One day we’ll have a service where you can just type “how many legs do lobsters have” without personally going to Maine and renting a boat and lobster traps to find out.

                1. 7

                  The wikipedia article is fascinating:

                  Lobsters live up to an estimated 45 to 50 years in the wild, although determining age is difficult.

                  Research suggests that lobsters may not slow down, weaken or lose fertility with age, and that older lobsters may be more fertile than younger lobsters. This longevity may be due to telomerase

                  Lobster longevity is limited by their size. Moulting requires metabolic energy and the larger the lobster, the more energy is needed; 10 to 15% of lobsters die of exhaustion during moulting, while in older lobsters, moulting ceases and the exoskeleton degrades or collapses entirely leading to death

                  1. 5

                    I wonder what would happen if the exoskeleton was artificially reinforced in an old lobster. Would it live a lot longer?

                    1. 1

                      Isn’t there a similar story with squids, that they grow as long as they have the resources to do so?

                  1. 32

                    I don’t see why this progress bar should be obnoxiously put at the top of the page. It’s cool if you wanna do a donation drive but don’t push it in the face of everybody who comes here. Honestly at first I thought this was a bar for site expense. Then I realised it’s to ‘adopt’ an emoji.

                    1. 7

                      Lobsters isn’t a daily visit for most readers, probably even for most users. They can’t see it to join in if there isn’t anything visible for it, and it has an id for adblocking if you prefer not to see it.

                      1. 22

                        Personally a check this site quite regularly on my mobile device… which doesn’t have an ad-blocker.

                        1. 13

                          That sounds awful. If you’re an android user, normal uBlock Origin works on Firefox for Android just like it does on desktop. :)

                          1. 3

                            Or use Block This!, which blocks ads in all apps.

                            1. 3

                              Oh, that’s a cool little tool. Using a local VPN to intercept DNS is a neat trick. Unfortunately doesn’t help with in this case because it blocks requests to domains and not elements on a page via CSS selectors.

                              That does make me want to actually figure out my VPN to home for my phone and setup a pi-hole, though.

                            2. 2

                              Ohh! Good to know, thanks.

                            3. 2

                              Firefox 57+ has integrated adblocker nowadays, on both desktop and mobile; plus, there’s also Brave.

                            4. 27

                              That is still annoying that I need to setup my adblocker to fix lobste.rs. So much for all the rant articles about bad UX/UI in here.

                              1. 11

                                maybe one could just add a dismiss button or sometimes like that? I don’t find it that annoying, but I guess it would be a pretty simple solution.

                                1. 1

                                  I concur, either a client side cookie or session variable.

                                  1. 1

                                    Well, yeah… that’s how you could implement it, and I guess that would be the cleanest and simplest way?

                                2. 2

                                  It’d be great to see data about that! Personally I visit daily or at least 3 times a week. Lack of clutter and noise is one of the biggest advantages of Lobsters. And specifically, I looked at the link, and I have no idea who this Unicode organization is, or their charitable performance, or even if they need the money. I’d imagine they are mostly funded by the rich tech megacorps?

                                  1. 1

                                    [citation needed] ;-)

                                  2. 3

                                    Adopting an emoji isn’t the end goal: the money goes to Unicode, which is a non-profit organization that’s very important to the Internet.

                                    1. 5

                                      If this bar actually significantly annoys you, I’m surprised you haven’t literally died from browsing the rest of the internet.

                                    1. 4

                                      I’ve really enjoyed writing in Nim for this year’s Advent of Code puzzles. Its type inference support enables static typing, which I find comforting, with the clean syntax of a dynamic language like Python. In fact the syntax is largely similar to Python. Love that it compiles to C like a “real program”, and I’m interested to try out its JavaScript compilation backend.

                                      1. 1

                                        Glad you’re enjoying the language :)

                                        Be sure to pop into #nim on Freenode (or Gitter or Discord) if you’ve got any questions.

                                      1. 11

                                        It sounds like they’re on EC2, but haven’t migrated their thinking away from what you might do with physical servers. A different thing they could have done is build a new AMI, launch a new VM based on it, unmount & detach the EBS data volume, reattach the data volume on the new VM, and move the EIP. Basically the “pets vs cattle” idea.

                                        1. 4

                                          I was thinking that myself. They’re on disposable cloud systems, why are they doing anything except throwing them away?

                                          1. 1

                                            I got the impression they are on multiple cloud providers, not all of which support moving IP addresses.

                                            I agree that separating the egress address from the app server would simplify things though.

                                          1. 2

                                            FYI, on Linux you can use setcap to grant specific binaries access to bind to low numbered ports. Gone are the days of needing to run daemons as root. Once saw a team use iptables port forwarding to get around this, and it actually used a huge amount of cpu.

                                            1. 1

                                              This solution sucks though when the binary you need to permit is java or python or something anyone could write code with. The FreeBSD MAC solution is better because you’re permitting a user.

                                            1. 2

                                              This is disappointing.

                                              With an automated, zero-cost CA, there are very few legitimate cases for wildcard certificates, and the risks increase with their use.

                                              I don’t understand why LE couldn’t simply allow for higher thresholds on certificate issuance, and instead support certificates that are actually a worthwhile goal: free S/MIME that doesn’t involve suckling at the Comodo teat.

                                              1. 8

                                                The biggest use case for wildcard certs is SaaS. If I have 10,000 SaaS customers with hosted domains like customer.example.com, LE wouldn’t want to issue (and renew!) that many certs. It also may exceed their rate limiter.

                                                1. 3

                                                  Yes, this is exactly why I can’t use LE for my business right now.

                                                  1. 2

                                                    LE creates SAN certificates, which let you group together multiple domains under one certificate. So you can use LE for a SaaS product like this if you’re clever about automatically grouping domains together. See: https://letsencrypt.org/docs/rate-limits/

                                                    1. 5

                                                      I know that LE can support up to 100 domains in the same certificate with SAN certificates. But I feel like the complexity implied by grouping domains together is not worth the few hundred bucks of a wildcard certificate.

                                                      1. 2

                                                        I’ve not known many companies that want to publish their full customer list so publicly :)

                                                  2. 4

                                                    What are the risks for wildcard certificates?

                                                    1. 2

                                                      I do like the option when it’s there. For example when SNI is not available and you are running low on IPs.

                                                      1. 0

                                                        The main concern is phishing.

                                                        If you look at your URL bar and see a green lock next to https://www.paypal.com.mysite.biz/login.php, you’re a lot more likely to log in.

                                                        1. [Comment removed by author]

                                                          1. 3

                                                            I agree. If you can prove you own the domain, shouldn’t you be able to call your domain whatever you want and get a certificate for it?

                                                            So the real risk, it seems to me, is in the way you show that proof. If the CA asks for this proof in a way that’s not secure, that to me would be a problem.

                                                          2. 7

                                                            You may be interested to know that browsers limit wildcard certs to one level deep, for this reason.

                                                            1. 2

                                                              What does this risk have to do with phishing?

                                                              In any event, the CAs aren’t the right place to solve phishing, services like SafeBrowsing are.

                                                          3. 1

                                                            I like supporting wildcards but I do wish they’d dramatically increase the rate limits and decrease the suspension time. Getting banned for a week after a fuckup or bug is nuts.

                                                            1. 1

                                                              Agreed 100%.

                                                          1. 3

                                                            To see what a remarkably diverse speaker line-up looks like for a very technical conference, check out Syntaxcon in Charleston, SC. And 100% of these talks were top-notch.

                                                            1. 2

                                                              This is a cool writeup, but honestly this is not “Docker on OpenBSD” – it’s Docker running on a Linux VM on OpenBSD, the same as a Linux VM running on any other hypervisor.

                                                              1. 6

                                                                This is also how Docker runs on Mac OS. So I’d consider it a valid solution to interoperating with other platforms that run Docker. Sure it’s nothing magical, but it’s nice to know that virtualization on OpenBSD has reached a point where, if Docker is a requirement for your job (or whatever), it no longer means you can’t run OpenBSD as your host OS.

                                                                1. 1

                                                                  They use xhyve to do the lifting, which is based on FreeBSD’s bhyve!

                                                                2. 9

                                                                  That’s why I tagged it Linux, OpenBSD and virtualization. Editorializing titles is against lobste.rs rules.

                                                                  Still I find the story interesting as it shows at what level vmm virtualization is at right now. Unlike Linux VM hypervisors, vmm is a very young codebase.

                                                                1. 1

                                                                  I was kind of interested in running an IPFS Wikipedia mirror. I got through the install and 10GB download. But then the instructions said “now tell people the URL of your server”. But I don’t know anyone who wants this – that’s the point, to “help society”. I kind of assumed this would be like running a Tor node where some kind of discovery protocol would send traffic to my mirror. Did I miss something?

                                                                  1. 2

                                                                    You only need to tell people the URL of your server if you want to provide a public gateway. By mirroring the content and pinning it your node will distribute the content automatically to people who request it within the IPFS network, or via other public proxies.

                                                                  1. 1

                                                                    Why make one for $3500 instead of hundreds for $100 or thousands for $20?

                                                                    1. 4

                                                                      What happens when you make thousands of CDs and don’t sell them all?

                                                                      1. 1

                                                                        I’m guessing the organizational costs and stress related to taking care of a thousand sales wasnt as worth it as one big sale, also the bidding gives it value and a story.

                                                                      1. 1

                                                                        increase the size of the volume and expand the file system to match, with no downtime

                                                                        Are there Linux filesystems that can be safely expanded without unmounting the volume?

                                                                        1. 3

                                                                          not a linux expert at all – but my understanding is ext3 (and thus 4 as well) can be resized without un-mounting first.

                                                                          1. 1

                                                                            xfs_growfs supports online growth of XFS fileystems (although it doesn’t support shrinkage).

                                                                            Like pyvpx said, ext3+4 supports online resize.

                                                                            It also looks like Btrfs supports online resize.

                                                                            Is there any Linux filesystem that can’t be safely expanded without unmounting the volume? :)

                                                                            1. 1

                                                                              Indeed, there are a number - ext3, ext4 and XFS, along with btrfs and ZFS too.

                                                                            1. 1

                                                                              Tldr: building EC2 machine images (AMIs) with the needed Docker images pre-pulled reduces instance launch time.

                                                                              Except, then I have another step in my deploy process. A docker container shouldn’t be so large this would matter anyway, especially if pulled from a source close to the VM (e.g. S3).

                                                                              1. 1

                                                                                Using tools to create docker images and using tools to create AMIs seems like overhead. Maybe they will change the docker containers for new ones once the AMI is running, but still it feels like I would just deploy onto the AMI and spare myself some docker-related pain.

                                                                              1. 1

                                                                                Sounds like 10% of the functionality of ZeroMQ?

                                                                                1. 5

                                                                                  I’ve been a Linode user since 2005 and can’t recommend them highly enough. Excellent support, great performance and good prices (not to mention continuous free hardware and virtualisation upgrades over the years). There have been a few security hiccups over the past few years (mostly because of their previous web platform) as well as DDOSes, so bear that in mind. Neither have affected me though.

                                                                                  I also use AWS and Azure but they’re not really in the realm of “VPS provider”. FWIW, two providers I’ve not used but who I’ve heard good things about are Vultr and Bitfolk (the latter have a UK-only presence though).

                                                                                  1. 5

                                                                                    There have been a few security hiccups over the past few years (mostly because of their previous web platform)

                                                                                    I wouldn’t say that. The causes of some of their security hiccups have included an admin panel publicly exposed on the internet, user data stored on an internet-accessible machine that wasn’t monitored by their security team, not changing admin credentials that were compromised in a previous hack, and using ColdFusion (I kid, but apparently their ColdFusion stack had major obvious misconfigurations).

                                                                                    Stuff happens, but also their response in some cases has been pretty poor. They’ve avoided/buried disclosure, kept a potential compromise under wraps for months before disclosing, glossed over that they couldn’t figure out how one of the compromises happened and usually downplay as much as they can when they do disclose.

                                                                                    1. 1

                                                                                      Good points - I’d forgotten just how badly they’d handled some of the security issues. It was the previous ColdFusion-based frontend I was thinking of (as that was the source of a number of problems in the days before the more serious security problems).

                                                                                    2. 3

                                                                                      Same here. For straightforward VPS duty, I can’t think of a reason to go elsewhere. Currently I just have a small FreeBSD instance serving static webpages but I ran my YC startup on Linode as well and never had a reason to complain.

                                                                                      1. 1

                                                                                        Agree, I’ve loved Linode for years for my small “tinker box”. They are one of the few providers to include native IPv6, and they will even route you a /64, useful if you want to run your own v6-in-v4 tunnel (a la tunnelbroker.net). Fast and knowledgeable support. And they just did a major WAN network upgrade which cut my VM’s ping latency in half. You can usually find a $10 credit coupon from conferences or certain web sites.

                                                                                      1. 2

                                                                                        “We demonstrated this with a real war-driving experiment in which we drove around our university campus and took full control of all the Hue smart lights installed in buildings along the car’s path.” – Probably a good idea not to confess to what might be a felony in the middle of a research paper.