1. 2

    Relatedly: the 8 year outstanding bug to make Net::HTTP handle character encodings at all.

    1. 3

      One of the GoCardless SREs here.

      Happy to discuss anything and answer questions, though I’m going to bed in the next hour!

      1. 2
        1. What’s one lesson you learned from this incident that would be useful to share with developers who are not SREs?
        2. What’s one assumption you had challenged?
        1. 5
          1. Cold code is broken code. Code that only exists to handle failure is most susceptible to this. A more common example is an infrequently run (e.g. monthly) cron job. In many cases I’d prefer it to run daily, even if it has to no-op at the last second, so that more of the code is exercised more often. Better still, in some cases it could do its work incrementally! Either way is better than having the job fail on the day it really has to run.
          2. Our ability to manually take actions that have been handled by automation for a long time. Turns out that’s not so good, and prolonged the incident even after we’d decided to bring Postgres up by hand.
      1. 2

        The article’s claim seems bold. Could you not apply this quote to Prolog?

        Instead, we specify some constraints on the behavior of a desirable program (e.g., a dataset of input output pairs of examples) and use the computational resources at our disposal to search the program space for a program that satisfies the constraints.

        1. 4

          Yeah, one of the more common criticisms of this article floating around is that it seems unaware of large parts of the idea not being new. Which is a bit odd since Karpathy is a smart and well-read guy, so maybe it just leaves out the “related work” section for punchiness and rhetorical effect. But since the whole claim is that this is a totally new way of looking at software it makes for a weird read.

          Prolog itself doesn’t do exactly that; with standard logic programming, you encode the logic directly by writing clauses, rather than giving input/output examples. But inductive logic programming is a version that does; you give it input/output examples and it induces the program’s clauses. There’s also genetic programming as a somewhat better-known set of techniques. As well as program synthesis, a more formal-methods take on it.

          The real new part is the implicit claim that, essentially, “it works now”. GP is notoriously difficult to get anything useful out of, and clearly Karpathy thinks NN-based program induction won’t suffer the same fate. But that to a large extent remains to be seen…

          1. 1

            Gonna blame tiredness for that one. The bit in parens in my quote definitely doesn’t fit Prolog! The part I’m contesting is the idea that specifying programs in terms of constraints and relying on computers to explore a program space is new.

        1. 1

          Projects have their own goals, and I don’t see why those should be dictated by distros.

          I’m very much in favour of projects setting out their approach to support in a way that works for them. Ultimately, if $distro wants to maintain an ancient version of your work indefinitely, then good luck to them.

          One project I’m involved in has a take on this which boils down to “we work on all versions of Ruby and Rails still in security support by upstream”. It felt like a reasonable trade-off to make, considering the finite amount of time we have to work on it.

          1. 4

            The hard part is gonna be balancing my time between the two!

            1. 1

              I think with the newer Macs you can use Touch ID to protect keychain entries. Combine the two and you’re getting closer to the security level of the separate hardware key!

              1. 5

                The behaviour is really surprising when you’ve not run into it before (and hard to reason about even when you have).

                The title is super clickbaity though. I can’t think of a single mainstream relational database that defaults to serialisable transactions.

                1. 1

                  It may slip under the radar for many, but I have so much <3 for this commit mentioned in the article.

                  I have a pretty strong preference for handling failover at Layer 7 rather than Layer 2/3 (i.e. with virtual IPs). This change makes that way easier!

                  1. 6

                    I’m starting to find it odd when a service with 2FA doesn’t offer TOTP as the main option.

                    It’s widely supported. You don’t need a bunch of different physical tokens/separate apps to authenticate. It’s more secure than SMS.

                    1. 3

                      Most embarrassing is the fact that PayPal still only offers SMS. Their 2FA messages are often delayed or dropped, too.

                    1. 1

                      I think deadlines passed with every I/O (including lock acquisition) are the only way out of this.

                      https://golang.org/pkg/context/ is the only time I’ve seen it supported at a language level.

                      1. 2

                        Not really clear to me what the author means. Should everyone just use Spanner? There isn’t anything else out there like Spanner (although CockroachDB is trying).

                        1. 2

                          I don’t think there’s a production-ready equivalent that you can run yourself (closed or open source).

                          FoundationDB had a bunch of the guarantees, minus the SQL interface. Then Apple bought it and shut it down right away (side note: how terrifying is the idea of your database software no longer being available?).

                          CockroachDB doesn’t seem quite there yet. I really want it to be.

                        1. 1

                          Reminder that nothing is tradeoff-free.

                          Reminder that you’ll have to structure your data a certain way not to run into a throughput wall (true of many databases).

                          Reminder to read the Spanner paper to find these things out.

                          That said, Spanner seems dope.

                          1. 2

                            I am considering using postgresql for a project and the only thing that concerns me about it is the upgrade story. As someone who comes from using distributed DBs where zero downtime upgrades are the norm, several months of effort to do an upgrade in postgresql seems unacceptable.

                            Does anyone know if there are any plans to make this better?

                            1. 2

                              Random finding in my twitter feed after reading your comment: http://www.slideshare.net/dataloop/zero-downtime-postgres-upgrades

                              1. 1

                                Interesting. Unfortunately it still seems quite a bit more complicated.

                                1. 3

                                  Author of the talk here, it is. I think Postgres has a long way to go on upgrades and clustering.

                                  Since it’s not linked from that SlideShare page (and that page is controlled by the meetup hosts), here’s the video.

                            1. 4

                              The clickbait version of my opinion.

                              The nuanced version of my opinion.

                              I’d like to specifically call out the thing I talk about right near the end of the post. If you already have good automation around building VM images and blue-green deploys, containers probably don’t give you anything worthwhile (caveats: ease of using the same setup for development, machine utilisation).

                              1. 4

                                I didn’t realize they had to practically build an emulator for NES/6502 just to read a sound file. Wow. At the beginning, the author says the exploit activates without playing the file. Offers to explain that later. I must be overlooking the explanation. Why does it execute without opening the file?

                                1. 7

                                  It was briefly touched upon in one of the bullet points about attack vectors (it was seemingly unrelated, so you may have skimmed and missed it):

                                  When the Downloads folder is later viewed in a file manager such as nautilus, an attempt is made to auto thumbnail files with known suffixes (so again, call the NSF exploit something.mp3). The exploit works against the thumbnailer.

                                  1. 4

                                    Appreciate it! Makes me smile as I disabled thumbnails on most systems worried a parsing attack would happen at some point. I think they already did on Windows but can’t recall with bad memory. A general principle of mine is I want to control when something dangerous happens. Specifically, safe by default with me consciously making that the decision to do something risky and being aware of it.

                                    1. 3

                                      Absolutely. This is a close relative to the “autorun” exploit on older Windows versions where it would execute whatever was defined in a removable disk/drive’s root “autorun.inf” file.

                                      Ubuntu core maintainers should be aware of this type of attacks against thumbnailers, as there’s a ticket open for sandboxing thumbnailers (“gnome thumbnailers should have an apparmor profile”): https://bugs.launchpad.net/ubuntu/+source/totem/+bug/715874

                                      But no meaningful progress has been made to address the ticket apart from a PoC from 2011.

                                      1. 3

                                        Makes me smile as I disabled thumbnails on most systems worried a parsing attack would happen at some point.

                                        http://anti-reversing.com/Downloads/HES_2011_Presentations/USB%20Autorun%20attacks%20against%20Linux%20-%20Jon%20Larimer.pdf

                                        1. 1

                                          Nice write-up. Yeah, all kinds of issues apparently.

                                        2. 2

                                          The whole “parsing is one of the riskiest things we do” thing only hit home for me recently, when I read the qmail paper (PDF).

                                          In this case, the huge number of different parsers a file browser may decide to invoke is pretty damn scary!

                                          1. 4

                                            Indeed. And if you think about the number of frameworks and applications that make use of file(1), either directly or indirectly, to determine file types, you’d never sleep at night… OpenBSD’s implementation has been privilege separated since 5.8.

                                            1. 2

                                              That was a great paper. The people publishing the most on parser and protocol issues at language level are LANGSEC:

                                              http://www.langsec.org/

                                      1. 3

                                        The problem is in, however, how those images get produced. Take https://github.com/CentOS/CentOS-Do... for example, from the official CentOS Dockerfile repository. What’s wrong with this? IT’S DOWNLOADING ARBITRARY CODE OVER HTTP!!!

                                        What’s wrong with auditing the Dockerfile? Seems to me Docker is a lot more transparent than other methods. Thoughts?

                                        1. 5

                                          It’s nice that you can audit them, but they’re all written like this. Docker claims it can be used for reproducible builds, but the first lines in every single Dockerfile are apt-get install a-whole-bunch-of-crap and npm/pip/gem install oh-my-god-thats-a-lot-of-packages. Nobody is actually trying to manage their dependencies or develop self contained codebases, just crossing their fingers and hoping upstream doesn’t break anything.

                                          1. 1

                                            How is this different from build systems that don’t use Docker? Sure, you might be using Jenkins to build stuff (and have to manage those hosts for the OS-level packages), but the npm/pip/gem/jar, etc., there’s no difference. You still have to manage your dependencies. In my experience, the Docker stuff helps with the OS-level packages (previously we had multiple Jenkins hosts that had the versions of things specific to projects – god help you if you accidentally built your project on the wrong host).

                                            1. 4

                                              I use maven, where the release plugin enforces that releases only depend on releases, and releases are immutable, which together means that builds are reproducible (unless someone used version ranges, but the culture is to not do that). You can also specify the GPG keys to check signatures against for each dependency. It’s not the default configuration and there’s a bootstrapping problem (you’d better make sure the version of the gpg plugin cached on your Jenkins machine is one that actually checks), but it’s doable.

                                              1. 1

                                                On personal projects and at work I’ve been putting all the dependencies I use in the source repository. Usually we include the source code, for build tools (premake, clang-format) we add binaries to the repo instead.

                                                There are never any surprises from upstream, and you can build the code on any machine that has git and a C++ compiler.

                                                There’s some friction adding a new library but I don’t think that’s a bad thing. If a dependency is really too difficult to integrate with our build system then the code is probably going to be difficult too. If we need to do something easy people will write it themselves.

                                            2. 1

                                              At the risk of stating the obvious: if you audit the Dockerfile and it says “hey we downloaded this thing over HTTP and never checked the signature” there’s no way to tell if you got MITMed.

                                              1. 3

                                                Okay, so then you use another Dockerfile (or write your own). This is a very strange tack to take; you may as well say that Rust is an insecure programming languages because with a few lines of code you can create a trivial RCE vulnerability (open listener socket, accept connection, read line, spawn a shell command).

                                                For what it’s worth, almost every Dockerfile I’ve used installs its dependencies using something like apt or yum/rpm – and signatures are checked! And when installing via apt isn’t an option, Docker doesn’t keep you from doing the right thing (download over https, check signatures). You’re just running shell commands, after all.

                                                1. 1

                                                  My point exactly. There’s nothing wrong with taking an existing Dockerfile that you find to be suspect, beefing it up by correcting some obvious security issues, and resubmitting it as a patch.

                                                  I fail to see what the author of the article thinks is a better alternative. I’m open to be convinced otherwise, but saying it’s actively harmful seems overstated.

                                                  1. 1

                                                    For what it’s worth, almost every Dockerfile I’ve used installs its dependencies using something like apt or yum/rpm – and signatures are checked!

                                                    OK, so the signatures are checked. You still don’t know what version you got.

                                                    1. 4

                                                      Then pin the damned versions (apt-get install <pkg>=<version>), point at snapshot repos, and upgrade deliberately. This problem is totally orthogonal to Docker. All typical package repos suffer from it. I only know of Nix that doesn’t.

                                                  2. 1

                                                    This is why companies that care host their own registry for Docker images, just like they’ve done for Java, Python, Ruby, etc., for years. It is unfortunate that Docker didn’t design the registry system to be easily proxied, but this is easily worked around with current registry tools (Artifactory, for one).

                                                1. 9

                                                  I find this whole issue really interesting, and this post is really acutely timed for me, thanks for putting it up.

                                                  Early trials of Docker put me right off, but I’ve dug into the workstation client recently and I’ve been really pleasantly surprised. Seems a nice, simple way of running jail-like envs with nice isolation, which could most likely replace Vagrant in my workflow - if the deployment story is straight. But looking into that I find a bunch of stories like this, and this one is kind of the icing on the cake.

                                                  Is there anyone here on lobste.rs who’s using Docker really successfully in deployment systems and can give an insight into this? What’s the deal, are you getting more or less downtime and hassle? Are you having to hack round things to get things running smoothly like the guy in this post suggests? Do the benefits it brings compensate sufficiently? How comparable is the amount of work you’ve had to do to get a stable Docker workflow in place with what you’d have had to do using another system?

                                                  1. 6

                                                    We’re using Docker in production at work, and not looking to back away from that decision.

                                                    I’m not gonna sit here and say the original post is wrong - a lot of stuff in it is right. Yes, you need to write a script to clean out images (and it’ll be janky). Yes, something breaks in every release (the last two changed the output format of their syslog adapter, which was frustrating).

                                                    Honestly though? It comes down to approach. If Docker doesn’t give you (or a group of people in your organisation) some clear benefits, don’t use it. That’s a cultural issue, not a technical one. If you do decide it’s worth it, then remember this quote from Julia Evans:

                                                    You don’t just set up new software and expect it to magically work and solve all your problems—using new software is a process.

                                                    Oh, side note: we don’t run our databases (or anything stateful) in containers, but never say never. Docker may not be the container system most suited to it, but I don’t think putting cgroups and namespaces up around a database process is an inherently bad idea.

                                                    1. 5

                                                      jail-like envs

                                                      So, honest, honest question (please don’t tell me it’s just because duuuuuh, Linux users are stupid, hahahha, stupid LInux users)… why are we using Docker instead of BSD jails? I don’t really know much about either, but if jails is what people seem to think we should have done, why didn’t that become the popular option? The top google hit I can find for this question is that Docker is not at all like BSD jails, without further explanation. So, someone out there thinks that Docker does something that people need which BSD jails don’t do. What is that?

                                                      And I doubt it is “runs on Linux”, because seeing how the kernel seems kind of incidental (you need a VM anyway to run Docker on Windows and macOS), there must be a deeper reason. Can someone who understands both jails and Docker well enough explain?

                                                      1. 5

                                                        Docker provides a lot of management mechanics over top of raw containerization (where by my understanding—having actually used neither—e.g. LXC is much closer to jails in terms of raw functionality). I’ve personally found the Docker features I’ve used to be handy, though I can’t speak to how robust, well-designed, or generally applicable any of them are. And I think “runs on Linux”, or more precisely, “runs Linux binaries”, is actually a killer feature: there’s a surprisingly large amount of proprietary server software for Linux exclusively out there, for which jails provide zero help. Once you’re using it to run your Linux binaries on your Linux servers, the ridiculous contortions to also run it on non-Linux systems almost make sense, from the perspective of maintaining a consistent interface.

                                                        Also, Docker has a marketing department, which unfortunately almost always becomes the “killer feature” in a corporate environment.

                                                      2. 1

                                                        I’m late to the party here, but figured someone might still get value out of this: We use docker containers to send between 100 and 150 million emails a day, and to keep a few legacy applications together on some old hardware.

                                                        It’s a solution that more or less works, but the ‘Docker’ bit is the least reliable part of the whole architecture (CentOS, Docker, postfix, custom scripts). Basic commands often fail and require cleanup (e.g. docker attach) and there’s the docker daemon SPOF.

                                                        Networking and logging are more complicated and limited than I feel is necessary, and we don’t do anything with storage except for mounting postfix queue directories into the containers.

                                                        Would we use it again? Maybe. Our devs say they like Docker, but I think they like the idea of containerization more than they like Docker itself. I don’t see any huge advantages over something like LXC or rkt. I actually came to Docker from LXC, expecting something significantly different or better, and was baffled by the hype and popularity.

                                                        Although they’re architecturally different, I really like FreeBSD jails, especially with ZFS, nullfs, and other goodies that don’t exist on Linux. It seems like a much more solid base to build infrastructure on top of. See projects like cbsd (https://www.bsdstore.ru/en/about.html) if you want to see some crazy-cool ideas.

                                                      1. 9

                                                        There is, of course, a thread on HN (posting it here against my better judgement). According to one Googler:

                                                        I managed to find [the questions] and I don’t work in recruiting, they are for SRE pre-screens. The guy misunderstood most of the questions which is why he failed and then worded them incorrectly on his blog, it wasn’t the fault of the questions or the interviewer.

                                                        ([] edits are mine)

                                                        Make of that what you will, but the whole practise of this type of test (whatever the questions) is rather off-putting to me.

                                                        I can understand completely why Google interviews take the form they do - with the volume of applications they get, they need a system that filters the wheat from the chaff quite quickly. The problem I have is that the rigid Q&A with no room for discussion strikes me as far too inflexible.

                                                        1. 15

                                                          Make of that what you will, but the whole practise of this type of test (whatever the questions) is rather off-putting to me.

                                                          That’s because the whole premise is predicated on the power imbalance of “we’re Google, so jump through these hoops” rather than a discussion that paves the way to a deeply technical discussion. I’m not saying they’re being nefarious here, it’s more this weird institutional behavior that results from achieving any sort of notoriety, where the bar gets raised ridiculously high for potential hires because “omg one bad hire could ruin us.”

                                                          There are definitely interviewers that delight in this sort of thing, but I really believe this is a breakdown in a system where every candidate, even if they come in for an interview, is automatically ‘not-fit,’ and must perform near-perfectly in order to become ‘fit.’

                                                          1. 10

                                                            it’s more this weird institutional behavior that results from achieving any sort of notoriety, where the bar gets raised ridiculously high for potential hires because “omg one bad hire could ruin us.”

                                                            “One bad hire could ruin us” is an admission of managerial incompetence. If a company is so fragile against bad hires that an incompetent junior programmer can take the whole thing down, then maybe the VPs and the C-words earning $250,000 per year aren’t doing their jobs.

                                                            Also, “false negatives are better than false positives” is not always true. False negatives lead to false positives, because you still have to fill the role and if you shut out too many good people, you end up deeper in the barrel. Besides, people can’t be linearly ranked. The person who’s too picky to date people with/lacking Superficial Feature X at age 25 ends up dating a larger proportion those with Serious Deficit Y at 30, because that person rejected too many people for bad reasons.

                                                            1. 5

                                                              the power imbalance of “we’re Google, so jump through these hoops”

                                                              Exactly, and much like the “Techtopus” wage fixing scandal, the hoop-jumping just spreads from firm to firm. Some time back I read of someone interviewing with Amazon and (IIRC) he had seven interviews before being made an offer. Sheesh, I’m pretty certain medical doctors don’t have it so hard!

                                                              1. 4

                                                                I interviewed for a job at Mozilla and went through six interviews before being rejected. I did poorly on the sixth interview, and I understand why they passed after that, but the fact that there were that many interviews was a bit ridiculous.

                                                                I think they didn’t want to do a panel interview, so each member of the team I’d have potentially joined did their own interview. I would have preferred a panel, if only because it wouldn’t have used so much time or required so much reorganization of my schedule to accommodate.

                                                                1. 2

                                                                  What exactly does “six interviews” entail?

                                                                  • 6 individual phone screens on 6 different days
                                                                  • One day with 6 different sessioss
                                                                  • Six days with six sessions each
                                                                  1. 3

                                                                    1 phone screen. 5 technical interviews over Skype, each on a separate day, each requiring me to rearrange my work schedule to be at home in the early afternoon during the work week.

                                                              2. 4

                                                                where the bar gets raised ridiculously high for potential hires because “omg one bad hire could ruin us.”

                                                                I don’t think this frames it the right way. I have been heavily involved with the interview process for 2 unicorns under heavy growth phases. I’ve helped develop the interview process as well as performing the most interviews of my department last year. So I’ve done a lot of interviews. From that I can say, it’s not the idea that “one bad hire could ruin us”, it’s that when you’re trying to hire hundreds of people in a short period of time you can can let tens of bad hires in in one round, and that has the potential to be pretty bad. Striking that balance is really really challenging and it’s simply much easier to be conservative about it if you can afford it.

                                                              3. 5

                                                                filters the wheat from the chaff quite quickly

                                                                Or like…tosses a coin or whatever. ;P

                                                                1. 3

                                                                  I do wonder how long until they start to see a large increase in declines from actually taking part in their interview process.

                                                                  I obviously think that they will still get a steady stream of CV’s from fresh grads but I do think they will meet more ‘not interested’ replies from people they spear fish themselves. I know of at least a few people that don’t even want to bother with them but they would gladly go through the hoops a couple of years ago.

                                                                  1. 5

                                                                    They contact me every year or so, and I always say: “Would you still expect me to relocate to the bay area?” and the answer, so far, has always been “yes,” so…

                                                                    1. 4

                                                                      I’m done with them. Never again. But I’m not who they’re looking for anyway.

                                                                      1. 2

                                                                        I think the bigger risk is that by using this approach they will narrow their potential field of possible employees and thus end up with a lack of diversity.

                                                                        If you all think the same way how can you solve those problems that require a different approach?

                                                                        1. 5

                                                                          This is exactly the problem. If you talk to their recruiters they’ll tell you they are having a really hard time with diversity. When you go through their process, you’ll see why.

                                                                      2. 1

                                                                        wow. Protocols of interviews should not be accessible to the whole company. Also sharing a summary of them is a second privacy breach. Both things would be illegal in my country.

                                                                        1. 10

                                                                          How can sharing a list of questions that get asked during interviews be a privacy breach?

                                                                          1. 2

                                                                            Don’t know if we are on the same page. I was complaining about a Google employee writing a comment on HN based on internal information, that should be kept confidential (protecting the individual). So I am not against sharing the list of questions, I am against a Google employee sharing an assessment of the applicants performance.

                                                                            I don’t have much against that guy sharing the questions he was asked in the interview.

                                                                            1. 7

                                                                              I interpreted the comment to say “I looked at the actual questions we ask, and the ones in this post are similar but not the same.” If someone accuses a company of asking shitty questions, I think it’s fair game for the company to respond and say that the allegedly shitty questions have been misrepresented. The googler didn’t just show up out of the blue and announce “this guy sucked”. If you don’t want people discussing your interview performance, don’t write a blog post about your interview performance.

                                                                      1. 7

                                                                        I’m just in the process of switching to the ErgoDox EZ. One week in, and after a sweary first few days, I’m mostly typing without thinking about typing again. Very happy with the keyboard so far. Looking forward to customising the layout.

                                                                        This switch makes my history of ergonomic keyboards:

                                                                        1. 2

                                                                          I’m still on the microsoft keyboard line and very happy with the Sculpt Ergonomic. The compact design is a big improvement over the Ergo Keyboard 4000. Pricing of the Microsoft line of keyboards is roughly a third opposed to the ErgoDox EZ, but price may not be your first priority.

                                                                          1. 2

                                                                            The upgrade was very much a treat. I was on the 4000 for 5 years (though I had 2 in that time due to wear), and the Sculpt for a bit over 2. I had a £100 gift to put towards something, and thought I’d treat myself.

                                                                            Aside from the lack of mechanical keys the Microsoft keyboards are awesome!

                                                                          2. 2

                                                                            Hah, my order actually looks similar, except switch the ErgoDox EZ with the Sculpt. I went from the 4000 to ErgoDox EZ, but realized the ErgoDox is much too large for my hands, and I really wasn’t having a remotely good experience typing on it. So back to the Sculpt I go.

                                                                          1. 21

                                                                            With the company shutting down, we also wanted to find a new home for our team … We’re excited that the members of our engineering team will be joining Stripe

                                                                            If they actually went out and found work for the whole engineering team, that’s extraordinary. Bravo.

                                                                            1. 1

                                                                              I wonder if they went through a route like this.

                                                                              1. 3

                                                                                Offhand, I’d guess not. They both had same seed rounds from A16Z and A16Z is very good with relocating talent within portfolio (source: i’ve worked for like 5 of the portfolio companies)