1. 29

    My pager went off this morning telling me it had one, then twelve, then forty alerts. That many problems all at once is a sure sign of either network, power, or monitoring system failure. On investigating we discovered one of our switches had spontaneously reboot. By the time we’d determined we had a switch reboot it was back up: we decided we’d schedule a maintenance window for the weekend and replace it Saturday.

    Alas, the switch didn’t wait. It reboot again later in the morning and then kept doing it. We keep spare equipment at the data center and brought a new switch online.

    For network failures like this we have a so-called out-of-band console. We’re able to log in to prgmr.com and debug network failures remotely. From this out-of-band connection I was able to look at the logs on our serial console and determine which equipment had reboot. Further I was able to see all of our equipment was still powered on–meaning switch failure rather than power failure.

    As @pushcx said, welcome back!

    1. 3

      I don’t want to kick a man while he’s down, but should we assume this means your network isn’t redundant then?

      1. 6

        We do have some single points of failure in our network. We’ve been eliminating them as time permits but it is a work in progress.

        We peer with HE and use NTT as a backup. If the event of an outage with HE we can get to the Internet with NTT instead. This switch that failed also had a ‘backup’ but not quite as warm as our peering connection. As a result of this failure we’ll be experimenting with Multi-Chasis Link Aggregation (MLAG).

        Do you have any experience with MLAG or other switch-level redundancy?

        1. 3

          Hey,

          Nice to hear a host (owner) admit where they have areas for improvement!

          I’m afraid it’s a long time since I had my hands on an IOS console, and I took a different direction after graduating, so I’m definitely not the guy to help you here!

          1. 3

            Do you have any experience with MLAG or other switch-level redundancy?

            I have a bit of experience with HPE’s IRF (on Comware), but only in a lab environment. As far as I can tell once you set it up (not terribly difficult) it works as advertised. The downstream devices obviously should be connected to at least two physical switches in an aggregated switch domain, and preferably use some link aggregation, like LACP or various Linux link bonding techniques.

        2. 3

          Succinct write-up! Cheers :)

        1. 12

          I’m planning to build (and open source) a ~13” e-ink tablet, with a kickstand so I can use it as my personal computer as well (alongside my Atreus keyboard). I’d ideally like to run OpenBSD on it—currently I use macOS for work for pragmatic reasons (on a 2015 Macbook Air), but wish to more fully align with my ideals and so my plan currently is to put OpenBSD on an external SD card and see what my pain points are.

          Would love to hear about the current best options for desktop BSDs! I just like OpenBSD’s philosophy, and feel I can make a fun project out of addressing any pain points I have with it (I’m a little worried to see the current state of font rendering, for instance—but that is completely unfounded and I really just need to dive into OpenBSD this weekend.)

          As far as my e-ink display project, it’s a tool that I want to exist but doesn’t. While researching to make sure I wasn’t reinventing the wheel, I did find an eerily similar product, but it doesn’t meet all of my criteria. I’m not worried about the cost of developing such a device, as it’s a passion project. Once it’s finished, I’ll gauge interest in crowdsourcing a production run for anybody else who would find it useful. I still have research to do on the current state of e-ink refresh rates and multicolor e-ink tech, as well as a hefty amount of research to do on actually architecting the device. Once I have any sort of progress done, I’ll be sure to share. :-) I’m just grinding through my life’s backlog currently.

          1. 16

            I’m a little worried to see the current state of font rendering, for instance—but that is completely unfounded and I really just need to dive into OpenBSD this weekend.)

            I can assure you—you get used to Comic Sans being the only font on the system. It’s not so bad.

            1. 3

              I can assure you—you get used to Comic Sans being the only font on the system. It’s not so bad.

              It’s not nice to tease /u/molloy like that. OpenBSD has lots of nice fonts packaged, like Source, Fira, Roboto, Noto, Cantarell, Ubuntu, etc.

              1. 4

                I love being teased ^_^

                But I appreciate your thoughtfulness <3

            2. 5

              I’m a little worried to see the current state of font rendering, for instance

              Is your problem that fonts look blurry? OpenBSD has freetype’s autohint disabled by default, the difference is night and day. See here for the general idea of how to enable it.

              1. 1

                JPEG artifacts kinda kill the comparison.

                1. 4

                  Hmm, let’s try again with an image host that hopefully doesn’t compress PNGs like imgur does.

                  Bonus: newsblur is among the worst I’ve seen when autohinting is disabled.

                  1. 3

                    Bonus: newsblur is among the worst I’ve seen when autohinting is disabled.

                    Hmm, are you saying NewsBlur makes thing blurry?

                    1. 2

                      The word newsblur there is a link to an image showing what it looks like with autohint enabled vs disabled. Hyperlinking the word newsblur probably made it look like a link to the site, I should have chose that better.

                      1. 3

                        Sorry, I was trying to make a joke, and it failed badly.

              2. 5

                I still have research to do on the current state of e-ink refresh rates and multicolor e-ink tech

                Prepare for some disappointment. :-) I mean, I’ve been checking refresh rates on e-ink OEM spec sheets for the past few years and they’re improving, but not even close to usable for a general-purpose display. See for example the PC Mag review of the product you linked to. (Unless perhaps you’re really patient and can adapt to paging rather than scrolling, etc?) It comes down to the basic physics of e-ink, which involves mechanical rotation of micro-capsules: sort of intrinsically slow relative to electronics. Some devices optimize for updating only a very small part of the display at a time: the best I know of is the reMarkable, which put a lot of work into that.

                As for color e-ink tablets, I’ve never seen such a thing, but a coworker claims that some Russian schools use them, just the colors are very faded.

                If (like me) you just want a sunlight-readable display, you’d be better off with a transflective LCD… which are also hard to find. Can’t beat e-ink for low power consumption, though.

                1. 4

                  OMG this sounds amazing! (I say this, typing on an atreus!)

                  Please do share when you’ve done it!!

                  1. 3

                    Yes indeedy I would read and upvote such a post with relish :)

                    1. 2

                      Will do! I’ll most likely blog about the process as well :-)

                  1. 2

                    We need a better classification and naming for these SgxSpectre/MeltdownPrime/SpectrePrime/BranchScope/Meltdown/Spectre-like branch prediction vulnerabilities as using arbitrary names is a mess. What was wrong with simply referring to these vulnerabilities by their CVE numbers? Do we really need a logo and marketing glossy for each new variant? I blame the Heartbleed people.

                    1. 3

                      Six months from now, you’ll remember the difference between CVE-2017-2345 and CVE-2017-2435?

                      1. 3

                        What about naming them from a rotating list of names, à la tropical storms? Particularly devastating ones could have their names retired. That or using some other arbitrary naming scheme that could complement the CVE numbers.

                        1. 3

                          We’re at number CVE-2018-9105 for this year, and it’s not even the end of March.

                          https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-9105

                          1. 2

                            How about a combination system, something similar to a set of DiceWare-style wordlists?

                            You’d likely need at least 5 lists to make enough phrases, and new wordlists can be chosen each year - maybe by some ridiculous contest system which could be used to promote computer security. An [proper-name’s adjective noun adjective verb] system might be fun, first thing I randomly generated with such a system was:

                            “Merlin’s Automatic Priesthood Relevant Chat” and MAPRC for short.

                            When you discover a vulnerability, when it’s assigned a CVE number, you’ll be given the opportunity to choose a new from a set of random rolls, and then for major vulnerabilities, the words used will be retired, and the system can exclude future rolls of the same acronym.

                            Interesting note: This might be impossible because of politics. I’m in Florida, and can tell you first-hand the ridiculousness that goes into naming. After protests by women and allegations of sexism, they started adding men’s and women’s names to the lists. The system has to be continually tweaked by an international committee at the World Meteorological Organization. In recent times after some very destructive hurricanes (Katrina, etc.), some women have protested that attributing destructive and deadly forces to women is inappropriate, and that such ‘negative’ violence and destruction is the realm of men. Then, in regards to the names themselves, they now include popular French, Spanish, Dutch and English names because they don’t want to use names from only one culture or appear exclusionary, but now in recent times some other people are complaining they don’t want “their” culture or country associated with these terrible things. I believe the list was determined by allowing for names and cultures from anywhere the storms hit.

                            Anyway, this is why we can’t have nice things.

                        2. 2

                          Probably not, but I’m not going to remember the difference between all these names anymore either! Perhaps something similar to the CARO/virus scanner industry solution, of a somewhat descriptive and roughly hierarchical name that’s hopefully self-describing … however, this still breaks in implementation with different scanners giving an identification of the same threat very differently.

                      1. 32

                        Git is horribly broken

                        I hear this sentiment a lot, but I’ve never heard a detailed argument as to why or how it’s broken from people who have invested time into learning it. I use it daily even when working on solo projects. While it had an initial learning curve and caused a few headaches while diving in, now that I grok it pretty well I can’t think of a better way to implement most of its functionality. Are there any git detractors that have used git extensively here who’d like to weigh in with git’s downfalls or better ways to implement a distributed version control system?

                        On a side note, if one’s frustration is with git in the command line, magit is phenomenal and is a good enough reason in itself to install emacs.

                        1. 17

                          Are there any git detractors that have used git extensively here who’d like to weigh in with git’s downfalls or better ways to implement a distributed version control system?

                          I use Git every day but I think the model could be better. I think the work in patch theory is useful:

                          https://tahoe-lafs.org/~zooko/badmerge/simple.html

                          https://pijul.org/documentation/model/#why-care-about-patch-theory

                          http://darcs.net/Using/Model

                          For example:

                          Pijul lets you describe your edits after you’ve made them, instead of beforehand.

                          1. 16

                            I think if you think people who struggle with git won’t struggle with emacs, you are about to be disappointed. Git has some rough edges with ux, and this is only made worse by the fact that the problem it solves isn’t trivial to understand. That being said I think the author is wrong, people will use git despite its usability shortfalls. requiring “always connected to the internet” is a fundamental misunderstanding of why git is presently popular, and since the author is investing in this direction they are poised to lose a lot of money. I genuinely can’t imagine a non-opensource VCS toppling git, either it’s just silly.

                            1. 15

                              I hear this sentiment a lot, but I’ve never heard a detailed argument as to why or how it’s broken from people who have invested time into learning it.

                              There have been papers about that. They were discussed previously along with the VCS built to adress the shortcomings:

                              1. 9

                                Not sure how Gitless fits here. Gitless is based on Git, at the core of Gitless, is Git. If Git itself is broken, how Gitless can fix it?

                                1. 2

                                  Gitless’s authors main complaint is that Git’s interface is conceptually complex, Gitless reduces the API and combines some of the underlying concepts in Git. It seems like a reasonable goal although I haven’t actually used Gitless myself so I can’t really give a first hand account on if it’s successful. In any case, it seems reasonable to build a functional system by some criteria on top of a broken system by that same criteria.

                                2. 4

                                  Thank you! Exactly what I was looking for, checking these links out now.

                                3. 7

                                  The only issues I’ve heard relate to the command line being inconsistent.

                                  1. 4

                                    while I think git is great at solving a complex problem and aside from the command line being inconsistent, the man pages have been awful for someone not knee deep in the nuances of how it works. For example, the man page of a rebase use to be laughably obtuse at best. It has gotten much better though as now it reads: git-rebase - Reapply commits on top of another base tip, which is about as succinct and clear as you can be for a complex feature.

                                  2. 4

                                    As someone who uses git a lot and has spent time doing fancy tricks with it:

                                    • The staging area is inconsistent (in particular in how it interacts with stash) and not useful enough to be worth the complexity budget. I would remove it entirely.
                                    • The command line is bad, in particular checkout is horribly overloaded. I would move the functionality of git checkout -- into git reset.
                                    • You end up with too many copies of every branch that can get out of sync with each other, made more confusing because git almost-but-not-quite hides your local copy of the remote. This introduces an extra asymmetry where fetch has no converse. I think I would probably remove the local copy of the remote and have an online-first workflow where you can only pull/merge remote branches when you’re online; if you want a local clone of the remote repo distinct from your working repo then set one up explicitly (as a bare repository).
                                    1. 8

                                      The staging area is inconsistent (in particular in how it interacts with stash) and not useful enough to be worth the complexity budget. I would remove it entirely.

                                      Yikes, no thanks! The stage is one of my favorite features. Building up a commit piecemeal is soooo much better than the old subversion commit style that encouraged devs to just throw in every dirty file.

                                      1. 1

                                        If you’re committing something that doesn’t include every dirty file you’re almost necessarily committing something you haven’t tested, which may well not even build. That’s a big drag when you come to bisect (which is one of the key advantages of git IME).

                                        1. 1

                                          If you’re committing something that doesn’t include every dirty file you’re almost necessarily committing something you haven’t tested

                                          Or, y'know, git stash -k -u && rspec

                                          1. 0

                                            That’s possible, sure, but when one makes the bad practice easier than the good practice one is going to have a bad time.

                                            1. 3

                                              There is absolutely nothing “bad practice” about building up a proposed commit, testing the staged commit, and then committing it.

                                              It’s certainly a better practice than the “tweaked the renderer, adjusted the logging, added two new export formats, and some other stuff I forgot about” kitchen sink dogshit commits that “commit all dirty files” inevitably leads to.

                                              1. 2

                                                The bad practice is committing something that doesn’t work. That’s much worse than a commit that contains two unrelated changes (which is not something I see very often - if you’re ending up with that try committing more frequently).

                                        2. 1

                                          commit --amend accomplishes the same thing with much less confusion.

                                      2. 3

                                        I like git, but I’ve had the chance to onboard smart developers who are more familiar with mercurial. They’ve made some convincing arguments as to why mercurial is nice to use.

                                        I don’t know if the next billion programmers will use git, but I have no doubt they can use git. Most people come out of a 6 week coding bootcamp knowing how to use git and github. That’s a pretty low bar.

                                        1. 1

                                          My main problem with just about every article I’ve ever seen espousing this sentiment is that it comes along with zero solutions.

                                          If you’re going to say “Git is bad.” then you need to also offer an alternate model of collaborative version control that works better. So far I haven’t seen that.

                                          1. 3

                                            I don’t think that’s true. What you’re saying is that I don’t have the right to say something is bad unless I also have the ability to fix it, which seems ridiculous to me. I hear that a lot with open source software, “if you don’t like it, just fork it and change it!”.

                                            Just because I use Git doesn’t mean I know how to write a VCS. There are, however, people who do know how to do that, and I can appeal to them in the hopes that they might agree with me.

                                          2. 1

                                            Sadly I hear the same quite often as well, our tech leads and architects prefer Accurev over git. While I can see some benefits for accurev, git was dismissed as being ‘too computery sciencey’.

                                            1. 4

                                              git was dismissed as being ‘too computery sciencey’

                                              Speaking as someone who writes software. That’s horrifying.

                                              prefer Accurev over git

                                              I just googled accurev. My gut reaction was that it looks a lot like a nightmare of a product I was once forced to use called IBM Jazz something.