1. 10

    At ${job}-2 I gave this a lot of thought and wrote a detailed guide on our conventions at the time: A Proper Server Naming Scheme. Essentially, the hardware would get a permanent/unique name for its lifecycle, and then CNAMES were added with more conventional structured names and convenience names. One detail I liked was using the UN/LOCODE codes instead of IATA airport codes for more specific geographic information.

    These days, I do a lot more work with dynamic/ephemeral hosts where it doesn’t come up as much; but, we still do have some static hosts and have settled on:

    <role>-<id>.<project>.<datacenter>.<provider>.<tld>
    

    …which ends up looking something like this for a Jenkins worker, for example:

    worker-0428a29567cb818a7.build.us-west-2.aws.example.com
    

    That said, being 100% in the cloud changes the situation a bit compared to pointing at bare metal in your own data centers.

    1. 2

      That’s an excellent naming scheme, which I’ve been using since a few years when there are less than 100 physical servers. I’ve found that above that number I’m more likely to use ephemeral hosts, where I do not really care what the naming scheme is because I very rarely need to connect to them.

      1. 1

        And then the hostname -s maps perfectly to this, and you can use hostname worker-0428a29567cb818a7.build.us-west-2.aws.example.com in confidence.

        I might copy-cat you…

        This also works in case of VM migration, which means network name change (unless you use loopback IPv6 everywhere then IPs might follow your VMs).

        Even bare metal in your datacenters can make use of this scheme I guess…

      1. 3

        If you’re putting binary files into git you’re doing it wrong. One could argue about small files, but compiled code/executables, photos or “gifs for the readme” are definitely misplaced in a git repository.

        1. 12

          I do find that having image files in a resources/ directory for something like a website is often simpler than separating the two. Even then making sure that images are compressed and generally not bloating repo size / git history is essential.

          1. 18

            I do find that having image files in a resources/ directory for something like a website is often simpler than separating the two.

            Yeah, the is exactly the use case here. Mercurial (and git) aren’t designed for handling large binary files, but if you’re checking in static assets/resources that rarely change it still tends to work fine. This repo was fine on Bitbucket for many years, and is working fine on an hgweb instance I’ve spun up in the mean time.

            I specifically asked about limits because if it’s just the size of the repo being a technical problem for their infrastructure, I can understand. But they would not specify any limits, but just reiterated several times that Mercurial wasn’t designed for this. So I don’t know which of these was the actual problem:

            1. The repo is so taxing on their infrastructure it’s causing issues for other users.
            2. The repo is so large it’s costing more to store than some portion of the $100/year account price can cover.
            3. They are morally opposed to me using Mercurial in a way that it wasn’t designed for (but which still works fine in practice).

            Cases 1 and 2 are understandable. Setting some kind of limit would prevent those problems (you can still choose to “look the other way” for certain repos, or if it’s only code that’s being stored). Case 3 is something no limit would solve.

            1. 3

              If you want to store large files and you want to pay an amount proportional to the file sizes, perhaps AWS S3 or Backblaze B2 would be more appropriate than a code hosting website? I don’t mean to be obtuse, but the site is literally called source hut. Playing rules lawyer on it read like saying “Am I under arrest? So I’m free to go? Am I under arrest? So I’m free to go?” to a police officer.

              1. 5

                B2 or S3 would make things more complicated than necessary for this simple repo. I’ve spun up a $5/month Linode to run hgweb and it’s been working great. I’m all set.

          2. 6

            This case was hg, but the same limitations are present. Hg has a special extension for supporting this:

            https://www.mercurial-scm.org/wiki/LargefilesExtension

            And it’s considered “a feature of last resort”. It’s not designed to deal with these use-cases.

            LFS support requires dedicated engineering and operations efforts, which SourceHut has planned, but is not ready yet.

            1. 5

              I have a repository with mostly PNG files. Each PNG file is source code; a chunk of data inside each PNG file is machine-readable code for the graph visually encoded in that PNG’s pixels. What would you have me do?

              I suspect that you would rather see my repository as a tree of text files. While this would be just as machine-readable, it would be less person-readable, and a motivating goal for this project is to have source files be visually readable in the way that they currently are, if not more so.

              git would not support binary files if its authors did not think that binary-file support were not useful; that is the kind of people that they are and the kind of attitude that they have towards software design.

              With all that said, I know how git works, and I deliberately attempt to avoid checking in PNGs which I think that I will have to change in a later revision. It would be quite nice if git were able to bridge this gap itself, and allow me to check in plaintext files which are automatically presented as PNGs, but this is not what git was designed to do, and we all can imagine the Makefile which I’d end up writing instead.

              1. 1

                I like the project, but pardon my ignorance - aren’t the PNG files still binary assets produced by the “real” source code, which is the textual expression parsed to generate both the embedded bitstring and the dot graph? If they’re machine readable, that places them in the same category as compiled object files.

                1. 3

                  The real source code is non-textual; it is the diagram (WP, nLab) which is being given as a poset (WP, nLab). To achieve optimal space usage, each poset is stored as a single integer which codes for the adjacency matrix. However, this compressed format is completely unreadable. There are several layers around it, but each layer is meant to do one thing and add a minimum of overhead; JSON (in the future, BSON or Capn) for versioning and framing, and PNG for display and transport. There isn’t really source code; there’s just a couple Python and Monte scripts that I use to do data entry, and I want them eventually automated away in favor of API-driven development.

                  For example, the raw integer for this “big” poset is (at the time of writing) 11905710401280198804461645206862582864032733280538002552643783587742343463875542982826632679979531781130345962690055869140174557805164079451664493830119908249546448900393600362536375098236826502527472287219502587641866446344027189639396008435614121342172595257280100349850262710460607552082781379116891641029966906257269941782203148347435446319452110650150437819888183568953801710556668517927269819049826069754639635218001519121790080070299124681381391073905663214918834228377170513865681335718039072014942925734763447177695704726505508232677565207907808847361088533519190628768503935101450078436440078883570667613621377399190615990138641789867825632738232993306524474475686731263045976640892172841112492236837826524936991273493174493252277794719194724624788800854540425157965678492179958293592443502481921718293759598648627823849117026007852748145536301969541329010559576556167345793274146464743707377623052614506411610303673538441500857028082327094252838525283361694107747501060452083296779071329108952096981932329154808658134461352836962965680782547027111676034212381463001532108035024267617377788040931430694669554305150416269935699250945296649497910288856160812977577782420875349655110824367467382338222637344309284881261936350479660159974669827300003335652340304220699450056411068025062209368014080962770221004626200169073615123558458480350116668115018680372480286949148129488817476018620025866304409104277550106790930739825843129557280931640581742580657243659197320774352481739310337300453334832766294683618032459315377206656069384474626488794123815830298230349250261308484422476802951799392281959397902761456273759806713157666108792675886634397141328888098305747354465103699243937608547404520480305831393405718705181942963222123463560268031790155109126115213866048693391516959219000560878337219324622230146226960346469769371525338127604307953786112516810509019551617885907067412613823285538493443834790453576561810785102306389953804151473860800342221969666874213156376831068606096772785272984102609049257833898258081466729520326827598704376424140779421965233471588921765110820238036094910936640446304632443760482611408445010230964335747094869968021425396439555206085281953007985784739643408074475440039274314217788647485602069097474262381690379456154426900896918268563062231294937080146199930562645748389040251871291840481739518244706752426504146889097315360662429293711705265772337748378759001582638301784557163848933046038798381667545043026975297902178839764134784634179453671000024868722179355800776002690855305662785522771116635997791339179517016284742206819482196944663461005128697584753594559406283638837841370287286682993990297923202976404261911087739188860505577427942276773287168600954693735964671046522557013031834557159173262849132567983767216098382093390056878765856939614383049277441.

                  1. 1

                    Ah, okay, I see. Makes sense, thank you for explaining!

              2. 4

                I’ve seen this argument quite a number of times, and almost always without a coherent explanation of why is that wrong. What’s the rationale behind this argument?

                1. 4

                  Shameless plug, I contributed heavily to this help topic back when I was the PM for Microsoft’s Git server: https://docs.microsoft.com/en-us/azure/devops/repos/git/manage-large-files?view=azure-devops

                  FWIW I disagree with the comment up-thread which says that GIFs for READMEs don’t belong. If you’re going to check in a logo or meme or whatever, that’s perfectly fine. Just don’t do 1000 of them and don’t churn it every week.

                  1. 2

                    I think a big part is also “are my tools there for me or am I slave to my tools?”

                    If I have a website and most content is under version control, it’s annoying and complicated to have (big) assets outside. Most people simply want one repo with everything inside, and it’s mostly additive, often once per week - it simply doesn’t matter if it’s the wrong tool.

              1. 1

                Writing the business plan for the infrastructure/DevOps/cloud consulting service which I plan to launch before the end of the year, hopefully.

                1. 5

                  A stupid question from somebody that doesn’t really know anything about PKI: is it a problem that they make certificates for 30% of domains? Would it be a problem if they made certificates for 100% of domains?

                  1. 18

                    I kind of wish that Mozilla, Apache, maybe Microsoft etc would offer similar services with a compatible API so we could spread the load a bit and diversify. Wishful thinking though

                    1. 3

                      I haven’t tried their services, but https://www.buypass.com/ssl/resources/acme-free-ssl claims to offer free SSL certificates by using the same ACME protocol as Let’s Encrypt. It might be worth checking it out, if only for the sake of diversity as you say.

                    2. 6

                      The danger that jumps out at me is that getting your root CA trusted by browsers is a slow and expensive process. If LE gained a monopoly and tried to cash in, getting a competitor up and running would be a non trivial amount of effort.

                      That said I think there are factors that work against that as well:

                      • big companies need the ability to purchase certificates with >90 day lifespan because enterprise
                      • other major players offer similar products — Amazon’s free certificates are even lower friction (in the AWS ecosystem ;))
                      • governments have anti-monopoly counterweights built into the system in the form of their own trusted CAs that could be used

                      Overall I don’t think anyone can argue that LE hasn’t dramatically improved the landscape of TLS CAs.

                    1. 1

                      Oh no, not again.

                      It’s clearly time exim was sent to the great mailer-daemon in the sky. How many RCE CVE’s in the last two years? Too many.

                      Sadly there don’t seem to be any open source SMTP servers written in memory safe langauges around. Unless I’ve missed one?

                      1. 1

                        Fortunately there are SMTP servers with a proper design that greatly reduce the severity of the effects of memory corruption.

                        1. 2

                          I disagree that those 2 examples are good solutions: Postfix’s configuration is even less readable than Exim’s, and OpenSMTPd is really under-documented and looks much more trouble to run on Linux than it’s worth. Exim is still the least bad of the bunch.

                          1. 1

                            All of them should be run in a container (or jail/chroot) if not a VM (QubesOS). Furthermore, we need to get rid of root.

                            **edit, oh jebus. It looks like ASN.1 parsing strikes again. The most profitable back door in the history of computers.

                        1. 0

                          Too little, too late. Surpassed by Ansible

                          1. 2

                            Ansible, the security nightmare of allowing automation to SSH to machines, copy over a python blob to a temp directory, and execute it as sudo/root.

                            Yes, people should really use that instead. /s

                            1. 2

                              It is also unbelievably slow at what it does, despite applying all documented optimizations and using Mitogen. The simple existence of Mitogen points out that Ansible’s design is fundamentally wrong.

                              1. 1

                                Also: Ansible, the tool that uses YAML for semi-declarative semi-programming automation. I kinda like it but feel dirty for doing so.

                                I kinda like (in a less kinky way than I like Ansible) that Chef is at least using a real programming language.

                                And I like that they wrote Habitat in Rust. But that’s also a bit kinky.

                                1. 2

                                  I kinda like (in a less kinky way than I like Ansible) that Chef is at least using a real programming language.

                                  How much Chef have you written? Every Chef best practice ever cites the fact that if you drop into Ruby it’s a major anti-pattern.

                                  I don’t mean that as an attack, I’m honestly curious.

                                  1. 1

                                    How much Chef have you written? Every Chef best practice ever cites the fact that if you drop into Ruby it’s a major anti-pattern.

                                    None at all :) Good to know, that wasn’t obvious.

                              2. 1

                                I don’t think that’s accurate at all. I also don’t think Chef is doing this just to compete with Ansible.

                                I don’t have any source of for the market breakdown, but I’m guessing it’s a safe bet Ansible is the market leader. With that said, though, I think there’s still room in the market for other entries. Saltstack has been making pretty big inroads for example.

                                1. 1

                                  @steveno is right, you’re oversimplifying and also not taking a number of important factors into account - chief being installed base. There is a HUGE amount of Chef code running out there in production.

                                  Also, Ansible is amazing but it is not well suited for every task, especially configuration management challenges that embody a fair bit of implementation complexity.

                                1. 5

                                  I use WordPress for https://www.datafaber.com because by now I know it well enough to have built a decent semi-automated workflow for updating it and backing it up. I have 4 other sites running on it and in a former company we ran a high-traffic site where WordPress served ~ 90% of the pages.

                                  1. 3

                                    PHP 7 has made a world of difference in terms of performance when it comes to WordPress.

                                    I get about 2,000 visitors (unique’s?) any given day. I’m using a low end shared hosting platform that sets me back about 1 USD a month. After blogspot went down for a week I’ve been self hosting.

                                    1. 2

                                      Yes PHP7 made running WordPress robustly much simpler and cheaper.

                                  1. 1

                                    I self host since a few years, using Exim, Dovecot replicated on 2 different servers at 2 different providers, Rspamd and Rainloop for the few times when I have only a web browser available.

                                    I have SPF, DKIM and DMARC records, I’ve had the chance of inheriting relatively clean IPs from the providers in question (Hetzner and OVH) and I’ve never had any issue in removing those IPs from the 2 or 3 blacklists the previous owners had managed to get into.

                                    For the moment, both Gmail and Hotmail accept my emails (that is, no bounces, no deliveries in spam folders and no email disappearing into thin air).

                                    I do not plan to move to a hosted provider for my email, the only maintenance I perform is staying up to date with the above packages and having some alerts in the logs if something goes wrong.

                                    If I had to do it again, I would use OpenSMTPD instead of Exim, only because reading the configuration file seems way easier.

                                    1. 5

                                      I’m not sure about Heroku part. I recommend a small instance on Prgmr.com. If you are reading this, then they can handle running Lobsters. They always do. ;)

                                      1. 5

                                        I’ll ruin the joke in favor of clarity, and to explicitly thank the folks at prgrmr. Lobsters is not only hosted on Prgmr.com, but has @alynpost is the owner of prgrmr and a sysop here.

                                        Thanks for all you guys do - community-wise and tech-wise!

                                        1. 6

                                          Thank you both. I’ll close the loop by saying that lobste.rs is running as a Xen DomU with 2 vCPUs, 8GiB RAM, and a 50GiB disk. Since beginning to host the site last year we’ve added a 2nd vCPU to deal with contention between the MariaDB work queue and the Ruby / Unicorn work queue. We’ve doubled the memory from 4GiB as traffic and utilization has demanded. The disk is DRBD and replicates to a secondary RAID10 on another physical host in the same rack.

                                          We’re under 50% disk capacity, and sites with less traffic can certainly be tuned to run on machines with less than 8GiB RAM. We use memory in part as cache to improve responsiveness.

                                          All that said, at least one lobste.rs user is a Heroku engineer, @apg. As @355E3B reports the codebase can be deployed to Heroku. I do not know what instance size you’ll need but other folk probably do.

                                          1. 4

                                            Just curious, but why DRBD and not MariaDB replication? DRBD is very fragile and difficult to get right in my experience.

                                            1. 5

                                              DRBD works at the block-device level, and so integrates with our cluster management software, Ganeti. That lets us failover or migrate instances between physical hosts ~regardless of what applications those instances are running. It solves a general problem of moving instances between physical hosts for us.

                                              I’ll give database replication a closer look if and when the compute resources required to run the site exceed what we can get with a single physical host.

                                              Some of this answer you can chalk up to path dependency. However, I have not found DRBD to be fragile or difficult to get right. It does what it says on the tin for us.

                                              Your book looks interesting, btw. Congratulations on publishing it.

                                          2. 1

                                            Oh it’s fine haha. Yeah, let’s go ahead to thank everyone hosting, admining, coding, and moderating the site for their time. I appreciate it a lot. :)

                                        1. 3

                                          @feoh why the heck is your blog grey on white? I’d love to read this but even after I increase the text size twice it’s still hard on my eyes.

                                          Contrast Rebellion - to hell with unreadable, low-contrast texts!

                                          1. 2

                                            Please take another look and see what you think of the new theme I installed. It’s the only theme in the default Wordpress arsenal that cites high contrast and accessibility.

                                            I couldn’t figure out how to adjust the text color in and of itself. Sorry, i’m not a web dev :)

                                            1. 2

                                              As a reader I thank you very much for taking into account the remarks.

                                              I really enjoyed the article, I’m still a junior in sysadm/ops and I hope I will Learn as much as you do!

                                              1. 1

                                                Welcome to the fold! It’s an incredible career path and I love my job to bits and am regularly excited to get up and go to work in the morning :)

                                              2. 2

                                                This text is much more readable. The layout of the site has lost a bit of ‘style’ and your header graphic is the same as the article graphic now which looks like a bug, but if you’re going for accessibility this is a bit better.

                                                I guess digging through Wordpress theme CSS is not much fun, but your original theme just with a tweaked font colour would have been fine too ;)

                                                (And nothing against Wordpress here, use it when I have certain kinds of projects that need to get deployed v. fast and with certain kinds of user constraints)

                                                1. 1

                                                  Digging through the CSS isn’t an option for me. I’m a System Development Engineer with Amazon Web Services. I mean, I know enough CSS to set a background and maybe change some spacing in HTML, but I haven’t the foggiest about how to dig in and modify a particular CSS attribute in Wordpress.

                                                  I’ll play with the theme more, I’d bet dollars for donuts that there’s a way to get the header graphic for my blog back, but accessibility is super important to me, so if I can’t with the time I have available then that’s a price I’m happy to pay.

                                                  Thanks again for the report.

                                                  1. 1

                                                    Ah. Interesting. In point of fact I CAN’T modify the CSS myself. To do that I’d need to go from paying wordpress.com $100 a year to $200 a year. Not gonna happen :)

                                                    1. 1

                                                      I have a dreamhost account which I use for their free unlimited Wordpress hosting, because it’s generally zero hassle and is a ‘proper’ full Wordpress install. Happy to host your WP there if it’s any use, with couple of caveats.

                                                2. 1

                                                  I’m partially blind so I’m super sensitive to this. Thanks for letting me know, I will choose a different theme post haste.

                                                  If you can manage to refrain from taking the usual dump on Wordpress (It’s what I use and like. Please deal appropriately :) do you have any suggestions on higher contrast themes you like? Or even other Wordpress blogs you find more readable?

                                                  1. 2

                                                    I like almost all of Anders Noren’s themes: http://www.andersnoren.se/teman/

                                                    The code quality is better than the average WordPress theme, and every one of those looks clean and readable (to me at least).

                                                    1. 1

                                                      I’m gonna confess to using wordpress.com so I pretty much only use themes they provide by default, but thanks for the pointer. If I get time and if I can install random themes I’ll definitely look into it!

                                                1. 1

                                                  Between DNS-based blocklists, anti-spam filters and general inbox overload, email is a very fragile medium for communicating anything, let alone authentication credentials.

                                                  There’s absolutely zero guarantee than an email would be delivered at all: Gmail and Office 365, to cite just a couple of the big email providers, sometimes drop incoming email without any notification for the sender.

                                                  Also, there’s absolutely zero guarantee that an email will be delivered quickly enough for this scheme to work.

                                                  1. 6

                                                    I also run my own DNS server, but I prefer to maintain just the master. I pay ~$15/yr to outsource the slaves to a third party company who specializes in such things, and I don’t have to worry as much if my VPS provider decides to go down for a few hours, etc. I get a more reliable DNS system, and I still get to maintain control, graph statistics, etc, to my heart’s content.

                                                    Glad to see the discipline of self-hosting isn’t completely going the way of the dodo in this day and age!

                                                    1. 2

                                                      Any recommendation for a good third part company for such outsourcing?

                                                      I also run my own DNS. The main reason is that I run my own mail using https://mailinabox.email/, which has been a reasonably simple and pain-free experience. Paying someone to get better stability could be interesting.

                                                      1. 3

                                                        I have added nameservers from BuddyNS to my secondary DNS. For the moment I’m just using their free plan since I’ve delegated to only one nameservers out of the 3 which are serving my zones, and the query count is low enough to keep me on the free plan.

                                                        1. 1

                                                          I loved BuddyNS but I went over their query limit and the only payment they accept is PayPal and I boycott PayPal after they stole $900 from me… I wish they would take other forms of payment

                                                        2. 3

                                                          I asked for some recommendations online. My biggest requirements were a ‘slave only’ offering, DNSSEC/IPv6 support, and ‘not Dyn’ (I just can’t give Oracle money these days). With all that in mind, I ended up choosing dnsmadesimple.com (edit: looks like they’re $30/yr, not $15 as above. Mea culpa) It was seriously easy to get everything set up (less than 20 minutes!) and now I don’t have to worry about what happens when my master goes down.

                                                          1. 1

                                                            Do you mean dnsmadeeasy.com or do you mean dnsimple.com?

                                                            dnsmadesimple.com doesn’t exist

                                                            1. 2

                                                              My deepest apologies, this is what I get for Internetting when I’m about four cups of coffee short.

                                                              dnsmadeasy.com is the correct one.

                                                          2. 3

                                                            Hello everyone! This is my first post. :)

                                                            I’m Vitalie from LuaDNS. We don’t offer slaves right now (only AXFR transfers), but if you don’t mind to fiddle with git, you can add your Bind files to a git repository and push them to us via GitHub/Bitbucket/YourRepo. You can keep using your DNS servers for redundancy as slaves.

                                                            You get backups via git and free Anycast DNS for 3 zones. :)

                                                            Shameless Plug

                                                          3. 1

                                                            Interesting - that’s not a bad idea.

                                                            If I were a corp I wouldn’t want this method, but for the single user, the investment has been well worth the pay-off - even if I decide to go with a vendor in future, I’ll understand what I’m paying for.

                                                          1. 8

                                                            I really dislike pairing, it takes away all the fun in programming and replaces it with a theatrical show.

                                                            Programming for me is 95% thinking (when writing and debugging) and 5% typing. My train of thought is orders of magnitude faster than my speech, so when pairing I get bored after 30 minutes and my productivity falls down.

                                                            1. 3

                                                              The ordinary domain owners amongst us would probably like a registrar that used real 2FA (i.e. no SMS tokens masquerading as 2FA) and had a phone tree that couldn’t be trivially socially engineered.

                                                              Anyone have any recommendations?

                                                              1. 3

                                                                I use NameSilo. They have 2FA (with TOTP) and an additional Domain Defender option that notifies you of changes.

                                                                1. 1

                                                                  NameSilo certainly makes it very difficult to transfer a domain out to another registrar.

                                                                  If you use their domain privacy service you’ll never get the emails of the other registrar, they only allow you to cancel the outgoing transfer and not to expedite it, and finally once you pass all the hurdles the actual transfer takes 7-8 days instead of the standard 5.

                                                                  I was with them for all of my domains, but after that recent transfer experience I’ll move everything elsewhere, regardless of how painful they try to make it.

                                                                2. 3

                                                                  Hover has TOTP.

                                                                  1. 1

                                                                    I use Hover. I switched to them about a year or so ago, specifically because they had TOPT/2FA, and my previous registrar did not. Hover is pretty ok so far.

                                                                  2. 1

                                                                    I’ve been using NameCheap for a while for all of my domain names, and they just started using a custom phone app to do their 2FA instead of only SMS. Never tried to check the security of their phone tree though.