Threads for jamesog

    1. 6

      My previous provider was… me :-)

      I self-hosted for around 20 years, initially from a server at home off my DSL line when I first upgraded from dial-up, eventually moving to a colocated server, then I got bored of maintaining hardware (it was also my day job) and moved between VPSes several times. Eventually I got bored of trying to keep up with spam filtering - and for a while outsourced inbound MX to a service run by friends who did a much better job at spam filtering than me - but in the end I realised running my own mail server wasn’t worth my time any more. Given how long I ran it myself, it’s pretty good that I only really messed things up 2 or 3 times, but even so few occurrences made me doubt ever wanting to touch the setup. Even a “straightforward” Exim upgrade once silently broke something that I wasn’t made aware of for several days.

      Now I’m on Fastmail and couldn’t be happier. The service is fantastic and the webmail is great. Best webmail I’ve ever used - and so much better than Gmail. I consider the cost of Fastmail offset by no longer running a VPS and my own time of maintaining it.

      I’d like to know if you have a custom domain (especially one with a nonstandard TLD like .bike, .cc, .ke, etc.) and whether that complicates things for you (mostly with regard to getting flagged as spam).

      I have many! My “main” domains are “standard” - .co.uk, .com, .net, but I also have some in .im, .li, .ninja, .dev. No issues there.

      1. 2

        I feel about the same as James here (and indeed use FreeBSD now based on his recommendation), but I was running mail for a group of people, if a steadily shrinking set. [1] So I felt like I’d best keep going for my users.

        I was able to get the server management under control over time. Key things were:

        • automating the server build env (KVM + a salt bootstrap, oddly the bootstrap especially was just much easier with FreeBSD than Debian preseed)
        • use OpenSMTPD + the Rspamd filter (the naive bayes classifier is fine, though I had to disable some negative scoring for DKIM, etc)
        • writing tests for the server build

        It’s been OK and kinda fun, especially once Let’s Encrypt made TLS certs affordable and low stress. Deliverabilty is weirdly OK, too, best I can tell.

        But, yeah, if you’re doing just your own mail, I dunno. Best do what sounds fun.

        [1] Artifex.org started on an ISDN line in 1996, moved to Hurricane Electric with the California Community Colocation Project, moved to an admin’s fiber line for many years outside Sacramento, and then went virtual (and from Debian to FreeBSD) with the CCCP’s successor, IO Cooperative. Oddly enough, after a move I helped out on in early 2021, it’s now back at Hurricane Electric, though in their newer facility.

    2. 3

      SREcon EMEA. SREcon is one of my favourite conferences. Always lots of great content and I always enjoy a trip to Dublin.

      If I get to go this year it will be my first time going to a conference since Covid which could be… interesting.

    3. 4

      Am I just lucky or is the M1 significantly less flaky than the M2? My M1 Pro has been happily buzzing along since I got it last year, I don’t think I’ve seen more than a couple apps crash. If anything, it’s been more stable than the Intel Mac it replaced.

      1. 3

        I have an M1 Pro MacBook Pro and an M2 Pro Mac mini. Both are great.

        I’ve had none of the flakiness described in the article, in hardware or software.

        If anything Ventura on my Apple silicon machines is more stable than on my (work-provided) Intel-based MacBook Pro, which has had several weird crashes. I’m more inclined to put any issues down to the OS than the hardware.

      2. 1

        This has been my experience as well. I have so few problems with the M1 that I can’t come up with any off the top of my head. The machine has been chugging along happily for 2 years. This is what prompted me to to seriously consider the M2.

        1. 1

          o_O

          Doubleplus ungood.

        2. 1

          I’m curious why you thought an M2 was much of an upgrade from an M1 in the first place. It’s well documented in many reviews that it’s only an incremental improvement over M1, and so only more likely to be worth it if coming from an older machine.

      3. 1

        Wow. That is not good to hear.

        Ah well, maybe it’s another justification for a 2nd hand M1 when they get cheap enough…

      4. 1

        I’ve been using a maxed out M1 Max MBP since it was released. Very few problems.

        But Apple does do lots of arbitrarily stupid stuff with their OSs and apps. I’m not here to defend the stupidity. I can totally understand someone getting frustrated with macOS.

        For software development, though, it has been a great machine for me.

    4. 27

      Tailscale: Securing access to my various servers and getting access to devices at home when away was always such a pain and slightly complicated in the past. Now it’s dead easy and I don’t have to think about it. It just works.

      Shottr: I didn’t think I needed a screenshot tool before I came across Shottr, and now I use it all the time. It’s great for being able to preview screenshots, add markup, etc. It’s a really great little tool.

      1. 4

        Some alternative i love and that do the same things:

        zerotier for securing my home network. Equivalent to tailscale, zero-conf, just works. It makes it very easy to secure access to a local instance of home assistance.

        flameshot for screenshots

        oh and since it’s not mentioned yet, for mobile: binary eye for anything qr code and barcode scanning

    5. 8

      I recently looked into it for the same reasons, and if you’re looking for a really no-frills registrar with a world-class security team (which was my priority) it looks like AWS Route53 Domains is really the only choice. The other option would be Google Domains, but I don’t trust their account closure policies.

      The list of TLDs they support has grown significantly, and they have everything I own but Google’s .dev.

      The UI is not particularly friendly, being the AWS console, but I use dnscontrol anyway.

      My bias here is that I generally trust a large business with serious customers that have confidentiality demands like AWS more than small businesses that market themselves as “privacy first” (with few exceptions).

      1. 1

        Interestingly, it looks like AWS isn’t actually a registrar and themselves use Gandi (amongst others) for domain registration: https://aws.amazon.com/route53/domain-registration-agreement/ §2.1.

    6. 12

      Not just managers, but companies in general.

      My previous employer basically had no path. All engineers are ICs (some are tech leads or team leads), but there’s no levelling there, so you can either move sideways (to another team or department) or hope a tech lead role opens up, or move to management. I was the tech lead for my team in my time zone, so there was no real opportunity for others in my team to level up in any official way. This frustrated a few of my team members quite a bit.

      My current employer has a great career path for ICs which is well supported at a company level and managers are strongly encouraged to work with their reports to make sure they’re getting what they want from their career and promotion ambitions.

      Now I’m at the Staff level (and as a tech lead) I’ve started reading The Staff Engineer’s Path. It’s a great read so far and I can really relate to everything it talks about.

      1. 4

        I’m quite skeptical of a book that starts the blurb with:

        For years, companies have rewarded their most effective engineers with management positions

        The first company that I’m aware of that stopped doing this was HP in the ‘80s. They had explicit management vs engineering paths and normalised the idea that an engineer might be at a higher level than the manager that they reported to (managing a small team may not require a senior manager, but the work that they’re doing may require some of the most senior engineers in the company). This has been something covered in a lot of different management theory books that I’ve read and seems to be universally regarded as a good idea (I’ve never read anything defending the management-is-the-only-promotion-path idea, the closest is to recognise that it’s good for managers of engineers to be able to understand explanations that engineers give and so they should have some relevant technical background).

        At Microsoft, I think the two-track approach was introduced in the ’90s, possibly earlier. The IC track ends with Partner, Distinguished Engineer, and Technical Fellow. A TF is the same level as a Corporate Vice President and is invited to the senior leadership team retreats and things but does not (normally) have any people-management responsibilities. As I understand it, the only level above CVP is Executive Vice President (and CEO) and those are somewhat intrinsically management roles.

        1. 1

          It definitely depends on the company, and the size of the company. All the companies you name are behemoths, and I’m not surprised they all have a good two-track system.

          My previous employer was a startup that grew quickly and took a very long time to even have formal managers. 10 years later they still don’t really have any real track for ICs, as I mentioned, and from what I’ve seen this is relatively common amongst tech companies that are still relatively immature and don’t take an early decision to create a good career path.

          I understand your skepticism, but as usual it’s all about context and experience. My current role is the first job I’ve had in my 20 years of working that actually has a good career path.

          1. 2

            I understand your skepticism, but as usual it’s all about context and experience. My current role is the first job I’ve had in my 20 years of working that actually has a good career path.

            I’m more skeptical of the book than the idea but my experience is somewhat tarnished by having recently had to do a load of manager training that got very excited about ideas that I’d read in books published in the ’70s and ’80s as if they were previously unheard of.

            It’s hard for very small companies to have this kind of separation, but it’s something that you can introduce as soon as you have a management track and it’s something that pretty much everything that I’ve ever read about managing engineers tells you to do. If a company doesn’t do it then I have to wonder what they’re doing in terms of expectations for managers.

        2. 1

          I was at HPE within the past decade. My first manager was an individual contributor who was promoted into management so that the team did not collapse, and incidentally they were also the person who built the product initially. My second manager was an external hire who did not know how to code. It looks to me like the book has a reasonable grasp of the situation.

        3. 1

          the idea that an engineer might be at a higher level than the manager that they reported to

          What does “level” mean in practise, at this point? Is it just about compensation, parking spaces, corner offices and generous travel allowances? Or is there some sort of implicit authority that comes with a higher level? (And what then does it mean to manage someone at a higher level?)

          1. 4

            What does “level” mean in practise, at this point? Is it just about compensation, parking spaces, corner offices and generous travel allowances? Or is there some sort of implicit authority that comes with a higher level? (And what then does it mean to manage someone at a higher level?)

            Seniority along their respective tracks. This normally comes with increased compensation and so on. Authority is contextual, but someone at a higher level on the management track would be expected to be able to manager higher-stakes and larger teams than a more junior person. Similarly, someone at a higher level on the engineering track would be expected to be able to solve more difficult engineering problems (for example, design the architecture for larger systems, implement things with more constraints on resources, and so on).

            Managing a senior engineer does not intrinsically require a more senior manager (often the reverse, since a more senior engineer may be more able to articulate their requirements and more able to accurately predict schedules) and that was something that their model reflected.

            1. 1

              I see. It sounds to me like the contextuality of the levels makes it hard to compare an engineering level to a management level, aside from compensation and similar.

    7. 23

      Maintenance cost of bash scripts is lower

      I would strongly disagree with this. Unless the script is very simple - only single commands, few variables - shell scripts can have many pitfalls that many people don’t realise until things go wrong. Pipelines and subshells and their side effects are not widely understood, for example.

      c.f. https://lobste.rs/s/5ldmtp/pipefail_how_missing_shell_option_slowed -> https://lobste.rs/s/wl6kem/simple_script_is_still_someone_s_bad_day

      Don’t get me wrong, shell scripts do have their place, but for anything even vaguely complex they’re generally the wrong choice. Use the right tool for the job.

      Oh and lastly:

      Every machine has bash installed

      As a BSD user, this is not true. :-)

      1. 4

        Unless the script is very simple

        You can do a lot of useful stuff with simple bash scripts. I probably write one per week. Lets look at an example:

        https://github.com/no-gravity/git-retroamend

        4 lines of code. Saves me a lot of hassle every time I need it.

        Pipelines and subshells and their side effects are not widely understood

        They can even be indeterministic:

        https://www.gibney.org/the_output_of_linux_pipes_can_be_indeter

        1. 2

          You can do a lot of useful stuff with simple bash scripts.

          I’m not disputing that. I’ve been writing shell scripts for a very long time, and I use them where they’re appropriate. What I am disputing is your statement that the maintenance cost of shell scripts is lower compared to other languages. If you’re only ever writing simple scripts this is often true; but if you’re comparing shell to other languages there was probably a need to use the other language in the cases they were used.

          They can even be indeterministic:

          Indeed. You’re kind of making my argument for me. :-)

          1. 5

            With “maintenance cost” I do not mean the cost to change the functionality. I mean that from time to time you have to change your script because the language changes. I expect Bash to have less of these breaking changes than most other languages.

            1. 3

              from time to time you have to change your script because the language changes

              What? I mean, there was python 2 to 3 and Ruby 1.8 to 1.9, but I don’t think breaking language changes are common?

              1. 5

                Python in my opinion is an example for high maintenance cost. They sometimes do backwards-incompatible changes within a major version. For example Python 3.7 introduced “async” as a reserved keyword, breaking code that used “async” as a variable name.

                If you follow each version update of Python, you will probably recognize all breaking changes as deprecations in earlier versions. But if I would just write a script, leave it alone for 10 years and then try to run it with the latest Python version, I would not bet on it running without errors. Whereas for bash scripts I would assume they still work.

                But for bash scripts it totally depends on which programs you call.

                1. 1

                  The needless breaking changes are my least favorite parts of Python and Node.

        2. 2

          You can put this directly in your .gitconfig. I have the following in mine:

            # "commit fix <commit>" - Add index files to given commit without changing
            # its message.  For more control, create a temp commit, then "rebase -i" and
            # use fixup and edit (which lets you change commits msgs)
            cfix =  "!f() { \
              local committofix=${1:-HEAD}; \
              git commit --fixup="$committofix"; \
              git rebase -i --autosquash "$committofix"^; \
            }; f"
          
      2. 0

        Also NixOS.

    8. 10

      I do something similar, but in a more generic way. I have two “search engines” defined in Chrome, one called Jira and the other called JQL.

      Jira is defined as https://company.atlassian.net/secure/QuickSearch.jspa?searchString=%s JQL is defined as https://company.atlassian.net/issues/?jql=%s

      The Jira one means I can do jira<space>project foo which activates Jira’s “smart” search and, if project is a valid Jira project, searches for foo in that project, or otherwise does a more generic text search for project foo; and jira<space>project-1234 also takes you right to ticket PROJECT-1234 as in this blog post.

      For those that use Alfred on macOS you can also create a similar search there, using {query} instead of %s

      1. 4

        Yes, Alfred is what I use and it’s extremely useful for this. My shortcut is jira and putting in a ticket number will go straight to it. I also created a workflow to take the full URL of a Jira ticket from my clipboard and replace it with just the ticket. All this saves a great deal of tedium.

        If you’re on macOS I can’t recommend it enough. https://www.alfredapp.com

      2. 2

        Yes the QuickSearch one is great, and that’s what I have j aliased to in Firefox. You can type in a ticket ID or a search term or whatever.

        One really frustrating thing is that Jira then redirects you to a combination search/ticket view and puts the first result’s ID in the URL bar. This makes it that much harder for people to reverse-engineer the search URL!

      3. 1

        Mine are very similar, I have ji for ‘JIRA issue’ and js for ‘JIRA search’.

    9. 1

      I think I’m missing something. Is English your native language, or Portuguese? Some things in the post aren’t connecting up for me.

      1. 1

        Portuguese is my first language.

        1. 3

          Sorry, I guess I’m still confused then. For you to work in your native language requires empathy… on your part? Someone else’s?

          The best I could figure is that you were looking to ask people to be empathetic to people who are speaking their non-native language, but that’s not what the title said, so I’m a bit lost.

          1. 1

            The article is directed towards native-English speakers, so My native language == your non-native. I understand the confusion, tho.

            Regarding empathy, I think both sides have to be empathic. Working in a second language can be exhausting, and people often feel like they’re less valued, less intelligent or treated differently for speaking “bad English”, so I’m making a case to say that you (non-native speaker) are not alone, and it’s okay to make mistakes. For native speakers, on the other hand, I ask for empathy because you’re an expert on your language, so your very presence can be intimidating to others. Using simpler language, for example, is a way to help foreigners.

            1. 4

              I’m all for using simpler language, not just for helping people who are less fluent in a language, but also to reduce the cognitive overhead required to understand what is being said. I’m not a native English speaker either (I’m Dutch), but oftentimes I get annoyed by the horrible ways people use English (even native speakers).

              For me, I approach it a bit like a programming language. It’s always good to have empathy towards your fellow developer (they might be tired, overworked or just new to the language), and if people are continually making mistakes it doesn’t make sense to chew them out, because it’s clear they’re at a language level below full mastery. But I still kind of expect a professional who has been in this business for several years to have mastered both English and the programming language at hand.

              1. 4

                I’m all for using simpler language, not just for helping people who are less fluent in a language, but also to reduce the cognitive overhead required to understand what is being said

                Simpler doesn’t always help. A lot of the complicated words in English are lifted directly from French or Italian and so might be more familiar to a native speaker of another romance language than the more common English terms (I recall a French friend had a habit of just putting French words in his PhD thesis and anglicising the word ending. It worked 90% of the time and was very confusing the few times when it didn’t).

                I once tried playing Balderdash with a group of Italian friends and it was no fun at all. You’re presented with an obscure English word and you’re all supposed to make up plausible definitions. You get one point for every person who believes that your definition is the right one. If you write the correct definition then yours is excluded from the ones people guess and you just get two points. Every round, all of them would write the correct definition because the obscure English words were everyday Italian ones.

                Even for native speakers, there are a lot of weird dialect variations, especially where one has dropped a word from a common phrase. For example, American dropped the ‘to’ from phrases such as ‘one through to five’ and just says ‘one through five’, which sounds to an English speaker like a sequence that starts at one, continues through five, and then probably stops somewhere later, but to an American means something that stops at five. In contrast, after people in England stoped wearing pantaloons, they stopped using the abbreviation ‘pants’ and so dropped the ‘under’ prefix from ‘underpants’. Americans kept the word ‘pants’ as a synonym for ‘trousers’ and so cause hilarity to English speakers (we’re easily entertained) whenever they refer to their pants. Pants is definitely a simpler word than ‘trousers’, but saying it increases the cognitive load for folks outside of the US, who first have to (by law in the UK) smirk, and then mentally translate.

              2. 3

                My experience as an English person is that native English speakers are often worse at speaking our own language than non-native speakers. Working with non-native speakers for my entire career has actually helped me realise where native speakers get lazy in their use of language, or use idioms that don’t translate well (or at all) - sometimes even across different variations of “native English” such as English vs Scottish vs American.

                1. 4

                  I’m very used to Scandinavians going “sorry, I’m not a native speaker” and delivering perfect English. (Well, it’s perfect until it isn’t - sometimes you can get very interesting mispronunciations.

            2. 2

              The article is directed towards native-English speakers, so My native language == your non-native. I understand the confusion, tho.

              Ahhh, I think I see now. Thanks.

    10. 4

      IRC, dict pop3 and whois. Whois is terrible, we really need a structured alternative. But there it is.

      1. 3

        RDAP is the proposed replacement for WHOIS.

        1. 2

          Yes, which not all TLDs support

    11. 2

      Despite everything, 2021 was reasonably good for me.

      • January: Approached by a recruiter at a great company, started interview process.
      • February: Got an offer from said company, glad to finally resign from a company I’d been resenting for too long.
      • March: Finished at old job, took a few weeks off.
      • April: Start at new job, mind very much blown.
      • May-November: Thoroughly enjoying new company and job, but realise towards the end of the period I’ve taken no time off during the year so took 2 weeks off.
      • December: Towards end of my 2 weeks’ leave I finally got myself an e-bike to try and get out more and get more exercise. Immediately have an accident, seriously fractured shoulder joint and had to have surgery.
      • Now: still in a sling :-(

      As for next year, well I’ll still be recovering from the accident for most of January and will need physio for a while. So in my first year of a new job I’ll have had to take 2 months off! However the company and team are great and I look forward to keep growing at this company.

      1. 2

        It is a good year! I am also starting to prep for interviewing as I have stayed too long with my current employer. I failed to get a promo to staff level earlier in the year despite working my ass off. Felt depressed and burnt out after that debacle. But, in rear sight, I learned that I need to look ahead beyond this company as in “I have a business that is providing my skills for whoever could pay me decent”.

        So leetcode and system design I go!

      2. 1

        Sorry to hear about your accident. I’ve been wanting to get an ebike as well.. which one did you get, do you recommend it?

        1. 2

          It’s made by a UK chain called Halfords. It’s a Carrera Impel im-3. From the brief amount of time I was able to use it, it’s nice. Comfortable to ride, the electric assist is great.

          1. 1

            awesome. I’ll check it out

    12. 2

      The author pronounces it [aɡe̞], like the Italian “aghe”.

      Does the author mean aghi? Aghe is not an Itailan word.

      Now I am confused. Is the pronunciation as ah-gee (as you would say aghi in Italian) or ah-geh (as you would pronounce aghe in Italian if that were a word)?

      1. 3

        It seems unlikely to me that the majority of people who encounter this library at its point of use will think to investigate how it’s pronounced. I expect most will assume it’s the English word. Naming things is hard; there are many pitfalls.

      2. 3

        I’m also confused. It links to google translate which translates it to “needles”, but I’ve never heard the word pluralized like that. I’m guessing it comes from FiloSottile’s dialect.

        1. 1

          I think I just got it. The link to google translate is there so you can play the pronunciation and not to translate it to English. I guess that’s helpful for everyone that is not Italian, lol.

      3. 2

        The latter. I’m sure he used to describe it as pronounced the Japanese way, but perhaps even fewer people understand that :-)

        1. 1

          I also thought was pronounced like in chicken karaage. However, I now suspect I pronounce that wrong also since I say “ah-hey” rather than “ah-geh”

          1. 3

            Heh yeah, for the record you’re pronouncing it wrong - a mora consisting of a g followed by any vowel is always a hard G sound in Japanese.

            So it’s kah-rah-ah-geh (more or less, in a standardish American accent, although with no aspiration because those Hs are just there to steer you towards the right vowel sound, and with the vowel sounds held for a somewhat shorter period of time than you might default to)

    13. 4

      I had no idea about Lunar but it sounds like just what I need! Thanks for this really great write-up on an interesting problem and for Lunar :-)

      1. 2

        Glad you enjoyed it! If you experience any issues with Lunar, feel free to reach me and Lunar’s community on our Discord channel listed on the lunar.fyi website.

    14. 1

      Just say no to cleaning up git history people! You don’t look at it enough to pay off.

      Unfortunately, I don’t have data to back up this claim. :/

      1. 13

        I don’t have the data either, but I do have the experience. Bad commit messages and dirty history are the bane of my life. The advice in this post is excellent.

      2. 5

        I look at it easily more than 10 times a day. I do think you are on the right track as to why a large population of developers don’t take the time to write useful commit messages. They treat think of it as a write only medium. If I used the git command line or the GitHub web UI to navigate history I wouldn’t check the VC history so often.

      3. 4

        If people looked more often, perhaps they would care more about their commits.

        I have a gutter with commit messages for each line/chunk in my editor for much of the day since it gives me some context about why a line/function looks as it does.

        1. 1

          Right, I am totally for writing good commit messages! They should contain a description of the changes and a link to the ticket. That way you get requirement + architecture idea. But I never look at the graph structure.

      4. 3

        I use our git history constantly (we have a clean, well-organized one). I work at a Very Large Enterprise, too. Could your experience be related to not working within a space where the history is clean enough to be reliably usable, rather than it being worthless?

    15. 2

      There’s one small omission in this, particularly relating to footnote 1, that you don’t have to use pq.Array. As long as you don’t mind using a non-standard type, lib/pq has types like [StringArray[(https://pkg.go.dev/github.com/lib/pq#StringArray) which handles reading a postgres text[] column into a (wrapped) []string - so you wouldn’t necessarily need to use pq.Array when using pgx.

      Similarly, there’s a pgtype package which can be used with either pgx or pq in database/sql mode to provide similar types. pgtype has many more types available.

    16. 3

      Do people use fzf at the CLI?

      Not by typing fzf but I started using its zsh bindings so it’s acting as my ctrl-R (history search) and I’ve been getting into the habit of using its alt-C instead of typing cd. The ctrl-T behaviour can be useful, but zsh’s menu completion is usually good enough that I don’t need it. Overall I love the fzf shell bindings.

      Other fun tools I use are fd, the find replacement, and k9s for interacting with Kubernetes clusters.

      1. 2

        Do people use fzf at the CLI?

        Not by typing fzf

        I actually do that with fzy, because I am lazy to study the fzf bindings for Bash (and find out whether they work for Korn shell, then create mine), but I am also tired of the imprecise algorithm of fzf, and rather use that of fzy.

        But, obviously, I am the exception, not the rule.

    17. 2

      I was cheering until unsafe was used. Come on, don’t play stupid games to avoid one allocation.

      1. 1

        Yeah, this was the one part of the post I didn’t agree with.

        I was also curious there was no mention of strings.Builder.

    18. 2

      I used mutt exclusively for more than 15 years. Last year I migrated my mail from self-hosted over to Fastmail and I found their web interface so good it’s all I use now.

      (That, and I somewhat rely on the “Report Spam” and “Report Phishing” buttons in the UI, which, although can be sort of replicated in an IMAP client with filter rules, it’s less convenient.)

      1. 2

        Same, 90%[1] of the time: you can use a keyboard combination to hard wrap plain text, I pipe raw messages into hg or git to get patches.

        Decent standard configuration and also good settings if you go deeper. Also, it’s ridiculously fast, I’ll never use imap with a desktop client and I’m the kind of person that doesn’t like to use web apps.

        [1]: The last 10% are occasional email that I send from a terminal when it’s easier to do so. I never read from there.

    19. 17

      One note with the “Passing request to Consul services” section: when you use a DNS name in proxy_pass, that name is only ever resolved at startup. If the IP address the domain name points to could ever change you should define an upstream. Nginx only does normal DNS TTL / refresh in upstreams.

      1. 6

        This is rarely mentioned and you learn it the hard way. @alexdzyoba I think it’s a good thing to add to your great article :)

      2. 4

        I have previously documented a workaround for this (although, really the open source version should just support this): https://tenzer.dk/nginx-with-dynamic-upstreams/.

        1. 1

          Ah yes, I had seen the variable hack once before! Both the variable hack and upstreams need the resolver set, so both will do the trick.

      3. 2

        Thanks, everyone! I’ve updated my post. Put the link to this thread and the post by Tenzer.

      4. 2

        Thanks for pointing out! But what about “valid” option in resolver directive? It should control the TTL for DNS cache.

        1. 2

          It does, but proxy_pass doesn’t use DNS TTL, it only ever resolves once, unless you use a variable or an upstream. The Nginx docs aren’t very clear, but the resolver setting says it’s used for upstreams.

    20. 8

      One fun incompatibility I ran in to a while ago is that GNU libc will try the first namserver first, and then move on to the second if the first doesn’t work. Musl libc however will just always pick a random nameserver from the list.

      So if you do something like:

      nameserver 127.0.0.53    # Nameserver with some special resolving rules.
      nameserver 8.8.8.8       # Fallback in case the above doesn't work.
      

      You’re going to run in to trouble on musl-based systems.

      1. 5

        One fun incompatibility I ran in to a while ago is that GNU libc will try the first namserver first, and then move on to the second if the first doesn’t work. Musl libc however will just always pick a random nameserver from the list.

        I was curious what musl’s documented behavior is, so I went searching, evidently it issues queries to all your DNS servers in parallel and then uses the first response it gets back in an effort to reduce latency for lookups.

        1. 1

          Ah yes, you’re right; I remember it now. I actually ran in to this about 5 years ago and misremembered the details. I ran in to this when I was working on my DNS-based proxy/filter program, which worked great on my OpenBSD laptop that I initially developed it on while I was away for a month, but had weird intermittent problems on my Void Linux musl desktop system after I got home. It took me quite a while to track it down to this 😅

          1. 1

            I can imagine, it definitely wouldn’t be the behavior I would expect out of the box if you asked me to predict how musl’s lookups would work.

      2. 2

        Which distributions use musl? The behavior you describe for musl is what I remember in whatever we were using in production circa 2003, either suse or redhat.

        1. 5

          Alpine Linux for one. Alpine is commonly used with Docker setups too (because the images are smaller it means you can pack more images per gigabyte of production storage); not to mention Alpine recently added Tailscale as a package so we at Tailscale really have to make it Just Work™️.

      3. 1

        glibc (and most other libcs I’ve encountered) also lets you do this with options rotate (among other things). Granted that’s not default behaviour in most places, but I’ve used that in the past where I wanted some internal systems to do more of a round-robin between internal nameservers.