Lobsters was down ~4 hours, until a few minutes ago, because our domain name expired. It’s now renewed. I woke up to the outage so I’m still a bit discombobulated, but here’s quick notes explaining the situation.
Our registrar had previously comped domain renewals and emails about this year’s renewal looked like it was proceeding as normal, but apparently that didn’t happen. I put our backup domain lobsters.dev up. (I registered lobsters.dev years ago because I was nervous about a country code domain with the only registrar I could find in a foreign country whose language I don’t speak.)
So I pulled up the customer control panel and could view an invoice, but the payment system was down and I was unable to pay the outstanding invoice. I emailed support and my contact there. About 15 minutes later the invoice status changed to ‘Cancelled’ (like previous years) and the domain showed as renewed to 2024. A few minutes later as DNS propagated, the domain started resolving.
I haven’t yet heard back from the registrar and will post comments below when I do. I’d appreciate any suggestions on how to better handle this potential failure mode.
Happy lobstering,
- Peter
For what it’s worth, I’m a citizen of the country in charge of the .rs domain and I speak the language (obviously) and I’m volunteering to be in charge of registrar communications/translation in the future if necessary.
Also, I know of a few more local registrars that popped up in the meantime who are now accepting foreigners as clients and can offer additional guidance/assistance in domain transfer if that’s what you decide to do (foreign registrars are charging exorbitant prices for .rs domains for whatever reason).
I don’t post here often but I do enjoy all the conversations going on and I’m more than willing to help keep the site up and running. Thanks for all the hard work you put into this!
Thanks, I will email you later today so I can better understand the lay of the land.
Lobster man, take me by the hand
Lead me to the land that you understand.
If I’m reading my timezones correctly, it’s too early in the morning for you to be getting paged for DNS. 😭
Thank you for your service, @pushcx! Sleep well!
This kind of things happen. Do not stress about it: nobody was ever injured for not being able to access lobsters.
It is also good to know there’s a backup domain.
Thank you a lot for all the work. Keep it cool for a few day to recover from the adrenaline rush.
If anything, it probably resulted in a net productivity increase.
It is absolutely insane how fragile the web is. You can’t really own a domain, only rent it, which can fail as it just did today.
It is true that the web is ephemeral, and stuff just go poof. So grateful for the Internet Archive, will donate to it when I get my first job.
There is an inverse problem available with ownership models. There was a video game I loved as a kid and this summer as I debugged my emulator setup I wondered if I could acquire the rights to release it on current platforms (with my own highly-opinionated gameplay tweaks, naturally). Unfortunately I can’t figure out who owns it! The original studio is long-since out of business; the company who released a port 15 years ago is also out of business and anyways I’m not sure if they owned the copyright or were just licensing; the publisher was absorbed into a larger publisher that also was eaten by a bigger fish who also went out of business; etc etc. There is no registry for copyright like there is with DNS, and even if there was, if it didn’t motivate updating with a “rental” model where rights lapse unless someone shows up with regular payment, it would also likely be a dead end.
I’m not particularly endeared to domain renewals this morning, but it still solves a host of problems that are especially acute in a system that picks ‘human readable’ and ‘secure’ (slash ‘usable’) on Zooko’s triangle. I guess it’s not engineering if there aren’t downsides to every solution.
There is, but it’s not mandatory and is quite unpopular in software circles especially
Thanks for the correction, I entirely forgot about the US copyright registry as it’s been optional for so long.
I said it, and I’ll say it again, who cares about “you don’t own a domain, can only rent it” when you can lawfully rent a domain, but the justice system can take it away from you because they feel like you shouldn’t have it. The same would happen if you owned it.
Ownership only ever exists to the extent the legal authority enforces your ownership.
In the U.S., people regularly have their property seized without even being accused of a crime.
So, does the Internet Archive own its domainname? Can you spot the problem?
No, and yes, I agree this is a huge problem. And I do not think there’s a solution unfortunately.
DNS is just one naming system. There have been other ideas in this space, blockchain-backed systems, the GNU Name System, etc.
What would a solution need to look like? What kind of properties does it need?
In reality things are named for other things that have been there for ages.
I am reachable via EU/Germany/Freising/somecounty/somestreet/somehousenumber I can not change this.
This is good, no other person can have EU/Germany/Freising/somecounty/somestreet/somehousenumber just like that.
This is also bad, because if somecounty is Hintersterbensöd (=deep rural county) somestreet is Am Galgen (=at the gallows) and somehousenumber is 1 (=probably the only one) - no one will want to visit it. And I cannot have Goldstadt, Millionärsallee 1 even if I was millionaire.
So maybe what is needed is a double adressing system. IPs would be allocated to EU/Germany/Freising/somecounty/somestreet/somehousenumber and my nicename.tld would be a pointer to it.
Even if I fail to defend my nicename.tld, I would still be reachable over EU/Germany/Freising/somecounty/somestreet/somehousenumber
You need free (as in possibility) naming, but it has to cost monthly (or else someone will register all words in dictionaries), first come, first served, and regular tests if the name is used at all.
I propose reading up on islamic water jurisdiction. they have yearly new negotiations for water rights and unused rights are freed by the process. It is a mapping between the available water and the available consumers (NOT the hoarders and speculants).
Not sure physical addresses are the best example: Countries exit unions, country names change or get invaded, residents are forced out during occupations, street names change, people get evicted, landlords price out residents, eminent domain is a thing, etc.
To top it off I’ve voluntarily moved enough times to see enough failure modes of mail forwarding that I wouldn’t trust it with anything serious. :)
Do you have a fav summary link I can read?
Sorry, I have no link any more, I read about it in the winter of 2008/2009, when we had reduced working hours due to the wake of the subprime crisis.
I was reading up about irrigation techniques and soil moisture sensors, and that was a side trail about different water rights.
I got an email from my contact at the registrar ~35 minutes ago. He apologized for the outage and explained that domains they comp go through different paths in the system, which is why it didn’t correctly renew automatically this year. He has volunteered to renew the domain until 2028-12-13, which I see reflected in whois.
I’m considering picking different failure modes, I’m going to research the available options later today.
Also, I should improve our outage monitoring, which is currently mostly me happening to see the IRC channel start to blow up, or one of the other mods calling me. This is not great for an outage that happens when I’m asleep. For defense in depth, does anyone use a free/reasonably-priced outage monitoring service they’d recommend for this use case of “if https://lobste.rs doesn’t return 200, call pushcx until he answers”. I’ll research later today, but maybe someone already uses one? (I haven’t hacked together a cron job on a home computer because I’d get regular false alarms about the internet service being down, and the data center VPSs I have access to are either on the same account as Lobsters hosting, not online 24/7, or a consulting client’s.)
And as a backup to a monitoring service, are there any regular commenters who live in European or east-Asian timezones who’d be willing to save my cell phone number and hassle me awake if they see the site offline? Please email or DM if so. I’d also need your phone number to put in my contacts so I can permit it to ring through my phone’s nightly do-not-disturb setting.
Your sanity is worth a lot more than our silly little comments! If the site went down again… we’d all be upset but we’d survive.
I appreciate the sentiment, but I would sleep better knowing the site had better alerting in place. Lobsters is important to me.
I have three weeks of PTO coming up without much to do, and I’d be happy to spend a bit of that time to set up some proper monitoring. I don’t think it would cost much, and it sounds fun.
Absolutely not. This site is amazing and I’d miss it if it went away, but it’s not critical infrastructure. If it’s down for a day, that’s a shame. If maintaining it costs you sleep, it makes it more likely that you’ll decide it isn’t worth the hassle and take it down permanently and that would be a tragedy.
You previously likened this to a barbecue in your back yard that all your friends can drop into and chat. A barbecue in your yard doesn’t need an SLA. If the charcoal won’t light and the food is an hour late, we’ll wait.
I agree with this and it’s very much worth saying. We don’t have to cosplay some 5 nines availability or something. It’s fine.
Pulsetic and BetterUptime have free tiers. Haven’t used BetterUptime in a long time, never used pulsetic.
Uptime Robot. Think the free plan is limited to five minute checks, but that’s still more often than “when USA wakes up”
I use https://updown.io since:
It has a variety of alert methods, I haven’t looked into whether it can repeatedly make phone calls. Looks like it can natively support SMS, and could otherwise integrate with Twilio for calls?
I raise my vote for updown: we use it professionnally and the price is minuscule for the support I got: we had a strange routing issue using BGP and it showed some strange up/down patterns.
I contacted the dev behind updown.io and he gave me a VERY thorough view of his side, which helped a lot. The dev is french (I’m french too but not affiliated) and was very responsive.
Other than that the UI is very clean, the site is fast and gives back very useful information. Everything is queryable via an API, which is a great plus.
Frankly, I’m fine with not all websites having five nines uptime or whatever. Lobsters is great, but it’s probably not going to ruin anyone else’s day if it’s down for a bit!
(Remember when major websites routinely had scheduled nightly downtime? It was fine.)
HetrixTools has a free plan with uptime, domain and SSL monitoring. I’ve been using them for years with very good results.
Some registries will display the expiration date in WHOIS. The .rs registry does that, so it would be technically possible to have monitoring and alerting set in place. To avoid false positives, one would have to know how many days in advance the registrar submits the renewal in normal cases. Alerts should only fire in the period after a normal renewal and the actual expiration, so that the alarm is trusted to be critical and not a false alarm.
You can use this to snipe the domain as well ;-)
It’s a double edged sword.
If I understand correctly though, there is typically a grace period when your registration expires. Here is what ICANN says regarding the “redemptionPeriod” EPP status code:
Yep, though the details can and do differ for country code TLDs, because they are not subject to ICANN’s contractual requirements. If you are considering or already using a domain in a ccTLD, it’s worth taking a bit of time to double check the fine print.
Yep. That is the reason not every registry has the expiration date in WHOIS. The .no registry is an example.
WHOIS’s date format is not standardized.
You should prefer RDAP over WHOIS.
For example:
curl -s 'https://www.registry.google/rdap/domain/lobsters.dev' | jq .eventsforlobsters.dev. Sadly, it looks like.rsdoes not provide an RDAP endpoint: https://data.iana.org/rdap/dns.jsonI think most registries display details in WHOIS. It’s the best method I’d found for keeping an eye on domains, without signaling interest.
I use the domain-check-2 script, which uses whois behind the scenes. I’ve got a system for monitoring a few domains, mine and others. Before that, I used calendar reminders, but that got complicated with grace periods. It’s easier to build the habit to check regularly with some other things I do. (I’ve added lobste.rs to the list, now.)
I didn’t even know I have such an affection for lobste.rs as in these 4 hours of uncertainty ;-)
For domains you must keep:
That should be enough. You can also pay people to handle this for you or house your domain at an entity where you have a longer-standing invoice based relationship (such as Route53), but just the above should be fine.
When I last looked ~6 years ago, I couldn’t find a large, ideally American company that registered .rs domains. (In particular, Route 53 did and does not support .rs registrations.) This would include some different failure modes, but the worst-case scenarios seem a little more approachable than me hopping on a flight from Chicago to Belgrade to knock on a door because I can’t get an email response. (Speaking of which, I see I got one a few minutes ago; I’ll post a top-level comment.)
Fun fact: Chicago and Belgrade are sister cities :-)
If it’s hard to find registrars for a ccTLD, that usually means the TLD does not support EPP, the standard protocol that mediates between registrars and registries. I had a brief scout around the
.rswebsite and the first thing I found that went into technical details confirmed my suspicion. https://www.rnids.rs/en/documents/technical-and-administrative-terms-and-conditions-operation-registrars-rs-domainsIf (like I used to be) you are responsible for a large org’s domain registrations then OpenSRS has some very handy documentation about the various TLDs. https://help.opensrs.com/hc/en-us/articles/205403577-Domain-Help-Topics (The spreadsheet linked at the top of that page is magnificent.) But they only list TLDs that they support, which generally excludes ccTLDs that lack EPP, such as
.rs.When considering registering a domain name, what are the min set of requirements you look for in a TLD?
When this was my day job I was generally providing advice to people who had already chosen a name, and they were almost never very adventurous. In most cases the TLDs that are troublesome rule themselves out straight away, e.g. strict eligibility rules, high fees, lack of EPP support (which meant our registrar of choice did not support them).
My prejudice is that ccTLDs who support EPP have sufficiently good technical ops that the contractual and regulatory side is unlikely to be a disaster: EPP is fairly tightly coupled to the business of being a registry, so the management needs to be mature enough to cope with that level of automation. But my experience is fairly parochial so I have not had to worry about counterexamples.
For some of those weird TLDs that are not supported by Route53 we still have a GoDaddy account on the side (which oddly enough does not register .rs domains it seems).
I can find it on Gandi, Transip (at a premium) and Marcaria (which looks weird but seems to be a thing for niche TLDs).
HugOps
It’s always DNS.
Thanks as always for the hard work @pushcx!
DNS renewal slipups happen to absolutely everyone, including multi-billion dollar giants. It’s a problem with the system, not with particular agents within it.
FYI: I have a few
.rsdomains (such as https://diff.rs) and I have registered them with https://inwx.de, a German registrar. They also have an English-language interface. I can recommend them, although if your existing registrar works well (and comps the cost) then it probably does not make sense to switch.Good time to double check my own auto-renewals.
Thanks for comping all of us.
Thanks for your service. We are Ok not reading Lobster for 4 hours.
speak only for yourself :-)
Thank you very much, Peter!
Peter :)
If it’s economically possible, I’d renew for 10 years and call it a day.
Reminds me of a particular private torrent site whose yearly expired SSL cert turned into a community meme. Every year like clockwork the site would be down.
Keep up the good work! 🫡 Lobste.rs is one do the best communities out there; a shining beacon of hope in the wasteland that is the modern Internet
Unsure if this is possible with the registrar you’re using but I tend to renew my important domains for five to ten years. Sure it’s kicking the can down the road some way, but at least you then have a while before it needs kicking again.
Thank you for your service, pushcx!
Funny because I was just debating whether to renew a domain I want to rely on for 1, 2, or 3 years. Part of my calculus for not doing a prepayment for multiple years is that this makes it too easy to forget about the renewal process — at least an annual process is frequent enough to set a reminder. But given how registrars have been operating, I like the idea of locking in annual costs today.
One thing I really wish domain registrars supported was a way to synchronize renewal dates of all your domains. Would make life easier when you have a handful. I guess there must be some ICANN reason or some such.
Auto-renewal is an option with some registrars. I would inquire into the availability of such a service.
There’s nothing worse than getting a domain up and running, growing a user base, and then having the domain snatched from you when it fails to renew – and there are predatory “businesses” out there doing just that.
It sounds like it was set to auto-renew, he even got emails, but then it didn’t auto-renew.
When I worked as the CTO of a startup, we were paranoid about a failed domain auto-renew, because of how much havoc we had seen it cause at other companies. Eventually, we discovered that a nice domain registrar, Hover, has a feature where you can preload a certain amount of money into the account, and it’s used as a credit for auto-renewals. They call this an “Account Balance.”
What’s great about this is that it lets you switch your process to: “Fill the Hover account with $200 on January 2 every year” (or something like this formula:
annual_fee * number_domains + $50), and then, no matter when the domains renew, and no matter what the status is of the associated credit cards, those auto-renewals will go through. And now you only have a single annual “domain account balance replenish date” of your choosing. I wish every registrar supported this smart feature. Described here on Hover’s website.Wouldn’t it make more sense to simply schedule domain renewals so that they always have more than a year until they expire? Payment going from the hover account to renewing the domain seems like an extra fail-able step with little benefit?
Most registrars build their autorenewal process around a “renewal date,” which is usually one or two business days before a domain’s expiry date. ICANN also has a mandated 60-day notification built around the expiry date. If you “add years” to a domain, you push out the renewal and expiry dates by that many years. So, what you describe is only possible if you rely on a manual process outside of registrar auto-renewal, to manually “add a year” to all your domains once a year, or somesuch. Sometimes doing this in bulk for a number of domains isn’t straightforward, to boot. The “account balance” system keeps you within the standard annual autorenewal process but with a guaranteed source of funds for billing, since the #1 cause of a failed renewal is an expired credit card or failed CC payment authorization.
This was the experience, yes. I got the same email sequence as the last five years so I was complacent about seeing them in my inbox.
Glad it worked again. Thanks Peter!