It takes some courage to write a descriptive incident report after the worst happens and I appreciate it - not all companies describe what went wrong in so much detail.
For sure. There are many companies that could describe part of their system as super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented, but the vast majority of them would never admit it.
This gives me more confidence in GitLab, but at the same time another reason to set up something to back my private repos up to BitBucket.
Agreed - I like how GitLab do so much in the open, particularly as things unfold and not just in a post-mortem.
It makes for quite a read, though - seems like too much of their architecture is brown paper and string (not uncommon, I guess). Even the fact that someone could accidentally delete data in an interactive session sounds like something that could’ve been prevented (although I guess database replication is not the easiest thing to manage with a configuration management system). Also, LVM snapshots - I’ve never been a fan of those, particularly in this day of ZFS' almost zero-cost snapshots (snapshots are not at all zero cost with LVM).
Of course, it’s so easy for me to say that as an outsider.
too much of their architecture is brown paper and string (not uncommon, I guess)
Since their main audience is developers, perhaps they figure developers already assume everyone is in this same boat with them, and pretending otherwise wouldn’t benefit them like it would a company whose customers don’t know how software startups work.
Am I the only one who thinks they should spend more time fixing their issues / backups / infrastructure than writing those large post-mortems? It sure takes a lot of time to write a document like this…
You can’t fix broken process without understanding it and writing a post-mortem about it.
Fixing for the sake of fixing things is a quick way to have more things needing to get fixed.
Post-mortems are expected by some customers (even if they’re never seen by those customers, the process is expected). And post-mortems are important for actually discovering and documenting those issues with backups and infrastructure they’re actually vulnerable to. Having a post-mortem also lets more people work on the solution than just the one person who screwed things up and has the problem set in their head.
Basically, post-mortems are good, and they should have written one regardless. Sharing them with the public is also good.
I’m guessing the point is to get a discussion going on social media about their product.
Great is probably a strong word for a language that doesn’t support parallelism (unless you are on on alternative runtime). Maybe guilds will fix this but they wont be ready in the near future.
-I do enjoy and write quite a bit of ruby
I agree. Ruby has missed the train with its close to no concurrency support. Also, its hyper dynamic and flexible nature, which seems like a productivity boost in the beginning, often end up being a maintenance nightmare, especially since software is getting more and more complex every day.
I also write a lot of Ruby for a living, but I am not blind.
9 out 10 web applications simply don’t need the power that Phoenix and its Erlang underpinnings provide,
Then it must be me, but most of the Rails projects I’ve worked with would have hugely benefited from many of the concurrency and resiliency features that Elixir / Erlang provide.
That number, 9 out 10, is based on nothing more than gut feeling.
and in these cases, developing them with Rails will be much faster.
Why do you think that developing with Rails is faster than with Phoenix? Having programmed Ruby + Rails for many years, I started a project in Elixir + Phoenix, and it took me a pretty short time to get up to speed. From all the new languages I’ve tried in the last couple of years, Elixir is probably the one that you can jump in the fastest (especially if you come from a Ruby background).
Agreed. The application I work at my day job is Rails, but all my web side projects are Elixir and Phoenix. Even ignoring the concurrency aspect, rendering is lighting-fast and my applications are light so I have 3 of them running on my $5/mo Digital Ocean droplet. It’s also much easier to understand and grow an application when you know you can just follow the function chain all the way down rather than worry about a lot of the bloat and overhead that comes with long-running Rails apps.
I appreciate the sentiment of the article, as I think programmers (especially of the kind that read HN/Lobsters) are a bet neophilic (I guess they’ve never heard of the Lindy Effect), a bit of neophilia is necessary in technology to correct for the previous neomania. (h/t Nassim Taleb for introducing these concepts to me via his writing).