devops should have installed some kind of error tracking
This reads as if there is a team called “devops”?
DevOps is a practice or principle - so an extra learning you could take away here is that as a dev you could try and work more closely with ops to ensure things like proper monitoring of your own systems are in place. Don’t assume that an ops team will magically know what to monitor!
And every time I see one I cry a little… It’s good if a company wants to follow DevOps practices but having a centralised team is often a smell they’re not actually doing it well.
As the company was apparently recording 50% of all interactions to their website for “marketing” purposes, they might have just tried to correlate the lost inputs with the recordings and got some intern to copy them after the fact…
The technical problems don’t matter. This was a human failing on multiple levels and among multiple people. At this point it’s a learning experience, and a hint that maybe the process is a little too much process and not enough justification.
It’s scary to make these mistakes, but at least you know where you made those mistakes.
First, as mentioned in the story, this should have had much better testing. I’m not sure how QA works there, but this should definitely have been caught by any decent QA, since it appears it resulted from a common enough flow from the consumer.
Not having metrics or tests in production is crazy! Especially around PHP apps where values might not always be what you expect during a happy path scenario. If you need to use string literals for naming, at least have their values set in constants/enums, so things like this are better avoided.
This reads as if there is a team called “devops”?
DevOps is a practice or principle - so an extra learning you could take away here is that as a dev you could try and work more closely with ops to ensure things like proper monitoring of your own systems are in place. Don’t assume that an ops team will magically know what to monitor!
At most places there is such a team.
And every time I see one I cry a little… It’s good if a company wants to follow DevOps practices but having a centralised team is often a smell they’re not actually doing it well.
I think it’s connected to metrics for compliance instead of enlightenment.
As the company was apparently recording 50% of all interactions to their website for “marketing” purposes, they might have just tried to correlate the lost inputs with the recordings and got some intern to copy them after the fact…
It’s pretty amazing that they weren’t collecting telemetry for debugging but they were collecting vast amounts of data for marketing.
“We know exactly what the leads are clicking on, but we don’t know who they are.” - marketing, you’re doing it wrong…
The technical problems don’t matter. This was a human failing on multiple levels and among multiple people. At this point it’s a learning experience, and a hint that maybe the process is a little too much process and not enough justification.
It’s scary to make these mistakes, but at least you know where you made those mistakes.
There are a few things sticking out here.
First, as mentioned in the story, this should have had much better testing. I’m not sure how QA works there, but this should definitely have been caught by any decent QA, since it appears it resulted from a common enough flow from the consumer.
Not having metrics or tests in production is crazy! Especially around PHP apps where values might not always be what you expect during a happy path scenario. If you need to use string literals for naming, at least have their values set in constants/enums, so things like this are better avoided.
I find the sentence
somewhat confusing…if the “edge cases” aren’t covered, is it really being tested at all?
You have to weigh costs and benefits. Having 100% coverage is exponentially harder to do than having 80% coverage.
If you spend too much effort on preventing bugs, it may turn out that the cost of writing bug-free code is higher than the loss from the bugs.
That’s very true in general. In this case though, if the tests don’t cover a clearly pretty common user flow, I question their value.
Unit testing for compliance instead of enlightenment.
smoke-unit-test