These tests should be run on a completely different time cycle than ordinary CI tests. Once per day, for example, rather than per commit.
Also known as Availability Testing. I think that the bigger lesson from the article is that reliance on third party APIs without an SLA is risky.
Indeed, though it sounds like in this particular case, the third-party api being down wasn’t that big of a deal.
While reading this I realised that integration tests with external services like that, that are outside our control, are really unlike other kinds of tests — unit tests or integration tests with services under our control, like a database we can spawn a test instance of — in the sense that they break at an independent timeframe for different reasons. And hence we should not aim to run them at the same time and with the same frequency as the other tests.
I see that there is value in keeping both “unit” tests with a mocked external API and real integration tests. The external API can break without any anticipation, but while we’re developing some feature against it we don’t normally need to run against the real API so a mock should suffice.
From reading the title, my first thought was that we should always hit the real API, since that’s what we want to test; unless there’s a specific, non-dogmatic reason to avoid it (“best practice” is dogmatic, “it’s slow” is vague, “X% of our testing time is spent waiting for these requests, testing takes Yx longer than the maximum feedback cycle time recommended by Z, and we’ve exhausted low-hanging fruit alternative measures like running these tests concurrently” is specific and non-dogmatic)
This seems like a use-case for self-initialising mocks, where we cache some real API responses and use those in tests; every now and then we can flush the cache to repopulate from the real API.
Also reminds me of an issue I faced, where I was asked to add some error handling to our code (hooray for outsourcing!) since a third-party was performing some maintenance and their API was giving errors that were causing our site to give a blank 500 page. I added the exception handlers then went back to other work. When the API was still giving errors a while later, we contacted the third-party who said their maintenance shouldn’t have affected that particular API. It turned out to have been a problem with the routing tables between our servers and theirs (or some similar networking thingy).
Jeffrey Way, a major contributor to the Laravel community, has been tweeting about this lately. I think his overarching point is that when we spend too much time thinking about how to write a perfect unit test for a given piece of code and not a perfect test overall, we get trapped in thinking about the specific boundaries between unit/functional/integration and not thinking about how best to test the code we write.
Great point. It’s good to drill into new programmers heads to mock out external dependencies, but part of a healthy application development lifecycle is making sure external dependencies are working as intended.
I usually prefer mocking external APIs.