I couldn’t not post something when I read the title. I’m currently waiting for a test which takes about 15 minutes. Thing is, I have to compare the output to values defined in a CSV, and the biggest chunk of the time is waiting for the input to be loaded. Usually, there are a lot of errors immediately, but I only see this after 15 minutes. Then, I have to open the Excel file which computes the values used for checking (takes ~10 minutes), and find the corresponding computation in both the program and the Excel file. When I fix the error, I have to fix the error in the code or in the Excel file. When it’s in the Excel file (it usually is), I have to run it (takes about 20 minutes), and update the CSV in some specific location, and run a program to put the new values in the database. This again, takes about 20 minutes.
So all in all, I spend about an hour for every error I find. This can be (and often is) something as simple as a typo in the Excel file. I have complained about this process before, but I don’t have the time/authority to change this (there are about 15-20 people working on this project, so I can’t just change the workflow if it’s not a task that is assigned to me). I don’t know why others think this is acceptable. Maybe because I’m usually the one ending up doing this tedious task, because I’m the ‘technical guy’ in my team.
This all used to really drag me down to the point of taking my work home and having a bad mood because of it. Now I’m a bit apathetic. If they don’t fix this, they’ll just pay me to do dumber work and be less productive. Their loss.
My favorite was a discussion about a review of your book the other day where someone complained that unit tests cause tests to be too fine grained, and that logic of your solution creeps into your tests. Seems like that’s a good litmus test for complexity getting out of hand.
My work code base mingles what are clearly integration tests with unit tests and as a result no one runs the full suite before pushing new code and means master has a 50/50 chance of being broken at any given time. It’s terrible.
Consider yourself lucky – the test suite on a project I worked on 10 years ago took over 6 hours to run (the developers didn’t consider the speed of the tests when writing them – since the tests are “not in the fast path”).
One thing we did do well is make sure that each test in the suite tested one and only one thing. If the suite failed, it was possible to re-run just one of the failing tests (which would take 10-20 seconds), rather than re-running the entire suite. In your case, it sounds like the developers of the tests might benefit from AAA (Arrange, Act, Assert) – which applies mostly to unit tests, but can also be used in integration tests.
I’m talking about an individual test, which takes about 15 minutes (we have some which take up to half an hour). If you run all tests, it takes like 4 hours, I guess. So it often happens that you run tests before pushing to develop, and while the tests are running, someone pushes to develop, so you have to merge and run the tests again (and hope that no one pushes this time). Or, you just run some important tests and push. Then, if you break develop, you’ll know after 4 hours (and of course, people will have pulled from and pushed to develop).
I feel like there has to be a way in which you can take advantage of everyone’s apathy and disinterest in that task, artificially increase the cost of it over time, like pretend the spreadsheet takes an hour to load, and then try to do something about it in the newly created dead time? It’ll be slow work, but incremental improvements do lead places.
Start getting paid by hour and detach yourself from the process and see your joy and happiness raise dramatically.
You’ll be absolutely delighted to know that tests have slowed down, as it allows you an extra cup of coffee/tea and another round of play with your doggo (or catto) :P
Usually, there are a lot of errors immediately, but I only see this after 15 minutes.
Surely there’s a way to make your test framework fail fast? If it’s a case of loading everything into memory (which I doubt), then again surely there’s some streaming library for your language.
Jup, it’s not even very hard. It’s just not a priority, so it’s never fixed. Also, most team leads don’t usually don’t do the job of running and fixing these tests, so they don’t really feel the pain of having slow tests.
I estimate that many steps in the process do about 10 to 1000 times the strictly necessary work:
If you want to compare tests results you have to obtain output from a big excel file, that is about 60 MB big (even though you just need one sheet).
If you load new input, you have to reload all the output (100s of MBs)
If you do a test, you first load all the values into memory
Agree 100%. The first time I wrote my own Ruby gem, I was thrilled that the tests ran in a fraction of a second - unlike the Rails apps I’d worked on. I could run the whole suite after every change and feel very confident I hadn’t broken anything, and I never got distracted waiting for tests.
Since I’ve bee working in Elixir, I’ve really enjoyed how little effort it takes to make tests fast and concurrent. It’s a big productivity boost. The only slow tests I’ve seen were some that fired up a headless browser.
Elixir/Phoenix are also the only language/framework pair I work in* where it’s easy for it to be concurrent and fast with end to end testing too! (see here for inspiration)
*I’m sure it’s possible in many frameworks, but I haven’t found a comparable way in Rails for sure
Hmm, it might be helpful to split automated testing into 3 levels :
Lint checks, unit tests etc which are triggered when you raise a merge request. The purpose of these tests is to reduce the load on reviewers.
Once the merge request is approved, it runs through integration tests. These tests should not be flaky and if they fail it should fail loudly and try to indicate where the problem could be. If they succeed, the merge request can be automatically merged. These tests can be slow, but it runs offline and allows developers to context switch safely.
Regression tests, which could contain flaky tests but gives us clues about potential bugs in the software.
I am playing with ghcid + foreign-store right now.
Coupled together they enable for really fast type-checking and immediate web server restarts. I have used stack build --fast --file-watch before, but it took at least several seconds to complete. This really helps me keep focused.
Nobody likes slow tests, but I always only see this from the “unit tests yay” camp.
I don’t want to argue against unit tests, but there are projects and teams that have a lot more focus on system/integration tests (often involving multiple systems) that seem to be a lot worse. (I’m looking at our setup, for example. unit tests have been green for months, run 36secs right now) whereas the integration test suite runs 6-12 minutes, and is kind of flaky. Depending on the hardware where it runs, and I’m trying to make them less flaky.
I can’t remember ever seeing a decent suite of integration tests involving real DB access and maybe even some message queues or other daemons that consistently ran below 3-4 minutes. Maybe back when I did PHP and we only had an SQLite DB in memory…
I don’t personally think unit tests are the magic solution to all testing problems, sometimes integration tests are needed. And they do tend to be slower.
But there’s still the question of whether it’s 6 minutes vs 20 minutes or 40 minutes or 60 minutes. You can’t always make things superfast, but there’s usually ways to make things somewhat better.
Agree. However, I’d argue you shouldn’t really wait on tests.
Sure, when tweaking something run relevant tests locally. That’s usually just a small subset though. Then push to origin and let Continuous Integration take care of the rest to test in the background. Don’t wait on it, just take action if it suddenly fails due to your changes. When merging, enable auto merge once testing pipelines succeed. Don’t wait on it, don’t waste time, continue with a new feature.
This is probably easier said than done, but I do believe there are quite a few projects and developers that could benefit from such a workflow. And I don’t want to use this as an argument to not-optimize your tests. But, when you do have to wait, be sure to use your time as efficiently as possible.
I’m sorry but the calculation of money lost while devs are waiting for tests is misleading.
Until testing takes up a significant portion of a developers time 6 minutes here and there mean nothing and further devs should be encouraged to take some time off the screen to clear their mind in between tasks for the sake of their mental health.
Also the time saved does not add up with each additional employee.
I couldn’t not post something when I read the title. I’m currently waiting for a test which takes about 15 minutes. Thing is, I have to compare the output to values defined in a CSV, and the biggest chunk of the time is waiting for the input to be loaded. Usually, there are a lot of errors immediately, but I only see this after 15 minutes. Then, I have to open the Excel file which computes the values used for checking (takes ~10 minutes), and find the corresponding computation in both the program and the Excel file. When I fix the error, I have to fix the error in the code or in the Excel file. When it’s in the Excel file (it usually is), I have to run it (takes about 20 minutes), and update the CSV in some specific location, and run a program to put the new values in the database. This again, takes about 20 minutes.
So all in all, I spend about an hour for every error I find. This can be (and often is) something as simple as a typo in the Excel file. I have complained about this process before, but I don’t have the time/authority to change this (there are about 15-20 people working on this project, so I can’t just change the workflow if it’s not a task that is assigned to me). I don’t know why others think this is acceptable. Maybe because I’m usually the one ending up doing this tedious task, because I’m the ‘technical guy’ in my team.
This all used to really drag me down to the point of taking my work home and having a bad mood because of it. Now I’m a bit apathetic. If they don’t fix this, they’ll just pay me to do dumber work and be less productive. Their loss.
And people wondered why I favored unit tests over integration tests in Working Effectively with Legacy Code.
My favorite was a discussion about a review of your book the other day where someone complained that unit tests cause tests to be too fine grained, and that logic of your solution creeps into your tests. Seems like that’s a good litmus test for complexity getting out of hand.
My work code base mingles what are clearly integration tests with unit tests and as a result no one runs the full suite before pushing new code and means master has a 50/50 chance of being broken at any given time. It’s terrible.
Consider yourself lucky – the test suite on a project I worked on 10 years ago took over 6 hours to run (the developers didn’t consider the speed of the tests when writing them – since the tests are “not in the fast path”).
One thing we did do well is make sure that each test in the suite tested one and only one thing. If the suite failed, it was possible to re-run just one of the failing tests (which would take 10-20 seconds), rather than re-running the entire suite. In your case, it sounds like the developers of the tests might benefit from AAA (Arrange, Act, Assert) – which applies mostly to unit tests, but can also be used in integration tests.
I’m talking about an individual test, which takes about 15 minutes (we have some which take up to half an hour). If you run all tests, it takes like 4 hours, I guess. So it often happens that you run tests before pushing to develop, and while the tests are running, someone pushes to develop, so you have to merge and run the tests again (and hope that no one pushes this time). Or, you just run some important tests and push. Then, if you break develop, you’ll know after 4 hours (and of course, people will have pulled from and pushed to develop).
That’s solved by having a good integration flow, like bors-ng or zuul, in place. If your patch breaks the test suite, your patch should not land.
I feel like there has to be a way in which you can take advantage of everyone’s apathy and disinterest in that task, artificially increase the cost of it over time, like pretend the spreadsheet takes an hour to load, and then try to do something about it in the newly created dead time? It’ll be slow work, but incremental improvements do lead places.
Start getting paid by hour and detach yourself from the process and see your joy and happiness raise dramatically.
You’ll be absolutely delighted to know that tests have slowed down, as it allows you an extra cup of coffee/tea and another round of play with your doggo (or catto) :P
Surely there’s a way to make your test framework fail fast? If it’s a case of loading everything into memory (which I doubt), then again surely there’s some streaming library for your language.
Jup, it’s not even very hard. It’s just not a priority, so it’s never fixed. Also, most team leads don’t usually don’t do the job of running and fixing these tests, so they don’t really feel the pain of having slow tests.
I estimate that many steps in the process do about 10 to 1000 times the strictly necessary work:
Agree 100%. The first time I wrote my own Ruby gem, I was thrilled that the tests ran in a fraction of a second - unlike the Rails apps I’d worked on. I could run the whole suite after every change and feel very confident I hadn’t broken anything, and I never got distracted waiting for tests.
Since I’ve bee working in Elixir, I’ve really enjoyed how little effort it takes to make tests fast and concurrent. It’s a big productivity boost. The only slow tests I’ve seen were some that fired up a headless browser.
Elixir/Phoenix are also the only language/framework pair I work in* where it’s easy for it to be concurrent and fast with end to end testing too! (see here for inspiration)
*I’m sure it’s possible in many frameworks, but I haven’t found a comparable way in Rails for sure
Hmm, it might be helpful to split automated testing into 3 levels :
I am playing with
ghcid+foreign-storeright now.Coupled together they enable for really fast type-checking and immediate web server restarts. I have used
stack build --fast --file-watchbefore, but it took at least several seconds to complete. This really helps me keep focused.Nobody likes slow tests, but I always only see this from the “unit tests yay” camp.
I don’t want to argue against unit tests, but there are projects and teams that have a lot more focus on system/integration tests (often involving multiple systems) that seem to be a lot worse. (I’m looking at our setup, for example. unit tests have been green for months, run 36secs right now) whereas the integration test suite runs 6-12 minutes, and is kind of flaky. Depending on the hardware where it runs, and I’m trying to make them less flaky.
I can’t remember ever seeing a decent suite of integration tests involving real DB access and maybe even some message queues or other daemons that consistently ran below 3-4 minutes. Maybe back when I did PHP and we only had an SQLite DB in memory…
I don’t personally think unit tests are the magic solution to all testing problems, sometimes integration tests are needed. And they do tend to be slower.
I’ve been writing about various approaches to dealing with this (speeding up databases: https://pythonspeed.com/articles/faster-db-tests/ (you can run Postgres and other databases in memory!) and fallbacks when you can’t speed things up, e.g. failing faster: https://pythonspeed.com/articles/slow-tests-fast-feedback/
But there’s still the question of whether it’s 6 minutes vs 20 minutes or 40 minutes or 60 minutes. You can’t always make things superfast, but there’s usually ways to make things somewhat better.
pytest -x # stop after first failure
Agree. However, I’d argue you shouldn’t really wait on tests.
Sure, when tweaking something run relevant tests locally. That’s usually just a small subset though. Then push to origin and let Continuous Integration take care of the rest to test in the background. Don’t wait on it, just take action if it suddenly fails due to your changes. When merging, enable auto merge once testing pipelines succeed. Don’t wait on it, don’t waste time, continue with a new feature.
This is probably easier said than done, but I do believe there are quite a few projects and developers that could benefit from such a workflow. And I don’t want to use this as an argument to not-optimize your tests. But, when you do have to wait, be sure to use your time as efficiently as possible.
Sure, mostly I don’t wait, but that brings on the second cost I talk about: context switching.
I’m sorry but the calculation of money lost while devs are waiting for tests is misleading. Until testing takes up a significant portion of a developers time 6 minutes here and there mean nothing and further devs should be encouraged to take some time off the screen to clear their mind in between tasks for the sake of their mental health.
Also the time saved does not add up with each additional employee.
I want to take a break when I want to. If I’m in a flow state the slowness of tests can be really frustrating.