Just reading this makes me see red–like, blinding rage.
Our build time at work was over 30 minutes. 30. Goddamn. Minutes.
After griping for a while, I finally said “Fuck it, I’m going to solve this problem this week, leave me the hell alone.”
Got it down to 13. Then 11. Finally, below 3 minutes. Still not doing anything clever like incremental linking or whatever–just removing unused includes, refactoring things into support libraries so we don’t keep recompiling the same damned things, etc. etc.
This on an aging Core2 Duo with 4gb ram and spinny disks. Because we don’t believe in spending money on our developer’s tools.
The answer I’ve gotten for like four months is “It’ll happen it’ll happen”…it’s like, fuck you I just saved the cost of three new machines in developer time spent goofing off–even told the suits this. Suits were amused at my list of numbers and charts showing saved money, and were like “dawww it thinks its an MBA…silly developer”.
I can’t even properly express my unhappiness about this.
EDIT: The thing that really pisses me off? The fixes were easy enough that the CTO should’ve done them a long time ago–it’s just a sign that a) his machine is so much better than ours that he doesn’t care and b) he doesn’t do enough coding to feel our pain. That, though, is a whole ‘nother story.
…but if I leave nobody will look out for my team. :(
EDIT: More seriously, it’s rough, because there’s good with the bad, right? The people are nice (even if they make decisions I disagree with sometimes), the problem is actually important (potentially fixing a lot of healthcare, no really, at like the fundamental new tools level), and I get to be pretty flexible in terms of commute and scheduling.
They really do seem to kinda want to help, but they aren’t prioritizing on things like mentioned above, and it isn’t clear what the cashflow/runway situation is in. And the clients aren’t doing us any favors by being super slow hospitals.
I’ve seen that situation before. You can’t fix the culture above you, and from the description of your meeting it sounds like respect has entirely broken down. As important as the project is: do you want to put that organization and culture in charge of the solution?
If you can’t take your team with you, write glowing letters of recommendation to help individuals leave when you go. But you have to go, for your sanity.
Joking aside, it’s astonishing to me that, in an industry where our tooling is by it’s very nature entirely plastic, we put up with garbage fires like sbt; in fact, not only do we tolerate it, we get so beaten down that we celebrate the lousiness of tools, treating our scars as a mark of character, instead of stupidity.
I’ve never worked anywhere with a reasonable build strategy. I jumped ship from the last place in part because I didn’t feel I could ever be productive, and a large part of that was the build process. It took about 2 hours to build and run unit and integration tests.
Then I ended up somewhere where compilation takes ~10 minutes. Unit tests take 2 hours. Integration tests take all night. So now I’ve accepted that this is the norm, and that I will never be even 5% as productive as I am working on my own projects or working for a company under 10 or so people.
Basically what I’m getting at is that you can’t examine pain threshold in a vacuum. It’s not a one dimensional vector where you can say “ok, this process has too much friction, fuck it”. There are external factors that play into a decision to alleviate that pain. The reason why I am not actively working on making the process better is because there’s a ton of momentum behind the current way of doing stuff, and organizational challenges in trying to change strategy such that I could work 24/7 for the next year and would not be able to put a dent in it. And the reason why I’m working at the place I’m at is in part because I’m not confident that anywhere else I would go would have a sane workflow either, as I’ve already gotten bitten by this once by jumping ship.
Eventually this sort of debt catches up to any company and progress grinds to a halt.
I wonder why you didn’t use the interview with the new company to make sure they do it better. I mean, isn’t the hole point of an interview to get information from the other side to be able to judge whether it is a good fit for both sides?
This is why TDD is silly. I typically code Monday through Friday afternoon without compiling or running. Then I spend half an hour fixing any typos that my editor missed and run the feature tests I wrote. Build times become a non issue if you don’t build often. Especially if you don’t have to run tests for every line of code you type.
If you can keep a week’s worth of code in your head without making errors beyond typos… then you’re not making effective use of your talent, and should be working on harder problems.
I can write code without tests. But I can go faster with tests than without, because having the confidence of the tests means I can spend much less time thinking when actually writing the code.
There are two tricks to making what I said work. First you write less code. Unit tests basically double your code base. And also test the easiest part to get right, individual units. The harder part is to make sure units are composed sanely. This is where the second trick comes in. I use Design by Contract to give me that confidence that you use unit tests to achieve.
In my previous company, we used an in-house build system that took 20 minutes on average to build the entire code base. That’s a lot of time gone down the tube, considering that we were +10 developers and each one of us built everything several times a day. So I resolved to improve the dire situation and, by parallelizing the steps that comprised a build, I got to reduce the time to a bit more than 1 minute. I felt like a hero.
Now, I read that these guys execute the build and the tests in under a second. FML
Apparently our pain threshold is obscenely high. The build time itself is not shocking, ~10 minutes, but the unit tests take many, many hours to run. So many, that you never run them on your own machine; you submit them to the test cluster and it splits them up and runs them on a bunch of the ~100 machines that are dedicated to testing. It’s “Enterprise Java” and there’s a lot of “oh well, this is just how it is”, so we’ve habituated to the high level of pain.
I think people do habituate. A previous job had it so that frequently it could take > 24 hours to find out if your change worked (this wasn’t the test suite per se, but it was still a vital part of validating your change). People basically classed this as “Yeah it’s quite annoying but you just sort of get used to it”.
That wasn’t the only reason I left, but it was definitely a contributing factor.
My current project takes about an hour to do a full test suite run. :-(
It’s not as bad as all that because I’ve spent a lot of work in organising it into fast and slow paths, the tests are all broken up, etc. so for 90% of my work my testing loop involves tests that take a couple seconds to run, and I have a segment of the test suite that still gives me 100% branch coverage and runs in ~a minute (although recently that’s crept up to closer to two), but it’s still one of the most frustrating things about working on this project. Possibly the most frustrating thing if we don’t count “Python” as one thing.
Just reading this makes me see red–like, blinding rage.
Our build time at work was over 30 minutes. 30. Goddamn. Minutes.
After griping for a while, I finally said “Fuck it, I’m going to solve this problem this week, leave me the hell alone.”
Got it down to 13. Then 11. Finally, below 3 minutes. Still not doing anything clever like incremental linking or whatever–just removing unused includes, refactoring things into support libraries so we don’t keep recompiling the same damned things, etc. etc.
This on an aging Core2 Duo with 4gb ram and spinny disks. Because we don’t believe in spending money on our developer’s tools.
The answer I’ve gotten for like four months is “It’ll happen it’ll happen”…it’s like, fuck you I just saved the cost of three new machines in developer time spent goofing off–even told the suits this. Suits were amused at my list of numbers and charts showing saved money, and were like “dawww it thinks its an MBA…silly developer”.
I can’t even properly express my unhappiness about this.
EDIT: The thing that really pisses me off? The fixes were easy enough that the CTO should’ve done them a long time ago–it’s just a sign that a) his machine is so much better than ours that he doesn’t care and b) he doesn’t do enough coding to feel our pain. That, though, is a whole ‘nother story.
Sounds like a ship to jump off of…
…but if I leave nobody will look out for my team. :(
EDIT: More seriously, it’s rough, because there’s good with the bad, right? The people are nice (even if they make decisions I disagree with sometimes), the problem is actually important (potentially fixing a lot of healthcare, no really, at like the fundamental new tools level), and I get to be pretty flexible in terms of commute and scheduling.
They really do seem to kinda want to help, but they aren’t prioritizing on things like mentioned above, and it isn’t clear what the cashflow/runway situation is in. And the clients aren’t doing us any favors by being super slow hospitals.
Admirable feeling, but you really need to look out for yourself. No sense prolonging the process if the writing is already on the wall.
I’ve seen that situation before. You can’t fix the culture above you, and from the description of your meeting it sounds like respect has entirely broken down. As important as the project is: do you want to put that organization and culture in charge of the solution?
If you can’t take your team with you, write glowing letters of recommendation to help individuals leave when you go. But you have to go, for your sanity.
Perhaps this is why your handle is “angersock”.
Would you say that that pain was “bold and unrelenting”? Or, more “caustic and burning”?
Joking aside, it’s astonishing to me that, in an industry where our tooling is by it’s very nature entirely plastic, we put up with garbage fires like sbt; in fact, not only do we tolerate it, we get so beaten down that we celebrate the lousiness of tools, treating our scars as a mark of character, instead of stupidity.
I’ve never worked anywhere with a reasonable build strategy. I jumped ship from the last place in part because I didn’t feel I could ever be productive, and a large part of that was the build process. It took about 2 hours to build and run unit and integration tests.
Then I ended up somewhere where compilation takes ~10 minutes. Unit tests take 2 hours. Integration tests take all night. So now I’ve accepted that this is the norm, and that I will never be even 5% as productive as I am working on my own projects or working for a company under 10 or so people.
Basically what I’m getting at is that you can’t examine pain threshold in a vacuum. It’s not a one dimensional vector where you can say “ok, this process has too much friction, fuck it”. There are external factors that play into a decision to alleviate that pain. The reason why I am not actively working on making the process better is because there’s a ton of momentum behind the current way of doing stuff, and organizational challenges in trying to change strategy such that I could work 24/7 for the next year and would not be able to put a dent in it. And the reason why I’m working at the place I’m at is in part because I’m not confident that anywhere else I would go would have a sane workflow either, as I’ve already gotten bitten by this once by jumping ship.
Eventually this sort of debt catches up to any company and progress grinds to a halt.
I wonder why you didn’t use the interview with the new company to make sure they do it better. I mean, isn’t the hole point of an interview to get information from the other side to be able to judge whether it is a good fit for both sides?
What is the size of your codebase?
This is why TDD is silly. I typically code Monday through Friday afternoon without compiling or running. Then I spend half an hour fixing any typos that my editor missed and run the feature tests I wrote. Build times become a non issue if you don’t build often. Especially if you don’t have to run tests for every line of code you type.
If you can keep a week’s worth of code in your head without making errors beyond typos… then you’re not making effective use of your talent, and should be working on harder problems.
I can write code without tests. But I can go faster with tests than without, because having the confidence of the tests means I can spend much less time thinking when actually writing the code.
There are two tricks to making what I said work. First you write less code. Unit tests basically double your code base. And also test the easiest part to get right, individual units. The harder part is to make sure units are composed sanely. This is where the second trick comes in. I use Design by Contract to give me that confidence that you use unit tests to achieve.
I recommend you check that technique out.
In my previous company, we used an in-house build system that took 20 minutes on average to build the entire code base. That’s a lot of time gone down the tube, considering that we were +10 developers and each one of us built everything several times a day. So I resolved to improve the dire situation and, by parallelizing the steps that comprised a build, I got to reduce the time to a bit more than 1 minute. I felt like a hero. Now, I read that these guys execute the build and the tests in under a second. FML
Apparently our pain threshold is obscenely high. The build time itself is not shocking, ~10 minutes, but the unit tests take many, many hours to run. So many, that you never run them on your own machine; you submit them to the test cluster and it splits them up and runs them on a bunch of the ~100 machines that are dedicated to testing. It’s “Enterprise Java” and there’s a lot of “oh well, this is just how it is”, so we’ve habituated to the high level of pain.
I think people do habituate. A previous job had it so that frequently it could take > 24 hours to find out if your change worked (this wasn’t the test suite per se, but it was still a vital part of validating your change). People basically classed this as “Yeah it’s quite annoying but you just sort of get used to it”.
That wasn’t the only reason I left, but it was definitely a contributing factor.
My current project takes about an hour to do a full test suite run. :-(
It’s not as bad as all that because I’ve spent a lot of work in organising it into fast and slow paths, the tests are all broken up, etc. so for 90% of my work my testing loop involves tests that take a couple seconds to run, and I have a segment of the test suite that still gives me 100% branch coverage and runs in ~a minute (although recently that’s crept up to closer to two), but it’s still one of the most frustrating things about working on this project. Possibly the most frustrating thing if we don’t count “Python” as one thing.