Really important point here is that most engineers are only at a job for a couple of years.
That being the case, it is extremely unlikely that we as engineers will be around as issues of scale come up, so we might as well err on the side of making our jobs as easy as possible.
I agree here. Having worked on a lot of old-ish codebases, application drift and environment drift tend to cause whatever the original author thought would be the biggest scale issues to be non-issues within a few years.
For instance, we have a handful of big hulking java codebases whose bulk is mostly related to making it possible to switch between Oracle and DB2. We haven’t used DB2 for anything in 15 years, and none of that code was useful for making it possible to switch to postgres. We have thousands of lines of code intended to compensate for unreliability in ancient DB2 installs, which actually make reliability worse against Oracle or Postgres while making all the processes much slower. The people who designed this left ten years ago, and for the past 5+ years the main bottleneck has been due to premature optimizations that were ultimately ill-considered (like using hadoop for mostly-linear processes that would be faster if done in a shell script, using expensive unreliable third-party services for what amounts to tokenization and ultimately doing our own tokenization most of the time when the third-party service times out or crashes, or adding unnecessary extra steps that require changing the format and location of tens of terabytes of data based entirely on the expectation of nonexistent performance gains for features that were never implemented or seriously planned).
Premature scaling is premature optimization. Just like optimization, it makes more sense to write something straightforwardly, identify actual bottlenecks, and rearchitect and rewrite to counteract them. If rewriting a project you’ve already written is hard, that indicates that you’re using the wrong tools (or that you’ve created the wrong design for the toolset you’re using).
That said, again, making our lives as existing engineers easier is also likely going to help future generations of engineers on the codebase. It’s very hard to execute a Great Leap Forward even if everyone is around to maintain it.
This is in line with ‘expect to throw one away’: the goal of your first implementation of some project is to learn how to implement it; if scaling later becomes necessary, you should expect a full rewrite to be necessary, and if that full rewrite isn’t easier than your initial implementation then you didn’t design your prototype right in the first place.
(Sure, sometimes market pressures make this hard. Market pressures suck, and should be ignored whenever possible. The goal of good software is basically incompatible with the goal of profitable software: pick one.)
I have worked on a fairly big (in terms of traffic and data, if not money) platform, and it was clear that a lot of things were not optimized for the growth they started seeing. If you have a product that as 1-2 years old and achieve success, then perhaps this is less of an issue, but a product that has 5+ years of development because customers trickled in before the hockey stick-growth hit, you suddenly have a lot of cruft that you need to change or rewrite completely, all the while keeping the ship afloat. That said, you can help yourself by keeping your data model flexible and somewhat future proof, then the code matters less.
Really important point here is that most engineers are only at a job for a couple of years.
That being the case, it is extremely unlikely that we as engineers will be around as issues of scale come up, so we might as well err on the side of making our jobs as easy as possible.
I’m not sure whether to invoke The Golden Rule right now or karma.
I would much rather work on a code base that didn’t preemptively try to scale than one that has non working scaling code.
I agree here. Having worked on a lot of old-ish codebases, application drift and environment drift tend to cause whatever the original author thought would be the biggest scale issues to be non-issues within a few years.
For instance, we have a handful of big hulking java codebases whose bulk is mostly related to making it possible to switch between Oracle and DB2. We haven’t used DB2 for anything in 15 years, and none of that code was useful for making it possible to switch to postgres. We have thousands of lines of code intended to compensate for unreliability in ancient DB2 installs, which actually make reliability worse against Oracle or Postgres while making all the processes much slower. The people who designed this left ten years ago, and for the past 5+ years the main bottleneck has been due to premature optimizations that were ultimately ill-considered (like using hadoop for mostly-linear processes that would be faster if done in a shell script, using expensive unreliable third-party services for what amounts to tokenization and ultimately doing our own tokenization most of the time when the third-party service times out or crashes, or adding unnecessary extra steps that require changing the format and location of tens of terabytes of data based entirely on the expectation of nonexistent performance gains for features that were never implemented or seriously planned).
Premature scaling is premature optimization. Just like optimization, it makes more sense to write something straightforwardly, identify actual bottlenecks, and rearchitect and rewrite to counteract them. If rewriting a project you’ve already written is hard, that indicates that you’re using the wrong tools (or that you’ve created the wrong design for the toolset you’re using).
Upvotes are always appreciated. ;)
That said, again, making our lives as existing engineers easier is also likely going to help future generations of engineers on the codebase. It’s very hard to execute a Great Leap Forward even if everyone is around to maintain it.
This is in line with ‘expect to throw one away’: the goal of your first implementation of some project is to learn how to implement it; if scaling later becomes necessary, you should expect a full rewrite to be necessary, and if that full rewrite isn’t easier than your initial implementation then you didn’t design your prototype right in the first place.
(Sure, sometimes market pressures make this hard. Market pressures suck, and should be ignored whenever possible. The goal of good software is basically incompatible with the goal of profitable software: pick one.)
I have worked on a fairly big (in terms of traffic and data, if not money) platform, and it was clear that a lot of things were not optimized for the growth they started seeing. If you have a product that as 1-2 years old and achieve success, then perhaps this is less of an issue, but a product that has 5+ years of development because customers trickled in before the hockey stick-growth hit, you suddenly have a lot of cruft that you need to change or rewrite completely, all the while keeping the ship afloat. That said, you can help yourself by keeping your data model flexible and somewhat future proof, then the code matters less.
I feel like this a long winded version of my moto, which is “Make it work. Make it awesome. Always in that order.”
And sometimes stop after step #1 :)