As someone who has spent the last three years doing almost nothing other than migrations I can agree with much of the content in the lesson’s learned section.
I think it’s unfortunate that almost everyone approaches a migration as an ad-hoc process. In general, a successful migration looks like:
Understanding the semantics of the data.
Making the data reversibly transformable on both systems (possibly adjusting the data model).
Writing data to both systems but reading from the old one.
Writing data to both systems but reading from the new one.
Removing writing to the old one.
What exactly steps 3 and 4 look like depends on the situation. In my case we have replication layer between the two systems and all data is resolvable. IME, people that get to this rough flow have a hard time fighting against the the urge to make the data perfect here. In many cases the data representable during steps 3 and 4 might be non-ideal for the destination system. But that is a new migration that has to happen after migrating systems.
In my experience, most people just focus on building things to send the data around and skip step 1, understanding the semantics of the data and making the data model replication-friendly.
In my experience, most people just focus on building things to send the data around and skip step 1, understanding the semantics of the data and making the data model replication-friendly.
Absolutely. This is a huge problem in my experience and it leads to data models that do a poor job of supporting the operations that eventually need to be performed on it.
The other bad part about migrations is getting the customer to understand their importance. You are constantly fighting the “I want it now!” mentality when step 1 is so very important to making a good product. Rewrites tend to happen because of a lack of understanding of the data model. And it’s rare that the customer even understands their data when they ask you to write software that uses it, so it’s almost inevitable that a rewrite, in some form, will happen. (This is essence of Brooks' famous “build one to throw one away” maxim, in my opinion.)
My advice is to always build for migration. Think about how you will transform what you have into, well, who knows what. But think about it deeply.
In the midst of a bit of a rewrite myself with my team, and it definitely took longer than expected (even after taking into account Hofstadter’s Law).
One key insight / lesson learned from this article I found to be absolutely critical: “… allow running the old and the new system side by side.”
I would have never embarked on the rewrite if I didn’t make this an absolute requirement, despite the additional server costs required and maintenance pain, etc. This is for two reasons.
One, I want to make absolutely sure that users actually view the new system as an “upgrade”. The only way to truly test this, IMO, is to allow users to opt-in to the new system, and also allow them to “roll back” to the old system. If users opt-in and don’t roll back, your rewrite was successful. If they opt-in, stay awhile, and roll back eventually, then something is wrong!
The second reason is what I might call, “release anxiety”. Since I expected the rewrite would take awhile, I wanted to make sure that everyday users were still guaranteed a stable, working, production system even while the new version was almost ready for prime time. This is to allow us to slowly roll out the new system and make sure we don’t have a “big bang” switch over. The “big bang” can be so scary that the release date will be perpetually pushed out, whereas making the new system available to a subset of users is not nearly as scary.
Great article, though. Lots of food for thought. By the way, deciding to rewrite was one of the hardest decisions our team ever made! (In our case, we kept the core programming language and frameworks the same, but changed data stores, data schemas, and user interface.)
As someone who has spent the last three years doing almost nothing other than migrations I can agree with much of the content in the lesson’s learned section.
I think it’s unfortunate that almost everyone approaches a migration as an ad-hoc process. In general, a successful migration looks like:
What exactly steps 3 and 4 look like depends on the situation. In my case we have replication layer between the two systems and all data is resolvable. IME, people that get to this rough flow have a hard time fighting against the the urge to make the data perfect here. In many cases the data representable during steps 3 and 4 might be non-ideal for the destination system. But that is a new migration that has to happen after migrating systems.
In my experience, most people just focus on building things to send the data around and skip step 1, understanding the semantics of the data and making the data model replication-friendly.
Absolutely. This is a huge problem in my experience and it leads to data models that do a poor job of supporting the operations that eventually need to be performed on it.
The other bad part about migrations is getting the customer to understand their importance. You are constantly fighting the “I want it now!” mentality when step 1 is so very important to making a good product. Rewrites tend to happen because of a lack of understanding of the data model. And it’s rare that the customer even understands their data when they ask you to write software that uses it, so it’s almost inevitable that a rewrite, in some form, will happen. (This is essence of Brooks' famous “build one to throw one away” maxim, in my opinion.)
My advice is to always build for migration. Think about how you will transform what you have into, well, who knows what. But think about it deeply.
In the midst of a bit of a rewrite myself with my team, and it definitely took longer than expected (even after taking into account Hofstadter’s Law).
One key insight / lesson learned from this article I found to be absolutely critical: “… allow running the old and the new system side by side.”
I would have never embarked on the rewrite if I didn’t make this an absolute requirement, despite the additional server costs required and maintenance pain, etc. This is for two reasons.
One, I want to make absolutely sure that users actually view the new system as an “upgrade”. The only way to truly test this, IMO, is to allow users to opt-in to the new system, and also allow them to “roll back” to the old system. If users opt-in and don’t roll back, your rewrite was successful. If they opt-in, stay awhile, and roll back eventually, then something is wrong!
The second reason is what I might call, “release anxiety”. Since I expected the rewrite would take awhile, I wanted to make sure that everyday users were still guaranteed a stable, working, production system even while the new version was almost ready for prime time. This is to allow us to slowly roll out the new system and make sure we don’t have a “big bang” switch over. The “big bang” can be so scary that the release date will be perpetually pushed out, whereas making the new system available to a subset of users is not nearly as scary.
Great article, though. Lots of food for thought. By the way, deciding to rewrite was one of the hardest decisions our team ever made! (In our case, we kept the core programming language and frameworks the same, but changed data stores, data schemas, and user interface.)