For a deployment failure, immediately reverting is always “Plan A”, and we definitely considered this right off. But, dear Redditor… Kubernetes has no supported downgrade procedure. Because a number of schema and data migrations are performed automatically by Kubernetes during an upgrade, there’s no reverse path defined. Downgrades thus require a restore from a backup and state reload!
I am so glad when tooling goes out of their way to work well for reverts. I think that a lot of operational tooling doesn’t when it totally should (looking at you, Postgres). Obviously hard work and hard to test, but when you have software where you can run one version behind and the current version next to each other, suddenly this sort of stuff feels a lot less scary.
It is possible to perform an incremental update of your Kubernetes control plane node, i.e. running multiple minor versions in parallel. I don’t know how officially supported it is but it’s normal to have at least a small window when you update them in order.
I am so glad when tooling goes out of their way to work well for reverts. I think that a lot of operational tooling doesn’t when it totally should (looking at you, Postgres). Obviously hard work and hard to test, but when you have software where you can run one version behind and the current version next to each other, suddenly this sort of stuff feels a lot less scary.
It is possible to perform an incremental update of your Kubernetes control plane node, i.e. running multiple minor versions in parallel. I don’t know how officially supported it is but it’s normal to have at least a small window when you update them in order.