It depends on what you’re trying to deploy and what constraints you have; there isn’t one magic bullet. One pipeline I was especially proud of for a Python app I wrote at Fog Creek worked like this:
Create a pristine dump of the target version of the source code. Say the revision is 1a2b3c4d. We used Mercurial, not Git, so the command was hg archive -t tbz2 -R /path/to/bare/repo -r 1a2b3c4d, but you can do the same in Git.
Upload this to each server that’ll run the app, into /srv/apps/myapp/1a2b3c4d
Based on the SHA1 of requirements.txt, make a new virtualenv if necessary in /srv/virtualenvs/<sha1 of requirements.txt> on each server hosting the app.
Copy general configuration into /srv/config/myapp/1a2b3c4d. Configs are generally stored outside the actual app repo for security reasons, even in a company that otherwise uses monolithic repositories, so the SHA here matches the version of the app designed to consume this config info, not the SHA of the config info itself. (Which should also make sense intuitively, since you may need to reconfigure a running app without deploying a new version.)
Introduce a new virtual host, 1a2b3c4d.myapp.server.internal, that serves myapp at revision 1a2b3c4d.
Run integration tests against this to make sure everything passes.
Switch default.myapp.server.internal to point to 1a2b3c4d.myapp.server.internal and rerun tests.
If anything goes wrong, just switch symlinks of default.myapp.server.internal back to the old version.
Now, that’s great for an app that’s theoretically aiming for five-nines uptime and full replacability. But the deploy process for my blog is ultimately really just rsync -avz --delete. It really just comes down to what you’re trying to do and what your constraints are.
I doubt you’ll find consistent views, which makes that the opposite of a stupid question.
My ideal deployment pipeline looks something like the following:
Deployable artifacts are built directly from source control by an automated system (e.g. Jenkins).
Ideally, some sort of gate is in place to ensure code review has occurred before a deployable artifact is built (e.g. Gerritt, though that project is very dogmatic and while I don’t disagree with it, I don’t strongly stand behind it either).
CI builds off unreviewed commits are fine, but I would consider them a developer nicety, not a part of the deployment pipeline.
Deployable artifacts are stored somewhere. Only the build tool should be able to write to it, but anyone should be able to read from it. (I don’t care what this looks like. Personally, I’d probably just use a file server.)
Deployment into a staging environment is one-click, or possibly automatic, from the artifact store.
Deployment into a production environment is one-click from the staging environment. The application must have successfully deployed into staging to be deployed into prod. Ideally, the application must go through some QA in staging to be deployed to prod, but that’s a process concern more than a technical one.
Operational personnel need to be able to bypass QA (in fact, bypass almost all of this pipeline) in outage situations.
Note that I’m coming at this from the server-side perspective; “deploy into” means something different for desktop/client software, but I think the overall flow should still work (though I’ve never professionally developed client software, so I don’t know for sure).
Tests are ran, gpg signatures on commits checked, we are ready to deploy!
Create a clean export of the revision we are deploying: hg archive -r 1a2b3c4d build/
Dump the revision # and other build info into a version file in build/ this is in JSON format as a {}.
Shove this into a docker image: docker build -t $(JOB_NAME):$(BUILD_NUMBER) . and push it to internal docker registry.
Update nomad(hashicorp product) config file to point to the new $(BUILD_NUMBER) via sed: sed -e "s/@@BUILD_NUMBER@@/$(BUILD_NUMBER)/g" $(JOB_NAME).nomad.sed >$(JOB_NAME).nomad.
Do the same as previous step but for the environment we will be running in (dev, test, prod) if required.
nomad run $(JOB_NAME).nomad
Nomad will handle dumping vault secrets, config information, etc from the template directive in the config file. So Configuration happens outside of the repo, and lives in Vault and Consul.
You can tell by the env. variables we use Jenkins :) different CI/CD systems will have different variables probably. If unfamiliar with Jenkins, BUILD_NUMBER is just a integer count of how many builds jenkins has done for that job. JOB_NAME is just the name you gave it inside of Jenkins for this job.
This is way off topic, but I’d love to hear why you went with Nomad and how it’s been working for you. It seems to fill the same niche as Kubernetes, but I hear practically nothing about it—even at shops using Packer, Terraform, and other Hashicorp products.
We started with Nomad before Kubernetes was a huge thing, i.e. we heard about Nomad first. But I wouldn’t change that decision now, looking back. Kubernetes is complicated. Operationally it’s a giant pain. I mean it’s awesome, but it’s a maintenance burden. Nomad is operationally simple.
Also Nomad runs things outside of docker just fine, so we can effectively replace supervisor, runit, systemd, etc with Nomad. Not that I remotely suggest actually replacing systemd/PID 1 with Nomad, but that all the daemons and services you normally run on top of your box can be put under nomad, so you have 1 way of deploying, regardless of how it runs. I.e. Postgres tends to work better on bare hardware, since it’s very resource intensive, but with the Nomad exec driver it runs on bare hardware under Nomad perfectly fine, and gives us 1 place to handle logs, service discovery, process management, etc. I think maybe the newer versions of Kubernete’s can sort of do that now, but I don’t think it’s remotely easy, but I don’t really keep up.
But mostly the maintenance burden. I’ve never heard anyone say Kubernetes is easy to setup or babysit. Nomad is ridiculously easy to babysit. It’s the same reason Go is popular, it’s a fairly boring, simple language complexity wise. This is it’s main feature.
Inside of Jenkins job config we have an ENV variable called MODE and it is an enum, one of: dev, test, prod
Maybe you can derive it from the job-name, but the point is you need 1 place to define if it will run in dev/test/prod mode.
So if I NEED to build differently for dev, test or prod (say for new dependencies coming in or something, I can.
That same MODE env variable is pushed into the nomad config:
env {
MODE = “dev”
}
It’s put there by sed, identically to how I put the $(BUILD_NUMBER).
And also, if there are config changes needed to the nomad config file based on environment, say the template needs to change to pull from the ‘dev’ config store instead of the ‘prod’ config store, or if it gets a development vault policy instead of a production one, etc. I also do these with sed, but you could use consul-template, or some other templating language if one wanted. Why sed? because it’s always there and very reliable, it’s had 40 years of battle testing.
So that when the nomad job starts, it will be in the processes environment. The program can then, if needed act based on the mode in which it’s running. Like say turning on feature flags under testing or something.
Obviously all of these mode specific changes should be done sparingly, you want dev, test, prod to behave as identically as possible, but there are always gotchas here and there.
I do a “git push” from the development box into a test repo on the server. There, a post-update hook checks out the files and does any other required operation, after which is runs some quick tests. If those tests pass, the hook pushes to the production repo where another post-update hooks does the needful, including a true graceful reload of the application servers.
If those tests fail, I get an email and the buggy code doesn’t get into production. The fact that no other developer can push their code into production while the codebase is buggy is considered a feature.
Since I expect continuous integration to look like my setup, I don’t see the point of out-of-band testing that tells you that the code that reached production a few minutes ago is broken.
I think that the point kaiju’s thread wants to make is that you shouldn’t be deploying from your local machine, since every developers environment will differ slightly and those artifacts might cause a bad build when sent to production. I believe the normal way is to have the shared repo server build/test and deploy on a push hook, so that the environment is the same each time.
The setup we use is not even advanced, but simply resilient against all the annoyances we’ve encountered over time running in production.
I don’t really understand the description underneath “the right pattern” of the article. It seems weird to have a deploy tree you reuse everytime?
Make a clean checkout everytime. You can still use a local git mirror to save on data fetched. Jenkins does this right, as long as you add the cleanup step in checkout behaviour.
From there, build a package, and describe the environment it runs in as best as possible. Or just make fewer assumptions about the environment.
This is where we use a lot of Docker. The learning curve is steep, it’s not always easy, there are trade offs. But it forces you to think about your env, and the versions of your code are already nicely contained in the image.
(Another common path is unpacking in subdirs of a ‘versions’ dir, then having a ‘current’ symlink you can swap. I believe this is what Capistrano does, mentioned in the article. Expect trouble if you’re deploying PHP.)
I’ll also agree with the article that you should be able to identify what you deploy. Stick something produced by git describe in a ‘version’ file at your package root.
Maybe I’m missing a lot here, but I consider it project specific details you just have to wrestle with, in order to find what works. I’ve yet to find a reason to look into more fancy stuff like Kubernetes and whatnot.
Going through Thorsten Ball’s book about writing an Interpreter in Go
Hey I’m reading this too! :) Currently on chapter 2
Ok, I’ll ask a stupid question. What does a great deployment pipeline look like?
It depends on what you’re trying to deploy and what constraints you have; there isn’t one magic bullet. One pipeline I was especially proud of for a Python app I wrote at Fog Creek worked like this:
1a2b3c4d. We used Mercurial, not Git, so the command washg archive -t tbz2 -R /path/to/bare/repo -r 1a2b3c4d, but you can do the same in Git./srv/apps/myapp/1a2b3c4drequirements.txt, make a new virtualenv if necessary in/srv/virtualenvs/<sha1 of requirements.txt>on each server hosting the app./srv/config/myapp/1a2b3c4d. Configs are generally stored outside the actual app repo for security reasons, even in a company that otherwise uses monolithic repositories, so the SHA here matches the version of the app designed to consume this config info, not the SHA of the config info itself. (Which should also make sense intuitively, since you may need to reconfigure a running app without deploying a new version.)1a2b3c4d.myapp.server.internal, that servesmyappat revision1a2b3c4d.default.myapp.server.internalto point to1a2b3c4d.myapp.server.internaland rerun tests.default.myapp.server.internalback to the old version.Now, that’s great for an app that’s theoretically aiming for five-nines uptime and full replacability. But the deploy process for my blog is ultimately really just
rsync -avz --delete. It really just comes down to what you’re trying to do and what your constraints are.I doubt you’ll find consistent views, which makes that the opposite of a stupid question.
My ideal deployment pipeline looks something like the following:
Note that I’m coming at this from the server-side perspective; “deploy into” means something different for desktop/client software, but I think the overall flow should still work (though I’ve never professionally developed client software, so I don’t know for sure).
We do:
hg archive -r 1a2b3c4d build/docker build -t $(JOB_NAME):$(BUILD_NUMBER) .and push it to internal docker registry.sed -e "s/@@BUILD_NUMBER@@/$(BUILD_NUMBER)/g" $(JOB_NAME).nomad.sed >$(JOB_NAME).nomad.Nomad will handle dumping vault secrets, config information, etc from the template directive in the config file. So Configuration happens outside of the repo, and lives in Vault and Consul.
You can tell by the env. variables we use Jenkins :) different CI/CD systems will have different variables probably. If unfamiliar with Jenkins, BUILD_NUMBER is just a integer count of how many builds jenkins has done for that job. JOB_NAME is just the name you gave it inside of Jenkins for this job.
This is way off topic, but I’d love to hear why you went with Nomad and how it’s been working for you. It seems to fill the same niche as Kubernetes, but I hear practically nothing about it—even at shops using Packer, Terraform, and other Hashicorp products.
We started with Nomad before Kubernetes was a huge thing, i.e. we heard about Nomad first. But I wouldn’t change that decision now, looking back. Kubernetes is complicated. Operationally it’s a giant pain. I mean it’s awesome, but it’s a maintenance burden. Nomad is operationally simple.
Also Nomad runs things outside of docker just fine, so we can effectively replace supervisor, runit, systemd, etc with Nomad. Not that I remotely suggest actually replacing systemd/PID 1 with Nomad, but that all the daemons and services you normally run on top of your box can be put under nomad, so you have 1 way of deploying, regardless of how it runs. I.e. Postgres tends to work better on bare hardware, since it’s very resource intensive, but with the Nomad exec driver it runs on bare hardware under Nomad perfectly fine, and gives us 1 place to handle logs, service discovery, process management, etc. I think maybe the newer versions of Kubernete’s can sort of do that now, but I don’t think it’s remotely easy, but I don’t really keep up.
But mostly the maintenance burden. I’ve never heard anyone say Kubernetes is easy to setup or babysit. Nomad is ridiculously easy to babysit. It’s the same reason Go is popular, it’s a fairly boring, simple language complexity wise. This is it’s main feature.
Thanks for the write up! Definitely makes me want to take another look at it.
Could you elaborate on this step? This is the on that confuses me the most all the time…
Inside of Jenkins job config we have an ENV variable called MODE and it is an enum, one of: dev, test, prod
Maybe you can derive it from the job-name, but the point is you need 1 place to define if it will run in dev/test/prod mode.
So if I NEED to build differently for dev, test or prod (say for new dependencies coming in or something, I can.
That same MODE env variable is pushed into the nomad config: env { MODE = “dev” } It’s put there by sed, identically to how I put the $(BUILD_NUMBER).
And also, if there are config changes needed to the nomad config file based on environment, say the template needs to change to pull from the ‘dev’ config store instead of the ‘prod’ config store, or if it gets a development vault policy instead of a production one, etc. I also do these with sed, but you could use consul-template, or some other templating language if one wanted. Why sed? because it’s always there and very reliable, it’s had 40 years of battle testing.
So that when the nomad job starts, it will be in the processes environment. The program can then, if needed act based on the mode in which it’s running. Like say turning on feature flags under testing or something.
Obviously all of these mode specific changes should be done sparingly, you want dev, test, prod to behave as identically as possible, but there are always gotchas here and there.
Let me know if you have further questions!
Thank you very much! Helps a lot!
I do a “git push” from the development box into a test repo on the server. There, a post-update hook checks out the files and does any other required operation, after which is runs some quick tests. If those tests pass, the hook pushes to the production repo where another post-update hooks does the needful, including a true graceful reload of the application servers.
If those tests fail, I get an email and the buggy code doesn’t get into production. The fact that no other developer can push their code into production while the codebase is buggy is considered a feature.
Since I expect continuous integration to look like my setup, I don’t see the point of out-of-band testing that tells you that the code that reached production a few minutes ago is broken.
I think that the point kaiju’s thread wants to make is that you shouldn’t be deploying from your local machine, since every developers environment will differ slightly and those artifacts might cause a bad build when sent to production. I believe the normal way is to have the shared repo server build/test and deploy on a push hook, so that the environment is the same each time.
The setup we use is not even advanced, but simply resilient against all the annoyances we’ve encountered over time running in production.
I don’t really understand the description underneath “the right pattern” of the article. It seems weird to have a deploy tree you reuse everytime?
Make a clean checkout everytime. You can still use a local git mirror to save on data fetched. Jenkins does this right, as long as you add the cleanup step in checkout behaviour.
From there, build a package, and describe the environment it runs in as best as possible. Or just make fewer assumptions about the environment.
This is where we use a lot of Docker. The learning curve is steep, it’s not always easy, there are trade offs. But it forces you to think about your env, and the versions of your code are already nicely contained in the image.
(Another common path is unpacking in subdirs of a ‘versions’ dir, then having a ‘current’ symlink you can swap. I believe this is what Capistrano does, mentioned in the article. Expect trouble if you’re deploying PHP.)
I’ll also agree with the article that you should be able to identify what you deploy. Stick something produced by
git describein a ‘version’ file at your package root.Maybe I’m missing a lot here, but I consider it project specific details you just have to wrestle with, in order to find what works. I’ve yet to find a reason to look into more fancy stuff like Kubernetes and whatnot.