I don’t like their solution, but indeed, every company working with python needs to reinvent the deploy flow. It gets really hard when:
a. Github is down
b. Requirements break down
c. Machines are autoscaling while you deploy
The approach in this article isn’t my favorite. An rsync based approach is faster. Also this post doesn’t handle the most difficult part, which is dealing with autoscaling or temporarily unavailable instances.
Does anybody know a good open source solution for running deploys of python projects running on AWS?
I’m curious, how do you deploy with rsync in a way that mimics what the article describes?
At $DAYJOB, we have Chef scripts which check out all of our Python infrastructure (uwsgi, et al.) onto a fresh instance. We mirror the PyPI and apt packages we use, to speed this up. A tag is generated and specified for the chef scripts, and they check it out into the appropriate location on the server. When we deploy a new version, we spin up a cluster using this automation, put it behind the ELB, and take out the old ones.
This is how I handle our Python deployments at my day job, and it works rather nicely.
We do a couple extra things on top of this, which gives us good results.
We execute the package build in a Docker container. This is just a way to get around the fact that our CI provider doesn’t offer a choice of build host image: it’s Ubuntu or the highway. Since we deploy to Debian stable, wrapping the build in Docker gives us consistency across the build and deployment environments. Added bonus: you can build the package locally for testing and expect it to come out the same as when it deploys.
We keep the build results in a Debian repository in S3, managed via deb-s3. We use apt-transport-s3 and IAM roles to grant our hosts access to the built artifacts. This is not a huge deal, but it means you can rely on your orchestration software’s abstraction over repositories and packages for deployments.
We set the Debian package version to "$(python setup.py --version)+$(date +%s)". This gives us monotonically increasing versions, making sure the latest version is preferred for upgrades, but also gives us a direct translation back into a timestamp if we need to roll back.
"$(python setup.py --version)+$(date +%s)"
We also have some Rails applications, which we deploy using pkgr for a similar experience.