It’s always nice to see something you have spent time developing being used in production.
A while ago I read an excellent blog post from Timothy Fitz called Continuous Deployment at IMVU: Doing the impossible fifty times a day, that got me started on trying to improve the way we deploy our websites at UKFast but just as important – being able to roll back to a previous version at the drop of a hat.
We had a few requirements such as, hook into current systems, deploy code with near zero downtime, run some post deploy commands, stop deploying on error and keep an archive of old deployments for simple roll back amongst other things.
As you’ve probably gathered by the title, we run Subversion as our version system, it works well as a standalone but that’s never enough for us at UKFast. The system that was developed so far has achieved everything that we need, new or existing sites are deployed usually within 30 seconds but that depends on the size of them, the freshly deployed code is activated online without any loss of service to visitors to the site which is probably one of the best features about this system. It also means that roll backs are done transparently and due to another feature, archives, depending on the (user configurable) number of archived versions we keep, we can roll back to a previous version almost instantaneously as the code is already deployed and ready to go – if we need to go back further past the archived version then a checkout is done on the old version.
Now once a site is deployed, that typically wouldn’t be the end of it. If someone had tried to perform the update manually then it most likely wouldn’t be case of just FTPing up the new website, for one, we usually have some heavy customisation on folder structure, permissions or just because we want something different. So the system needs to cope for this as well, it will allow users to issue post deploy commands from a predetermined list of allowed options. This has so far covered us from all the usual requests our web developers usually ask for and without have to give them a ssh shell!
In the event that any errors are detected at any point the deployment is halted and the user performing the update is notified, no need to roll code back in anyway as the new version hasn’t even gone live yet.
This system has meant we can stop using ssh / ftp accounts for website management, determine fine granular access to developers and mainly, allow us to update our websites at any stage of the day, knowing that it won’t impact site visitors and if it does, then we have a roll back process that takes us 10 seconds to perform and the site will be restored.
Unfortunately the way in a lot of our systems work means I can’t discuss it on an open blog, competitors might be watching. But if anyone wants to pick my brains then by all means, get in touch.
Hi Neil,
This is interesting, but I’m wondering how you deal with significant schema changes that need to go out? As noted in Timothy’s article, their system isn’t much more agile than anyone else when altering a table with millions of rows, and I’m wondering if you’ve done anything interesting there.
Thanks for the article.
–Bruce
| November 22, 2009 @ 01:45
Hi Bruce,
At the moment we are lucky enough that we don’t actually store a lot of data in our DB’s but we do rely on them heavily, changes to the schemas are done on a dev system first, then rolled out to a test environment before being deployed into live systems – all done manually.
If I had to implement a system to automate the deployment for DB schema changes then I would still have changes made to the dev systems done manually. Both the testing and live DB’s would be load balanced with at least two master db servers behind each VIP. When the system marks the updated schema for release, a db server behind the test VIP would be pulled out of service, the schema updated and then a test copy of the site or application would be tested automatically against the updated schema. If any tests fail then the db server is kept out of service to be either fixed, have it’s schema altered back or have the data resynched from another server so it can be brought back into service. If the tests succeed then the db server is put back into service and the remaining db servers are removed for the schema to be altered. Rinse and repeat for production db servers. You’d have to keep a track of what schema version each db server is currently running without replicating that info out to all of the other db servers but that should be simple enough.
I’d love to be able to have a crack at this but our usage just doesn’t warrant the work involved at the moment
| November 22, 2009 @ 20:05
What’s the basis of your automated deployment system? Ant or something more continuous integration oriented like Hudson, CruiseControl, etc?
I’m looking to take the first step with my development team, i.e. simply automate an FTP upload based on a subversion export of a certain tag (i.e. release-1-1) or a branch for bug fixes (i.e. maintenance-1-1). Of course, it should only deploy changed files/folders, not everything.
I like the looks of the CI stuff, but I think something more simple like Ant would be a good starting point, while I/we learn the ropes. Any thoughts/comments?
| February 10, 2010 @ 00:50
Hi Scott,
Firstly apologies for not authorising your comment for so long, it went unoticed in all the spam
Our system is built entirely in Perl except for the integration with subversion for checking out code. It sounds like you are going to use the hooks built into sv, we considered that as well but wanted to avoid the use of ftp or sftp and also have a bit more flexibility than it could provide. If it works for you though then don’t let me know persuade you differently
If you are going to deploy changes / updates, what happens if you want to go back 4 versions, will you be able to do this or would you have to rely on this process being done manually? I highly recommend the automation of deployment, it makes so much sense and allows a lot more time to be spent on the development / coding aspect of things rather than how and when you will update your site or code.
Neil
| March 1, 2010 @ 22:09