Covert Operations - deploying applications without anybody knowing
Before we get going, what’s been going on since last time?
- We’ve increased the number of AWS environments we have from 1 to 5, all built with Terraform and a 6th is going up shortly!
- Implementing Packer into our environment pipeline is well under way
- Tiago discovered unicorn-flavoured pretzels are a thing :o
When/how do you deploy your application?
Do you deploy application releases manually? Whilst you're deploying, does it cause downtime for your users? When is a good time to deploy that release to your production environment? Is it:
- 8pm on a Monday night?
- 1am on Saturday morning?
- 5pm every Friday just as everyone's leaving the office for the weekend? (if you do this, please let me know so I can slap you)
The answer is NEVER. There is never a good time. If you deploy during the day, your users get grumpy. If it's out of hours, your team get grumpy.
So how do you deploy your application at a convenient time for you without users noticing?
ENTER BLUE/GREEN DEPLOYMENTS!
What are blue/green deployments?
When deploying the latest version of your application, users shouldn't know it's happening. They don't want kicking off the system, or to suddenly receive errors because there's a bug that wasn't picked up during testing.
Blue/green deployments allow you to test your application before switching users over to it. This seems like a good time for me to do some exceptional drawing:
In this example:
- Green refers to the existing environment, happily chugging away in production with all your users being routed to it.
- Blue is the exact same infrastructure as green, but with the new version of the application deployed.
A load balancer sits in front of all this infrastructure, making sure users only go to the green instance until blue passes testing.
The idea is all users are routed to the green instance. Hidden away is the blue instance, that only your technical team have access to. They will be testing it and fixing it if there are any bugs, then when they are happy with blue, all users are routed to it. Once there are no users using green, that instance is terminated. The instance will be no more. It will cease to be. It is an ex-instance!
It is important that blue's infrastructure is the same as green's - we don't want the added uncertainty of different infrastructure. That's not a problem though, because we're using Terraform!
Note that there are other deployment methods such as:
- A/B - having two versions of the same application running with one version being slightly different, such as a user interface change. Users are routed at random to one of the versions then the results are compared. This is useful if you're certain there are no bugs, but want to check if your changes impact user behaviour.
- Canary - routing only a small percentage of users to the new application, checking it's working for them, then rolling out to everyone. You're dipping your toe in the water, making sure no bugs bite your toe, then dive-bombing in once you're sure it's safe.
Be sure to think about what you want to achieve first, then look at the different deployment methods to understand which is best for your situation. Blue/green isn't for every situation!
Hang on...I've been typing all this time and haven't mentioned the cloud once?! Let's change that!
Blue/green deployments in the cloud
The cloud makes blue/green deployments a lot easier to manage. It's possible to route users from one set of instances to another in just a few button clicks, with the bonus of being able to automate the whole process!
A lot of cloud providers offer services to help make it even easier. AWS has CodeDeploy, Azure offers Traffic Manager.
Blue/green with AWS CodeDeploy
I won't provide a proper step-by-step of how to set up blue/green deployments in AWS as they're pretty good at documentation.
- Create an Application in CodeDeploy - giving it a name and selecting whether you're deploying to an AWS service (such as EC2 or Lambda) or on-premise (yes, ON-PREMISE! Your instances don't have to be in AWS for CodeDeploy to deploy to them)
- Create a Deployment Group to configure how the application will be deployed. Here you:
- Select blue/green as the deployment type
- Tell CodeDeploy whether to deploy to an Auto Scaling group (yaaaay, automation!) or to instances you manually specify
- Decide whether the load balancer should route traffic to the new instance immediately or if you choose when it happens
- Whether to terminate the old instances immediately or leave them running for an amount of time
- Choose a load balancer which will manage the traffic flow
There are additional options too, such as Triggers (e.g. CodeDeploy can send an alert to AWS Simple Notification Service, which in turn can send an SMS/Slack message/email when a deployment fails or completes) or Rollbacks (to automatically redeploy the last working release if the deployment fails).
And that's it for the configuration!
The final step is to create a Deployment. Your application needs to be stored in either AWS S3 (AWS' storage service) or GitHub, and the only change needed is adding an 'AppSpec' file to your application package. This file, written in JSON or YAML, contains instructions telling CodeDeploy how to deploy the application, start and stop it, along with how it can check the application is running as expected.
BUT...it's not perfect for everyone right now
You can't just throw blue/green deployments at your application and wonder why everything is on fire. Look at your release process and application first to make sure they're a good fit in terms of the application itself and your release process.
Release process - is yours chaotic?
Imagine you need to release your application right now - does that make you curl up into a ball? Are you crying? Maybe some of this rings a bell:
If you or your team say any of these things, then you should look at your release process first before thinking changing the way you deploy. Releases should be small, and applications decoupled from each other.
So, if your releases currently take hours, have lots of manual steps and all your applications depend on each other, you'll do more harm than good!
Application - is yours ready for blue/greenifying?
Is your application ready for this sort of deployment process? Start by asking:
- Does your application rely on sessions with no session management in place?
- Does your application rely on a database to such an extent that one tiny change to a table would stop your application working?
For sessions - if a user is in the middle of something on an instance that's about to be taken out of load, that user shouldn't notice that they've been switched over to a new one. There are different ways of handling session management. For example, the Panintelligence dashboard currently uses an Apache Tomcat server, so Tomcat's in-built clustering can be used for session management. More information can be found here.
For databases - if you have a database that the application heavily relies on, you can't make huge changes that would cause the existing version of the application to stop working. Changes should be rolled out gradually, but this should be the case for releases anyway!
- Blue/green can minimise how long your application is down during a release, or remove downtime altogether
- Most cloud providers offer an easy way to implement blue/green, e.g. AWS CodeDeploy
- Make sure you review your release process and how your application works before throwing a new method of deployment at it!