We achieve stable, near-continuous deployment at Charlie by following a basic development flow. The idea is simple: incrementally improve our product, deploy frequently (several times a day!), and be process-driven to achieve a stable service with a lightning-fast feedback loop.
- Stress-free deployments: Anyone can deploy, at any time
- Stable codebase: Everything is reviewed by peers and business and tested before it’s merged into master
- Fast user feedback loop
- All developers familiar with majority of codebase
We build software similar to how GitHub builds software, but we make a few significant changes that we feel are incredibly important:
- Test & review changes in the QA environment before merging to master
- Continuously deploy master to Staging
The Development Flow
- Branch Master
- Code & write tests
- Deploy to a QA environment
- Open a Pull Request
- Peer code review (unless you pair)
- Business review & discussion (as needed)
- Merge pull request
- Continuous deployment to Staging
- Smoke test your changes
- Deploy to Production
- Master is always stable & deployable
- Deploy frequently, incremental changes (several times a day!)
- Anyone can deploy to production
- Thoroughly test your code (unit / integration / acceptance)
- Automated deployment script
- Continuous Integration on all remote branches
1. Branch Master
Create a branch from master for all changes you make (features and bug fixes). Give it a friendly name, like “company-news” or “fix-twitter-date-error.”
2. Write Code / Write Tests
Regardless of how you get there, adequately test your code. This includes a healthy mix of unit tests, integration tests, and acceptance tests. Ensure you take into consideration the different scenarios and use cases for your features. That’s all I’ll say about that because this is a topic beyond the scope of this post. I’m not going to ignite a TDD flame war, so watch this series for that debate: Is TDD Dead?
3. Deploy to QA
Here’s the missing key to the GitHub flow.
When you’ve wrapped up development, deploy your branch to QA for review and testing. The QA environment represents production plus your changes. The main goal here is to catch any issues and ensure things are implemented as expected before the code is integrated with master. Otherwise, you’d have master in a non-deployable state.
As a developer, you’ll uncover deployment issues before it hits master. For some reason every time you go from local to a hosted environment, you discover issues. May as well find then before they pollute the codebase!
Run a single command to deploy to QA:
ey deploy -e qa
Our QA environment is completely isolated from other environments, and is similar to production, but not identical. We have one web server, one Sidekiq worker, a separate MongoDB, and Redis. We have four developers on our team and one QA environment suffices. For a larger team, you may want to have additional QA environments.
Open a Pull Request
Open a Pull Request to start the conversation. Document your changes or simply link to the card in Trello. If necessary, list out instructions for how to test or reproduce, and include anything noteworthy like design decisions. Include screenshots of your new functionality from your QA environment using business-relevant data. Make it dead simple for the reviewer to see what has changed by annotating the screenshots if necessary (we use Skitch to do that).
Your Pull Requests become an amazingly detailed change log, complete with a writeup, screenshots, and every line of code that’s changed, with discussion.
Peer Code Review
Every line of code changed is reviewed by a fellow team member. Any team member, except for new team members, performs code reviews. If you pair program, you have automatic code reviews built in, so this step isn’t necessary. It’s important that you have everyone on the team review code so that:
- All team members are familiar with the entire codebase
- Junior team members learn from how more senior members code
- No bottlenecks if you have only the senior members code review
For new team members and especially new junior developers, have a senior team member pair up and review the code with them to educate them on how to properly review code. Do this until you feel they’re ready to fly.
In QA, the business folks can try out your new functionality before it’s integrated into master and give you fast feedback (all documented in the pull request). Sure, you’re UX person created beautiful mockups, but sometimes when you actually play with the feature, it just doesn’t feel right. It’s much easier to make those tweaks before the developer has shipped the feature and moved onto something else. Once it’s live, you’re more likely to work on something else and push off those minor tweaks until never.
Merge Your Pull Request
Once business has signed off (only when necessary), code review checks out, and the build passes, it’s ready to ship. The developer who opened the pull request closes the pull request. This is important because it allows the developer to put on any finishing touches and keeps her in control. Of course, if you make significant changes, ask your developer for a quick follow-on review of the final commit. Delete your branch locally and off the server once merged (don’t worry, Github always keeps a copy).
4. Verify in Staging with Continuous Deployment
Your staging environment represents what’s about to go to production. It’s not too far off, because it’s only ahead by one or a few pull requests at the most. The staging server configuration represents a very close (but scaled down) version of production. reference an article The build server continuously deploys master to staging after all tests have passed.
Verify in Staging
Before you deploy to production, do a sanity check in staging. Poke around with the new functionality and make sure everything’s as it should be. The real goal is to make sure that the integrated changes will work as expected in production. You want to catch any deployment issues before they’re live. You already have tests in place to ensure existing functionality remains in-tact, so you don’t need to go crazy and test features that are already live. This is just a precaution, but it’s a really good idea, especially for larger changes.
We keep track of what’s sitting in staging in a separate list on our Trello board, so you know what you’re about to deploy.
5. Deploy to Production
Here’s the fun part. No more stress-induced, ceremonial deployments. When it’s time to go live, any developer on the team runs a simple command that automatically deploys the code from Github. Trust the process and feel confident that your deployment will go fine.
“Software release can–and should–be a low-risk, frequent, cheap, rapid, and predictable process.”
- Martin Fowler, Continuous Delivery
Here’s our deployment command:
ey deploy -e production
Deploy during the day
We have our environment configured for zero downtime, so we can deploy multiple times a day, during the day. It’s important that you deploy during the day as opposed to at night. Why? During the day everyone’s sharp and on-point. There’s also time to recover if things go wrong. If you deploy at night, you may go to sleep and wake up to down alerts. That’s no fun!
We build Charlie following this development flow. We’ve had zero downtime in the last 6 months, largely due to this process. It’s a simple process that uses multiple environments to deliver software, where each is more stable than the last (Local > QA > Staging > Production). The important thing here is every small change is peer code-reviewed and reviewed by business (as needed) before it’s integrated into master. We use Continuous Integration to ensure all of our tests (we have thousands) pass before deploying any change. It’s an amazing feeling to deploy to production and not feel in the least bit anxious about something going wrong.
What do you think?
I’d love to hear your feedback. What do you think about our process? How do you deliver software?
GitHub Flow, Scott Chacon, 8/31/2011
How GitHub Users GitHub to Build GitHub (video), Zach Holman, 9/21/2011