The New Complete Joy Of Devops

There are a few areas in software engineering that are a bit odd. Everybody encounters them and needs to solve them. All the building blocks are there. The best practices are known. Yet the industry hasn't converged on a solution that can be packaged and reused easily. Authentication and authorisation is one such area. Devops – the thing that turns something that runs and looks great on your workstation into a maintainable, reliable service – is another.

Full disclosure, I hate devops. It doesn't agree with me. It's the iteration cycles. You can only test it in the actual environment that runs it, and it takes minutes just to spin up a pipeline. You change a configuration, validate it, run it, it fails, you fix it, you run it again, it fails at the next part, until it stops failing, and each of these iterations takes anything from five to fifteen minutes. It's complicated, messy, and above all slow.

It took nine such iterations to get the Tarinoi release pipelines working like I wanted. I hated every minute of it, but it was so worth it. Fair warning - not a lot of this blog post will make sense if you haven't wrestled with this kind of thing before, so proceed with caution. But if you have, you might find this interesting.

It's all about the ducts and the pipes, isn't it?

What the Tarinoi release automation does ​

There's a lot of activity that goes into cutting a release. This is what happens when we want to release version 0.9.4:

1. Merge everything we want in the release to the main branch if it's not already there.
2. Make a branch named release/v0.9.4 from main
3. Update the version number everywhere it appears in the codebase.
4. Convert the user-facing release notes we've kept in release_upcoming.md to update-info.json which the UI can format and display to the user, and reset release_upcoming.md to a blank template, ready to use for the next release.
5. Build and deploy this to the staging environment.
6. Verify that everything works as it should, rehearse a rollback, rehearse any database up/down migrations that may be needed, and so on. Anything that needs to be fixed goes into release/v0.9.4.
7. Promote the Docker images from the staged release to production in the file that keeps track of which images are where.
8. Tag the commit.
9. Merge back to main.
10. Run any database upgrade script in production.
11. Roll out the new images.
12. Smoke test production to make extra sure nothing broke.

Some of these steps need to remain manual – you can't automate QA, and a manual verification step is needed on the staged release. The busywork can be automated, and should be – that's where human error creeps in, and we want the process and automation to catch us when we trip up, rather than being a source of errors in its own right.

How we do it ​

So, we've been able to condense this to two scripts that we also run in CI:

1. release:stage – this performs steps 2-5 of the above list. After it's run, we have the new version in the staging environment, we can go there, see how the release notes appear to the user, check that everything works as it should, rehearse rolling back to the previous version, and do anything else we need to do to make sure it's ready to roll out. Anything we need to fix, goes into the release/v0.9.4 branch the script created.
2. release:ship – this is run on release/v0.9.4. It runs steps 7-11 of the list.

The only significant wrinkle is that we should hold off merging anything to main while we're still verifying a release on staging; if we do, we'll have to pull those changes into release/v0.9.4 manually, redeploy to the staging environment, and re-verify, before running release:ship.

What it looks like ​

When we want to cut a release, we click one button to stage, check that it worked, then click another button to ship. If a test fails, a build breaks, something appears in main when it shouldn't be ahead of release/v0.9.4 or any of a pretty large number of other conditions that should block release triggers, the pipeline will fail and we'll see why so we can fix it. It's pretty rigorous about stopping us from goofing off late at night.

We might even dare to release on a Friday, one of these days.

What it requires ​

This sounds complicated enough – and it is, there's a lot of scripting behind those two buttons – but for this to work, a fair few other things have to be in place.

• Tarinoi is a monorepo. We have a single Git repository that contains the whole shebang, all the source, all the automation, as a pnpm workspace, released under a single version number.
• We have a pretty smart set of build scripts. We don't build all the Docker images every time, only the ones that changed, and we generate files that keep track of which image is deployed where. That's how the promotion mechanism works.
• It's easy to update the release notes – just one .md file in a standard template – and we have a merge request template that reminds us to do it, and refuses to merge if you've marked a MR as user-facing but haven't updated the release notes.
• We keep track of what scripts have been run on the database, so it doesn't re-run a migration that's already been done, and knows what downgrade scripts need to run in case we have to roll back.
• We deploy to Kubernetes. In and of itself it's very powerful, and has features like seamless rollouts, you can upgrade the service with no interruptions so the users don't even notice.
• We use Helm – the industry-standard tool for managing more complex Kubernetes setups. We have a single Helm chart with a set of params for each environment, and we can use its jobs, upgrade, and downgrade features to do what we want to do.

So why is it so hard? ​

There's nothing particularly groundbreaking about our devops process and automation, and I'm not pretending to reveal any great innovations (or release any business secrets) in this fairly detailed blog post. Some variant of this pattern is widely used. It supports different flavours – we can do change-based releases (every time we finish something, we deploy it), calendar-based releases (we release what we have every week), or we can do feature-set based releases (we plan what goes into a release, finish that work, test it, and roll it out). So why is it so hard?

The difficulty is that the way we've set up Tarinoi isn't the only way to set this up. You can do more or less the same thing with Jenkins, Bamboo, TeamCity, GitHub Actions, GitLab Pipelines, or more. You can run Kubernetes in Amazon, Azure, Google, Scaleway, or any of a ton of other providers. All of these do more or less the same thing and support more or less the same tools, but they're all different in the specifics.

Then there are team differences – we have one metadata file for the upcoming release's release notes; a bigger team might want a separate file for each change. Right now we only have staging and production targets, but as we get closer to closed beta, we'll add a different environment for development and testing. Maybe we'll add a canary target, or want to do silent A/B testing. All of this adds even more variety and complexity.

So your one-size-fits-all turnkey solution is suddenly not so one-size-fits-all, it only works on the specific combination of platforms, tools, team practices, and deployment targets – the one we've happened to pick. Trying to make it work on any Kubernetes host, any secrets provider, any build automation platform, with any of these variants blows it up all over again. Maybe somebody will solve this one day, like we're solving one messy corner of gamedev with Tarinoi's process tools.

Somebody other than me though. I hate devops.