Non-Geek Guide to Continuous Software Delivery

Continuous delivery enables quick iterative releases of new features and bug fixes to your customers and, at the same time, minimizes the risk of making deployment mistakes, introducing new bugs and regressions. If you struggle to keep up with ever-multiplying bugs or you keep your fingers crossed for good luck before each release, you might need to rethink the way you build and deploy your product.

This article aims to help non-technical people like non-technical product managers, project managers, and executives to support their engineering teams on the path to building better high-quality products, keep high delivery pace and avoid unpleasant surprises for your valued customers.

Photo by Jannes Glas on Unsplash

How to continuously deliver?

The continuous delivery process starts with keyboard strokes of your engineering team and ends with the experience your customer has with the product. The goal of continuous delivery is to shorten the time and avoid mistakes between these two points in time. A well-oiled mature continuous delivery system takes deployment management burden off your engineering team, allowing them to focus on engineering and product build. As a consequence, a continuous delivery system decreases the on-boarding time for new joiners allowing them to start contributing within the first few days if not hours.

While there is a great variety of different continuous delivery approaches, you can split all involved activities and steps into the following:

1. source code management,
2. automated and manual code quality control — including verification of your unique functional, performance, security and compliance requirements (PCI, HIPPA, etc.),
3. stage-by-stage deployment with automated and manual acceptance testing, and
4. production deployment and continuous monitoring.

Store your code in the code repository

The very first and most important step towards continuous delivery is a centralized place to store and share code. Code is the key (and sometimes the only) outcome of software engineering activities, and this code needs a place to live. This place is a common and shared environment that everybody on your engineering team knows and uses every day. You can think of a code repository as a Google Drive or Dropbox for engineers to share and work on the code. A great place to start your journey to build continuous delivery is to introduce a code repository in your organization if you happen not to have one.

The code repository of choice is most often git available through providers like github, gitlab or bitbucket to name a few [1]. These providers offer powerful free tiers that will serve you for a long time until you will see a need to pay for premium features. Check-in with the team if they have any preference or prior experience before choosing any specific platform.

The most common way to approach your code management is to have one code repository per product (or a piece of product) that has its distinct life cycle. For example, backend code will have one repository, a frontend website will have another, admin and a metrics dashboard will be in a third repository. Take it is a rule that every piece of code should have its repository — starting from frontend and backend code, and ending with database schema code, serverless functions, and infrastructure definition.

Setting up a code repository might seem like a waste of precious time especially in a startup environment. However, look at it as a high ROI investment and major mitigation for various operational risks. Consequences of not having centralized storage for code artifacts can vary from minor things like a decrease in productivity and slow knowledge sharing to more disastrous events like loss of your entire products should your engineers’ laptops break or get stolen.

The code repository is an integral part of any software development process and should be adopted by any technological company early its days. The code repository makes it possible for engineers to store and share code, track revisions, and incoming change requests.

Review, inspect and test every product change

All changes to your products are made by humans, and humans are prone to making mistakes — which is not bad, but rather how things are. And while we can all ask our colleagues and teams to write bug-free code — bugs still do slip through the cracks. To minimize the risk of getting bad product changes into our customers’ hands we can introduce a couple of additional hops that the change needs to jump through. First, we can employ computer programs (automated tests and various code analysis tools) to be impartial judges of the quality of changes we introduce. Second, other people on the team might be able to spot issues or deficiencies in the change the author overlooked (manual review process). Both of these steps ideally would be triggered automatically on every code change.

Most of code repository providers (including github, gitlab and bibucket) provide tools to build continuous delivery pipelines. A pipeline is a set of predefined steps that run automatically following a particular event (code commit, time schedule, manual trigger). An example of a pipeline definition could be “on every commit, build the product and run automated tests on it”. Configuring and streamlining pipelines will require some time and iterations but will quickly bring the reward of stable products and happy customers.

One of the options how the process of accepting new code change into repository can look as following:

1. Submit new code change. The author of a code change submits a change request to the code repository. At this stage, code is placed in a separate “sandbox” (also known as a pull-request, or change-request). This is done to prevent potentially harmful changes to making their way into the main code base unverified.
2. Trigger automated tests. On changes submitted to the code repository, pipeline executes tests that validate basic functionality, compliance with code style, and security standards. If the pipeline fails, it notifies the author of spotted errors and waiting for the author to provide a fix.
3. Open a code review request. At the same, either automatically after smoke tests or on author request, the code management system opens a code review request where teammates can check out the change candidate, leave comments, improvement suggestions or ask questions. Once all involve satisfied and approve a request — change can make it to the next stage. 1–2 people to look through the code is usually enough.

Once steps above are completed, your code is ready to be merged to the main codebase where it will slowly progress through a series of stages and additional verifications.

Early validation of changes avoids costly impact on your customers. Setting up gatekeepers prior code merged into the main branch helps ensure that at any given moment your main code base is stable and functional. One of the additional benefits of having such gatekeepers is to allow engineers to work on different changes simultaneously and not stepping on each other toes. Each new incoming change is an increment on top of a stable and tested product.

There are countless articles on how to build your manual review process as well as on how to build test automation. Below are some articles that I would recommend as a starting place to dive deeper into the topic:

1. How to do a code review — Google’s Engineering Practices
2. How to Make Good Code Reviews Better — The Overflow
3. Level up your code review with continuous integration — Bitbucket

Deploy changes stage by stage

Your code change is now ready and finally merged into the main codebase. It might be tempting to get and deploy your change right to production. However, you run a risk of having a product that has functionally correct but either fails to communicate with its downstream dependencies (database, other services), or due to miscommunication it is not what you expected to see. For these reasons, you can decide to introduce a multi-stage deployment that will first deploy your code internally — for you and your team to have a look, and then externally — for the public. In its minimal setup, it translates into two stages — a development stage and a production stage.

The development stage is an environment where your product is similar way and structure as in production, with a difference that it doesn’t process production data, and only you and your team have access to it. In the development stage you can run integration tests to verify integration with downstream dependencies (databases, metric aggregators, etc.), run automated user experience tests to automatically check that all buttons, links, and fields are functional and behave as expected, as well as to try out the product by clicking through it yourself — perform manual testing. In the case of the customer portal, for example, the development stage would be a website with restricted access, in the case of an app — development stage would be a draft version of an app available through beta testing tools (like TestFlight for iOS).

The production stage is the final stage where customers interact with your product. The goal of the whole continuous delivery process is to ensure that as little bugs as possible end up here. The production stage works on production data, integrates with other production services, and serves real customer traffic.

While at least two stages can be considered as a minimal setup, depending on the type of the product you build (SaaS APIs, licensed software, software for hardware components, customer portals, machine-learning algorithms, etc.) you might need to have a broader set of stages. Here is a non-exhaustive list of possible options:

• Development stage — for your internal testing and validation
• Testing stage — to allow your customers to integrate with it in their test environment (relevant for SaaS companies)
• Performance Testing stage — to run load testing and measure the performance of a system similar in setup to production
• Canary stage — to deploy release candidate and divert production traffic partially to it. This will allow you to get a sneak peek preview of how new change will behave in production without impacting 100% of your traffic [2].
• Production stage — your final stage that your customers are continuously interacting with.

All major code repository providers have pipeline solutions to enable you to configure multi-stage deployments. Some will have rich pre-defined workflows and integrations, others will give you just bare basics to build on top of them. In addition to code repository providers there are solutions focused solely on continuous delivery — for example, circleci, travisci, etc which can help you solve some unique problems like mobile testing, custom approval workflows, and others.

There is no-one-size-fits-all pipeline configuration, so work with your team to figure out what stages, transitions, and notification you need and how to best configure and run your unique pipeline.

Monitor, rinse and repeat

If you have your code change deployed automatically, but then you manually go to the browser and nervously check if production is still up and running you can say you have a Schrödinger’s Production (analogous to Schrödinger’s Cat [3]).

With almost any kind of product, you would want to continuously monitor its status, performance, and other metrics that are important for your business (conversion rates, latencies, resource consumption, etc.). Monitoring approach and a specific set of metrics will largely depend on the type of product you deliver (SaaS, IoT, mobile applications, etc.). The main point is that in order for you be able to proactively react to outages and issues with your product you would need some minimal set of health checks that will trigger alarm notifications to you and your team. Last thing you would probably want it your customers reaching out to you notifying that your product is out of order.

An example of a basic health check for your backend server is a periodic ping from a monitoring system to your production server that expects a successful response. For a mobile application, it could be analytics platforms that provide you an aggregated view of your customers’ activity. Big unexpected deviations in customer activity might be an indicator of issues the newly released version.

There is plenty of metric aggregators and other solutions that can help you streamline your monitoring. For example, datadog provides beautiful easy-to-configure dashboards with countless integrations for your cloud and mobile needs; or CloudWatch for those who’d like to stick to AWS. Again, when picking a tool, ask around and check-in with your team if they already have some experience with any of the monitoring platforms. This could potentially save you a lot of time finding and onboarding the right solution for your needs.

Having said all that, issues do get sneak into production unnoticed despite our best efforts. In anticipation of that, you might decide to give your customers a quick and easy way to report problems with your service — a feedback form, a “report problem” button, etc.

Visibility into your product operations is an essential part of continuously delivering stable and reliable software products. A well-configured monitoring system will bring you peace of mind and will save precious time should things go south.

Why do I need all that?

Let’s face it. Building products is hard, and what is harder is to keep the pace up when product complexity grows. Through research and analysis, it became apparent that leaving the quality of your code unattended can bring disastrous consequences to your team and to your business as a whole. Without proactive measures to continuously keep the code quality up the amount of maintenance effort grows, while the speed of delivery of new features goes down. Investing time to automate your testing and deployment processes as well as dedicating time to removing technical debt is rather a necessity than an option. Check out a nice talk by Asa Schachar on the topic of technical debt and how to proactively tackle it.

Companies and startups too often fall into the trap of fast delivery and short-cuts forgetting to balance speed with sustainability. You might have heard the phrase “if it works, don’t touch it” that usually becomes a motto of teams that didn’t prioritize continuous delivery and automated validation in their development process.

Investing time in the continuous delivery system will make your customers and your engineering team happier by increasing the stability of the product and increasing confidence in introducing new changes. Tailored and well-functioning delivery system will have your back even if you happen to make a mistake.

This article doesn’t aim to provide a definitive guide to continuous delivery and continuous integration, but rather its goal is to help non-technical leaders or those starting on a technical path get an introduction into the software delivery process. Continuous delivery concepts are important to understand for everybody in the organization — including technical and non-technical leaders. It allows the team to synchronize around basic principles and constraints of software delivery and properly manage expectations.

While every company and every organization is different in ways they build and deliver products, introducing continuous delivery will bring immediate benefits to your team and your customers, and will help ensure the long-term sustainability of your delivery processes and your product.

I would love to hear from you about your experience of setting up continuous delivery in your organization and what issues you have faced on that journey.