Move your VMware and KVM applications to the cloud without making any changes

  • July 15, 2014

Test automation and Continuous Integration & Deployment (CI-CD)

SUMMARY: Software is being adopted at unprecedented rates which is putting a strain on traditional software development processes. Test automation and Continuous Integration / Continuous Deployment (CI-CD) are part of the answer.

We live in the age of consumerisation of IT, which I tend to mentally distill as "the age of the apps". While traditionally IT products were sold to enterprises first and then later trickled down to consumers, since about 2008 that trend has reversed. There are now more than a billion smartphones in the world, which gives companies and indie developers a channel that allows for an unprecedented velocity in getting software adopted. While in enterprise software sales cycles and implementation projects could take years, an app is bought in a few minutes: search the app store, read a few reviews, and hit "buy".

This extremely fast adoption represent a tremendous business opportunity, but it also puts serious strain on software development. To be blunt, if you're not able to conceptualize, design, implement and deploy a new web or mobile app in a few months, there will be someone else that will. And once your app is deployed, a constant stream of improvements is what people expect.

In order to deal with these challenges, there's a lot of things that are happening. First, developers themselves are more productive than ever. There's exciting new languages, better toolkits, and a focus on craftsmanship that was not there just five years ago.

In this post though I will focus on a second, equally important part of software development efficiency: process automation. When code is written by a developer, it doesn't just end up in production by magic. There's a lot of things that need to happen, especially around checking its quality.

You can check out our Labs for continuous integration to see an implementation of process automation. You can also learn all of this by doing, using a ready-made blueprint to set up your Jenkins CI sandbox on Repo, our public blueprint repository.

The relevant terms here are "Continuous Integration" and "Continuous Deployment", often used together and abbreviated as CI/CD . Originally Continuous Integration means that you run your "integration tests" at every code change while Continuous Delivery means that you automatically deploy every change that passes your tests. However recently the term CI/CD is being used in a more general way to represent everything related to automation of the pipeline from where a developer adds his change to a central repository until that code ends up in production. The latter is also the meaning that I will use in this post.

The Software Development Pipeline

From a high level, a CD/CD pipeline usually consists of the following discrete steps:

  1. Commit. When a developer finishes a change to an application, he or she commits it to a central source code repository.
  2. Build. The change is checked out from the repository and the software is built so that it can be run by a computer. This steps depends a lot on what language is used and for interpreted languages this step can even be absent.
  3. Automated tests. This is where the meat of the CI/CD pipeline is. The change is tested from multiple angles to ensure it works and that it doesn't break anything else.
  4. Deploy. The built version is deployed to production.

The entire pipeline should be as automated as makes sense for your organization. Maximum automation would be that a single commit by a developer will automatically result in an update to production a few hours later.

There is some great tooling available for automating CI/CD pipelines, including hosted solutions such as Cloudbees and Atlassian Bamboo and on-premise software such as Jenkins and Teamcity.

Note that this pipeline model requires an Agile methodology where you always have a working version of your product in production, and only apply small changes to it that can be individually tested and deployed.

Automated tests

Without a doubt, automated test are the most important part of any CI/CD pipeline. Let me repeat that since it's so important: without proper automated tests that run fast, have good coverage, and no erroneous results, there can be no successful CI/CD pipeline. It's that simple.

The automated tests are usually divided into multiple "suites", each with their own objective. The list below gives a small overview.

  • Unit tests. This is the suite that is run first, often by developers themselves before they add their changes to the repository. Unit tests normally test individual classes or functions. When those classes or functions need access to external resources, those resources are often faked as "mocks" or "stubs".
  • Integration tests. This is the next level up from unit tests. Integration tests make sure that the modules that comprise an application work properly with each other. Ideally integration tests are run in environments that are similar to the production environment. For example, you'd want to make sure that if your application uses a database that the same database is available in your integration environment.
  • System tests. Should test the entire system in an environment as close as possible to the real production environment.

Note that because software development is such a young craft, there's sometimes different definitions that are being used for these phases Also there's other type of tests that are sometimes relevant like performance tests or user interface tests. Nevertheless, the three types of tests above (unit, integration and system) should form the basis for the vast majority of CI/CD pipelines.

Test Environments

In addition to a quality tests, it is also important to take care of the environments where they run. The following are important:

  • Clean environments. It is very important that your tests are run in clean environments, otherwise there may be interference between different test runs. Ideally the entire test environment is reset before a test run using whatever technology available, for example virtual machine snapshots.
  • Fidelity. In order for the test results to mean anything, they should run in an environment that is as close to your production as possible.
  • Ideally test environments can be spun up and down quickly and are allocated on-demand. This gives the lowest cost and the highest throughput. Hosting these environments in the cloud is therefore ideal.


I hope this blog post was a useful introduction to CI/CD. If you are considering implementing CI/CD in your organization then I would recommend the following:

  • Focus your software development on small incremental change rather than big breaking changes. Break bigger changes down in smaller ones.
  • Invest in creating an automated test suite. Do this incrementally as it's too big to do it all at once. But don't stop until you have good code coverage (> 90%).
  • Once you have some tests in place, start setting up the automation for your pipeline.
  • Use feature toggles or canary deployments to gradually expose your user base to a new feature, rather than your entire base at once.

In the next post, I will show an example of how to set up Jenkins and use Ravello to run your integration and system tests.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.