Continuous Integration is at the core of automation in DevOps, where the pipeline goes through a series of stages to produce the final binary output which can then be deployed as a part of Continuous Delivery/Deployment.
What is integration?
Typically, in a project, several developers work simultaneously on their own features. These features put together are a part of a sprint. When their features are complete, the developers integrate them with others. This activity of regularly merging the features from different developers is called integration.
So, what is Continuous Integration (CI)?
All the features that are integrated several times a day need to be compiled, unit tested, analysed for quality & security vulnerabilities and then deployed to a binary repository management software in an automated way with a proper feedback mechanism is “Continuous Integration” in DevOps terms.
The diagram below depicts the various stages of a CI pipeline.
Importance of CI
A developer when working on their own features, tests them locally on their own machines. They may not be aware of what other developers are working on. At one point in time all these features will go to production together, so it is very important that one developer doesn’t break the other developer’s code. Therefore, CI gives timely feedback to take remedial actions. Apart from integration errors the code also needs to be unit tested, analysed for quality, security and the binary output, version controlled for promoting to different environments.
In another working methodology where every feature is independently deployed to production, CI is extremely important to give quick feedback so that corrective measures can be taken fast and the feature is eventually deployed to higher environments/production rapidly.
Let’s dive deeper into the tools & stages of a Continuous Integration pipeline.
Continuous Integration Tools
There are specific tools that are used to setup a CI automated pipeline. These tools orchestrate the entire CI flow from one step to another along with notifications management.
Some of the CI orchestration tools are shown below.
Stages of a CI Pipeline
Source Code Version Control
It is imperative to have a central place where the entire code base is stored and all developers contribute collaboratively using the same source.
In technical terms it is called “Version Control System” (VCS). There are majorly two categories of VCS, as mentioned below along with examples.
The developers check in the code to remote branch and resolve merge conflicts if any, this is the first step to integration.
Also, VCS is the trigger point of a continuous integration pipeline. Usually, as soon as the code is checked in, an automated pipeline is triggered either through a web hook, polling mechanism or a scheduled trigger.
Build & Dependency Resolution
Once the code is checked in, the build process starts. For eg. if it is a Maven based java project the code is cleaned, compiled and packaged. The build process yields an output. For eg. for Java projects it could be a *.jar, *.war or *.ear files etc or for .net based projects the output could be *.exe, *.dll, .cshtml, .aspx etc.
Generally, every project uses third party libraries/plugins etc., these need to be added at the build step for the project to be successfully compiled. These dependencies are resolved either through third party sources (like maven.org, nuget.org etc.). In case the internet access is restricted these dependencies are made available from an internal binary repositories like Artifactory or Nexus.
The developers are responsible for creating automated unit test cases using frameworks like JUnit, MSTest, NUnit, Jasmine etc. depending on the technology. These frameworks are integrated in the CI pipeline and utilize the automated test cases to run on the code and produce test results. Test coverage i.e. what percentage of code is covered under these test cases can also be produced at this step.
If anything fails, there is an option to set the pipeline to abort or continue with the next step, commonly called fail fast.
Code Quality Analysis & Security Vulnerabilities
Most of the times the security aspect is completely ignored initially and is considered at the end of the development and testing phases. Identifying security issues/threats early during the development phase also called shifting left is a very important stage of a CI pipeline.
The developers need to follow certain coding standards, security best practices, avoid duplication of code and pursue refactored coding processes. There are some very sophisticated tools that automate this analysis and these tools can be easily integrated in the CI pipeline. The development team can set thresholds against the code quality metrics, which when exceeded can be set to abort the pipeline. At the end of the day, it is the team’s choice whether to abort the pipeline or continue with it (again called fail fast).
The security expert needs to work collaboratively to help implement these security practices.
The popular tools used for quality and security analysis are SonarQube/Checkmarx/Veracode/Whitesource etc.
Binary Repo Deployment
In software terms, the build output is also called binary files or artifacts. For eg. A *.jar file is a binary output of a java application, similarly *.dll is an output of a .net application, so and so forth.
The binary files or the artifacts are the ones that get deployed on servers so that the application can be served and utilized by the end users.
These binary files also need to be maintained, version controlled and promoted to various environments like QA, staging, UAT or production etc. Here comes binary repository software like JFrog Artifactory, Sonatype Nexus, ProGet etc. which are used exclusively to manage the build artifacts. These binary management software can be integrated in the CI pipeline and the binary output from the post compilation step can be deployed/uploaded in them automatically.
With the deployment of artifacts to a binary repository, the CI pipeline is complete.
Another usage of a binary repository is to serve third party dependencies during the build step for downloading dependencies. For eg. Nuget packages can be stored and downloaded from Artifactory when the CI pipeline is compiling the project.
Getting timely feedback is crucial as this enables the developers to take corrective actions. At least there should be two notifications, one at the onset of the CI pipeline and the other as soon as the pipeline ends (success or failure).
The notifications can be sent for each step as well, with success or failure messages. A CI pipeline can be configured to send notifications as emails or integrate popular tools like Slack, Mattermost, Microsoft Teams etc.
These are the typical stages in a CI pipeline for most of the technologies. The stages can be in any sequence depending on the project requirements.