Break your CI jobs – build pipeline with Jenkins

Share it!Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInEmail this to someoneShare on Reddit

Since a few years ago we have been facing a rapid growth and evolution of Continuous Integration and Delivery ecosystems. Their complexity rises all the time. Build definitions are no longer a few simple command lines. It is not a rare case that our automated jobs consist of dozens of steps that each contain several instructions. Many new challenges appear:

  • very long builds (while a 5 hour build is a pain, imagine one which takes 30),
  • unstable builds, especially painful if joined with the above point,
  • unmaintainable build configuration.

We should tend our build definitions carefully. One of the most important principles is to avoid very long and complicated configurations. As we are breaking our code into classes and methods, we should do the same with CI jobs. As a result, each build should have only one responsibility assigned and be pretty short – 10 minutes is great, but 1 hour in many cases is not a tragedy. Moreover, such refactoring should also result in simplified configuration.

To visualize the problem, let’s imagine a project with classical CI requirements: there is a need to execute unit, integration, functional and capacity tests. Additionally, test environments can be created on demand and delivery artifacts can be promoted, either for automatic production environment update or sending to a customer. The most obvious way is to create one big job for all test execution. However, to avoid mentioned on the beginning pitfalls we should create several builds, each with one responsibility. It is important that they will not be triggered in any order. Moreover, we would like to be able to easily deploy a test environment knowing the results of all tests from specific revision. We have them same use case for artifacts promotion.

What we need to do is a job pipeline. The Jenkins Continuous Integration tool offers us a few ways to achieve that:

  • Parameterized Trigger plugin – it lets us trigger new builds when your build has completed, with various ways of specifying parameters for the new build,
  • Conditional Build Step plugin – it allows us to add a condition controlling build steps execution – mixed with the Parameterized Trigger plugin lets us create almost any configuration we would like to,
  • Build Pipeline plugin – adds visualization to jobs configured with the above mentioned plugins. Additionally this allows us to define manual triggers (requiring human confirmation) without using the Conditional Build Step plugin,
  • Jenkins Workflow plugin – one of the most powerful plugins. Using custom groovy DSL allows us to make magic within our jobs.

BTW, here is a compilation of the most useful Jenkins plugins.

Returning to our example, we can create a simple pipeline with Parameterized Trigger plugin. The setup will be the following:

  • commit stage – executing compilation, unit and integration tests. Providing fast feedback. Upon successful completion this triggers the acceptance stage,
  • acceptance stage – executing automatic acceptance tests, upon successful completion this triggers the performance stage,
  • performance stage – executing capacity tests, upon completion does not triggers any jobs,
  • user acceptance stage – creates a test environment, used for manual acceptance tests,
  • promotion stage – the last build in the pipeline, promotes artifacts in whatever way a project requires.

Depending on your build architecture, each stage will pass a revision, ready artifacts, artifacts timestamp or any other specific version identification to following ones. All solutions are more or less easily achievable with additional Jenkins plugins and/or by build systems.

Jenkins Pipeline will provide us a visualization, with the history of pipeline execution. For the above example, it will look similar to this:



The view provides us information about build status and allows us to trigger the pipeline at the proper stage.

Similar setup can be achieved with workflow plugin. Its capabilities are enormous, allowing us to specify almost any build configuration in groovy DSL. We can either call all the above defined builds from workflow job or move all their steps to it. Unfortunately, no matter which path you will choose currently there is no visualisation option in Jenkins open source edition.

On the other hand, Workflow plugin provides a feature which is not available within Pipeline and Parameterized Trigger plugins: execute several builds in parallel and upon completion of the last one trigger another job. Such behaviour is very useful when you would like to split your commit or acceptance stages into several jobs to lower their execution time. In such situations you can use Pipeline and Parameterized Trigger plugins for the whole pipeline while the exact commit and/or acceptance stages would be defined using Workflow plugin.

It is worth it to split your builds into small parts. They are easier to understand, maintain and are more stable. Additionally, they are also usually executing faster (if parallelized or additional hardware has been added). Jenkins is like a swiss scizor – with a combination of the above mentioned plugins you can achieve almost any setup you can imagine. Next time while modifying your job definition do not make shortcuts. Use the same approach as for your code – create clean and robust configuration. Do not be afraid to refactor. Just do not forget to let everyone in your team know that half of the jobs have been changed and dozens of new ones created!

Share it!Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInEmail this to someoneShare on Reddit