Every project faces build failures from time to time, no matter how careful developers are and how well Continuous Integration is implemented. From time to time someone will forget to compile or test their code (and I mean forget, not skip). Other times, there might be a small difference between development and CI environments. Some projects have to use different commands to run their build in CI and by developers. In fact, there are hundreds of reasons why a developer change can break a build.
One of the solutions of this problem is pre-tested commit mechanism. The idea is that the CI server automatically fetches developer changes and tests them on the CI environment, before publishing it to other developers and integrating with the main branch. Thanks to that it is almost impossible to introduce new failures as such gate will detect the majority of problems.
One of the first questions which appears is should such pre-tested commit be automatically merged to the main branch in case of its build success? Initially, it sounds like a great idea and probably for most projects is. It is fully compatible with one of CI best practices that commits should be small and frequently (in the best case scenario several times per day) integrated with the main branch. The problem with that approach is that developers know that each committed and pushed change can possibly reach baseline. They cannot perform a dry run of their changes to prove some concepts or just publish changes for another developer. The approach is also partially not compatible with the pull request mechanism. While such pre-commit builds can be executed after accepting a pull request, it is good practice to know that build passes before accepting it.
Thus we are reaching another little problematic matter: should we test incoming change before integration with the baseline, after it, or in both cases? Testing it before integration with the baseline has several advantages, like the possibility to execute many builds in parallel. Testing after integration with the baseline must be done in a synchronous manner. Without it, automatic integration with the baseline would be not possible. On the other hand, such testing proves that the change can be introduced into the baseline without conflicts and failures.
We can mix all the above concepts together and create an almost fully failure proof system:
- all development branch changes are automatically built by the CI server,
- developers use pull request to merge their changes to precommit branches,
- for the pull request to be merged it must prove that there is a corresponding successful CI build,
- pull requests are merged to pretest commit branches and are verified by the CI build after initial automatic integration with the baseline.
The flow is quite complicated but it is almost fully failure proof. In case you do not need to maintain the builds green all the time at all cost you might want to skip the last point.
Jenkins allows us to easily setup pretest commit mechanism. If you are using GIT and just want to build all changes from developer branches, it is just as simple as setting the following configuration in the Source Code Management job config section:
The clue is ‘branches to build’ field definition. Setting it to **/ready/* will make the job build all branches matching that pattern. The job will check regularly (according to poll SCM settings) for any new branches and new changes within already existing ones, fetch them and execute defined build steps. It is advised to set the two visible in the above picture settings:
- wipe out repository & force clone,
- prune stale remote-tracking branches.
They help to avoid problems with artifacts left by previous builds and branches which use the same names as the ones that have been deleted.
To facilitate tracking which exact build has built which branch, you can use Description Setter Plugin. Below is the configuration (post build action) that set build description to the branch name which was used. On the left side you can see how it looks in practice in builds history:
If you want your ready branches to be automatically merged to the main branch, you can manually setup build steps performing merge before the actual build and push after success or use Pretested Integration Plugin:
Integration with pull requests will vary depending on the VCS software you are using. For most popular systems the Jenkins plugin facilitating integration exists.
If your project build is relatively fast and your team consists of only a few people, it is quite easy to setup pretested commit mechanisms. However, in case of larger teams and tests executing for several hours there are several obstacles which need to be handled, like missing CPU power and long feedback time. Nevertheless, developers are getting used to it quite fast. Isn’t it great that you can commit and push your changes frequently without worrying about breaking builds? It is like your personal compilation and test failures guardian!