We have almost 50 developers working with different technologies to develop, test and ship apps for our demanding clients. With multiple git pushes and merge requests per hour there is a need for fast and optimized flow. Automating the CI/CD with multiple technologies and clouds/other deployment targets is critical for us. This is how we use powerful Jenkins Pipeline with Shared Pipeline Library with Jenkins and Gitlab.

Our backend runs mainly on Kubernetes cluster as NodeJS, mongodb, mysql and some php containers as well. Frontend with react, middleman and other technologies need to be tested quickly and deployed to dedicated webservers, Google Storage Buckets or ftp webhosting, depending on the client’s needs. Android and iOS team need fast and reliable CI/CD as well. All the teams need a way to build and test merge requests. Above all that, there is always something that needs to be automated in the infrastructure, so having a scripted automating platform for DevOps team is a number one priority.

Jenkins Pipeline

Some time ago, we used Jenkins freestyle jobs to checkout a repo, build it in the console with simple bash snippets and deploy it to some other server with SSH, rsync or something similar.

Then Jenkins Pipeline was released. Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. An alternative to old freestyle jobs.

With Jenkins Pipeline all you need to do is create a Pipeline or Multibranch Pipeline job and specify the Pipeline as a code. The code can be part of the definition of the job or directly as a Jenkinsfile in the project’s repository.

Example Jenkinsfile:

The pipeline has several stages, which are then showed thanks to Visualizer plugin. What it does is – build the node app with npm (sh means shell call), test it, build a docker image and run it, clean up the workspace and notify Slack about the pipeline result. slackNotify is a slack plugin call (the plugins “need” to support pipeline syntax in order to be called as a nice function with a Map of parameters)

jenkins pipeline
Jenkins pipeline – stages visualizer

Jenkins pipeline is very powerful, you can program it to do pretty much anything you want. Thanks to the fact it is using the Groovy language, you can use Java libraries in your pipeline.

Shared Pipeline library

As we work on a lot of projects we would have a lot of similar code or the exact same Jenkinsfile across all nodejs and other repositories. Imagine a simple change in slackNotify function call syntax or another cli client syntax change would mean changing the pipeline code in all repositories and branches! This actually happened recently with gcloud cli api client syntax – they deprecated the old syntax, changed gcloud docker push to gcloud docker — push. 

For our use case, we don’t want to have the pipeline logic in project repositories. Only thing we want to have in the repository is the pipeline configuration.

This can be easily done using Jenkins Shared Pipeline Library – a Groovy git repository with following folder structure

This shared pipeline repo is checked out every time a Pipeline job starts and you can load and use it in your Jenkinsfile, even specifying a branch:

Or load it implicitly (can be set in Jenkins configuration settings).

Configuration Jenkinsfile

Using the Shared Pipeline Library, the Jenkinsfile in the project’s repository can look like this:

Pipeline example

The Jenkinsfile in the previous section calls a function PipelineNodejs. That is a global function defined in the Shared Pipeline Library in vars/PipelineNodejs.groovy. A small example of the logic is here:

Check other pipelines for Android, iOS, React, middleman and others.

Merge request builder

For the merge request flow with Gitlab we can make a pipeline job that uses the Gitlab integration plugin. This can aslo be done for PRs in Github.

Merge request builder job example for iOS is the following:

Mind the gitlabBuild and gitlabCommitStatus for visualizing the Pipeline in gitlab and to allow/forbid merging the MR in Gitlab. This job can be used for all ios merge requests globally. Set the webhook to this Jenkins job from your repo and you are all done.

Important (and ugly) part is the scm checkout step with merging source and target branch. The env data (env.gitlabSourceBranch and env.gitlabTargetBranch) are parsed from the Gitlab webhook.

You can easily setup rebuilding the merge request when a new push was done to source or target branch. Another cool feature is rerunning the pipeline on a comment in the MR (we use the phrase rebuild pls and Jenkins automatically rebuilds the job!).

Job DSL and seed jobs

Thanks to Job DSL plugin, we can dynamically generate Jenkins Pipeline jobs via Jenkins DSL API.

A job that generates other jobs, is called a seed job. An example for generating the iOS merge request job:

Testing production pipeline

Now we sort of need to build, test and deploy our pipeline. CI/CD for the CI/CD Pipeline. Yo dawg, I heard you like Inception.

Place a simple Jenkinsfile in the Shared Pipeline Library and use it to seed the jobs in the previous chapter.

Other fun stuff you could do with Shared Pipeline Library

Anything you want. Really.

Links

Check other pipelines for NodejsAndroid, iOS, React, middleman and others.

https://github.com/AckeeDevOps/jenkins-pipeline-library

Leave a Reply

Your email address will not be published. Required fields are marked *