How we migrate 350+ Maven CI jobs to Pipeline as Code with Jenkins 2 and Docker!

eXo Platform Blog

A year ago at eXo, we decided to build all our projects in Docker containers.

In this series of articles, we will tell you our story of how we upgraded more than 350 Jenkins “standard” Maven jobs to Pipeline as code on our Continuous Integration servers, using Jenkins 2 and Docker.

This is a good opportunity to revisit the problems we encountered and the solutions we adopted, as well as to examine some best practices around Maven/Gradle/Android builds in Docker containers, all managed by the Jenkins Pipeline as Code pattern.

As with all important technical migration, we did it step by step. The 3 major steps were:

  1. Create CI Docker Images
  2. Use Jenkins Pipeline & Pipeline Docker plugins
  3. Generate all Pipeline jobs with DSL jobs

The following diagram represents the workflow for these steps:

Jenkins 2 and Docker Pipeline on Continuous Integration serversIn this article, we will explain the context (why we did it), and the first step of the migration towards creating your own CI Docker images.

What we build

eXo Platform is built upon open-source and open standards. It adheres to the the Java EE stack and leverages many open-source components.

Mobile 1Mobile 2Desktop

We have to manage many builds on our CI servers for a variety of reasons, which we can divide into several themes:

Platform components and add-ons

As the first step we have multiple git repositories for all the components required to create a distribution package for eXo Platform:

  • 25+ components to build
    • Platform projects (cf. diagram below)
    • Juzu projects
  • Add-ons
    • 15+ supported Add-ons
    • 100+ community Add-ons
  • Native Mobile projects
    • Android Application
    • iOS Application

multiple git repositories for all the components

Git Workflow

We used a Git Workflow, based on a branching model. All eXo projects are required to follow this model;

  • eDvelop : using the latest validated development changes.
  • Feature: Feature branches are dedicated branches for one big feature (.
  • Stable/: Stable branches are used to perform releases and write or accept fixes. “xxx” is the stable version name (e.g 1.0.x).
  • Fix Fix branches are dedicated to integrating bugfix on the Development branch. If needed the fix is then cherry-picked and transferred to a more stable branch.
  • integration: Integration branches are branches dedicated to automatic integration tasks (like Crowdin translations, for example).
  • POC: POC branches are dedicated branches to developing proof of concept

Branching model Git Workflow for eXo projectsPlatform versions and clients

Over the last 10 years eXo has released many versions of the eXo Platform. With eXo Platform 5 currently in development, in addition to the versions we maintain for our clients (eXo Platform 4.x : 4.0, 4.1, 4.2, 4.3, 4.4), we always need to manage several versions of the Build stack.

We build projects with the JDK6, JDK7 or JDK8, and Maven 3.0, Maven 3.2 or Maven 3.3 versions. The native Android application is built by Gradle.

Remember how to create Jenkins job via the UI?

Even if Jenkins 2 has benefited from the many efforts to enhance it, it’s always painful to create a new Maven job via the UI, as you can see below:

jenkins pipeline as code

“Jenkins DSL and Pipeline jobs with Docker to the rescue!”

As a result, we have had to manage a lot of builds via the Jenkins UI and many tools in several versions (Maven, JDK, etc.) on CI agents and development laptops.

Finding this inefficient and painful, we decided to find a solution to automate and manage it in a different way.

The container (Docker) technology combined with the emergence of automation tools layered onto Jenkins with Pipeline, and the efficiency of the Job DSL plugin, appeared to be the ultimate solution in the eXo context.

Step1: Create your own Docker Images

There are important requirements to be aware of when creating your own Docker images in general, and builds in particular.

jenkins docker pipelineChoose a distribution base image

All our CI agents are managed via Puppet and are based on the Ubuntu distribution, so we decided to use a lighter Docker image based on this distribution: the baseimage-docker.

This was a good compromise between a very small image like Alpine, which resulted in problems related to JDK installation, and the official Ubuntu Docker image, which was too big.

 

Define the locale

Who hasn’t had test failure due to encoding issues? It works on your Linux machine, but it fails on your colleague’s laptop running Windows.

Thus, it’s mandatory to define a locale to ensure that all developers will have the same behavior, whatever their operating system, and that it will be compliant with the CI server configuration.

Create a dedicated CI user

“…take care of running your processes inside the containers as non-privileged users (i.e., non-root)” is one of the most important best practices from the Docker security guide.

In the Jenkins context, it’s important to create a dedicated CI user that will fit the uid:guid of the user used by Jenkins on your server agents.

Indeed, when Jenkins will execute a Pipeline job, it will have to mount the workspace and other required folders into the Docker container.

For example, Jenkins will execute the following command on the server agent:

 

This dedicated CI user will prevent permissions issues with Docker volumes.

You willl notice that we define the EXO_CI_USER_UID variable with the Docker ARG instruction. This is important for the developer experience, which we will explain later in this series of articles.

ENTRYPOINT and CMD

In a Dockerfile, the ENTRYPOINT instruction is an optional definition for the first part of the command to be run. So both ENTRYPOINT or CMD instructions, specified in your Dockerfile, identify the default executable for the Docker image.

The best way is to combine both of them by using CMD to provide default arguments for the ENTRYPOINT.

In the earlier version of the Jenkins Docker Pipeline plugin, Jenkins didn’t use the –entrypoint option when starting containers, so the command was:

In that case, if your Docker Image declared an ENTRYPOINT as startup command, Jenkins was not able to use your image.

That’s why we added a custom script in our Docker Images, as a workaround, which allows one to execute Maven as a startup command, but also to execute the cat command and other commands executed with an absolute path:

Dockerfile

docker-entrypoint.sh

In the latest versions of the Jenkins Pipeline Docker plugin, this problem has been fixed, to allow use of the –entrypoint option so that the ENTRYPOINT instruction is always overridden:

Use inheritance to avoid code duplication

We have created and continue to create Docker images to cover our entire Build Stack. Currently this list contains:

  • exoplatform/ci:jdk6-maven30
  • exoplatform/ci:jdk7-maven30
  • exoplatform/ci:jdk7-maven32
  • exoplatform/ci:jdk8-maven32
  • exoplatform/ci:jdk8-maven33
  • exoplatform/ci:jdk8-gradle2

As you can imagine, there are not many differences between all those images, so we have created CI base images at several levels to be able to avoid code duplication as much as possible.

The diagram below shows how those images are organized:

create Docker images for entire Build Stack

Below is an extract of the Dockerfile for the exoplatform/ci:jdk8-maven33 CI Image. You can see that there are only Docker instructions related to the Maven installation, since all other configurations have been done in the inherited images (define locale, create user CI…).

As explained before, we combined ENTRYPOINT and CMD Docker instructions to be able to run all Maven commands easily in this container:

We also add a custom script, to be able to run the cat command and commands other than Maven by giving the absolute path to the command:

Finally, you will have noticed that we defined the M2_REPO and MAVEN_OPTS environment variables as Docker ENV instructions with all important parameters having default values, which can be overridden by the -e option in the docker run command.

Test your Docker Images

As with other source code, you can create tests suites for your Docker files and images. For the eXo CI Docker images, we use Goss via dgoss.

  • Goss is a YAML based tool for validating a server’s configuration
  • dgoss is a wrapper around Goss that aims to bring the simplicity of Goss to Docker containers.

The first step is to create a YAML file to describe what you want to test in your Docker container. There are some examples online that can help you to create this configuration file, but it can be also be generated from the goss command line.

For example, say we want to check that some Maven configuration files exist in the container. We want to test that the mvn –version command is compliant with the Maven and JDK versions installed.

goss.yaml

Then to execute those tests, it is a simple matter of executing the dgoss tool:

 

create tests suites for eXo CI Docker imagesConclusion

If you are interested in using or testing these Docker Images, you will find:

Feel free to give your feedback about this.

Next step

Now that we have created Docker Images for all our Continuous Integration stacks, we will explain in the next article (Part 2) how to use them on your laptop, whatever your operating system. Stay tuned!

Related Posts

I'm the software factory manager at eXo. I'm responsible for all software factory services (Jenkins CI, GitHub, Nexus, Sonar...) and the eXo release process. I help to make the eXo developer experience better by simplifying and automating as much of the process as possible. Passionate about IT and open source, I contribute to open source projects in different ways (speaking at conferences, writing blog posts...). Also, I have written a book about Apache Maven 3.

Comments
Leave a Reply

Your email address will not be published.

I have read and accept terms of the privacy Policy. Learn More

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">