Why automate Jenkins?
Have you ever managed a Jenkins that had more than 100 jobs and somebody asked you to modify every last one of them? Let’s say that your infrastructure resembles the one from this diagram:
Imagine that you manage hundreds of servers on a daily basis using Jenkins jobs and you have come across a similar situation:
- Someone asked you to install the new plugin and to integrate its functionality into all of your Jenkins jobs – so you go to the UI, install the plugin, edit one hundred jobs one by one and apply your changes – a lot of repetitive work.
- Your infrastructure is growing rapidly, and you need to add new jobs, but you know that they are going to be very similar to the existing ones, so you clone them – not a good strategy.
- Your Jenkins host died. The Operations team say that they will install Jenkins on another machine, but they can’t recover your jobs, but no worries you were keeping backup’s for all of them in .xml files so you can leverage Jenkins CLI to restore them. Still, the whole process is going to take some time.
As you can imagine (or know from experience) performing these tasks using Jenkins UI can be very tedious and very error-prone. When your project is growing, you are going to encounter more situations like those listed above. You’re going to waste your time on Jenkins configuration whose purpose is to make your life easier. At Appliscale, we have had similar problems and eventually decided to use Jenkins Job DSL plugin to to help us automate this task.
Job DSL Plugin
In a nutshell, this plugin allows you to convert Groovy scripts into Jenkins jobs, by creating a config.xml file. When you finish configuration scripts for your Jenkins jobs, you just need to create a seed job that will build all of your jobs from a remote repository. Later, if you wish to make any modification’s, you push your changes to the repository and re-run your seed job. In case you were wondering if this will remove your build history or job configuration history – it will not.
Keeping configuration as code in a repository gives many benefits:
- DRY – extracting common parts of the configuration.
- Pull requests – more eyes to check your work.
- Visibility – team members will see what have changed.
- Mobility – migration is not a problem anymore.
- Backup – in case a server has died.
- Testing – you can even write unit tests.
To summarise: no more tedious crawling through Jenkins UI.
Local Jenkins
Before we dive into the code, I highly recommend you take a small break and visit Appliscale GitHub to take a look at Centos-Jenkins-DSL repository. This project contains a configured Jenkins server on Vagrant with CentOS 7. Our company is using this repository for the development stage – to test if new jobs are working properly and to ensure that any modifications that we have made didn’t break anything.
Even if you’ve decided not to spend time on setting up your environment, it’s worth taking a look at how it is done.
Let’s get to work
Now let’s take a look at example job configuration in single_jobs.groovy script, which you can find in the vagrant directory:
job("example-job") {
logRotator(2, 10, -1, -1)
scm {
git {
remote {
url "git@github.com:mwpolcik/Centos-Jenkins-DSL.git"
branch 'master'
}
}
}
parameters {
booleanParam 'NOTIFY_QA', true, ""
}
configure { project ->
project / 'properties' / 'hudson.model.ParametersDefinitionProperty' / parameterDefinitions << 'hudson.plugins.validating__string__parameter.ValidatingStringParameterDefinition' {
name('IMPORTANT_GIT_TAG')
defaultValue("")
failedValidationMessage('Warning!')
description('''
Some big description of this parameter
'''.stripIndent().trim()
)
}
}
steps {
shell(readFileFromWorkspace('jobs/scripts/test.sh'))
gradle {
useWrapper true
tasks 'clean test'
switches '''
-Dhttp.proxyHost=proxy.example.com
-Dhttps.proxyHost=proxy.example.com
-Dhttp.proxyPort=80
-Dhttps.proxyPort=80
'''.stripIndent().trim()
}
}
publishers {
chucknorris()
}
}
Detailed documentation, which I highly recommend, can be found here: Jenkins DSL API.
Job declaration
job("example-job") {
...
}
-
- The first line we define what type of job we want, by default it’s a Freestyle one which is highly flexible and very configurable, typically it consists of:
- SCM, such as Git, CVS or Subversion.
- Triggers to control when Jenkins will perform builds.
- A script that performs a build.
- Steps to collect information out of the build, such as archiving the artefacts, etc..
- Steps to notify other people/systems on build result.
Go here to see a complete list of all job types.
- The first line we define what type of job we want, by default it’s a Freestyle one which is highly flexible and very configurable, typically it consists of:
Log Rotator
logRotator(2, 10, -1, -1)
-
- On a second line we are configuring log rotator with following properties:
- Days to keep logs: 2
- Max number of builds to keep: 10
- Days to keep artifacts: none
- Max number of builds to keep with artifacts: none
- On a second line we are configuring log rotator with following properties:
SCM – git plugin
scm {
git {
remote {
url "git@github.com:mwpolcik/Centos-Jenkins-DSL.git"
branch 'master'
}
}
}
Next we code-styleify remote repository for the job and which branch to use. There are a lot more options available, but I am going to keep it simple. Here is how the SCM section will look in XML:
<scm class="hudson.plugins.git.GitSCM"> <userRemoteConfigs> <hudson.plugins.git.UserRemoteConfig> <url>git@github.com:mwpolcik/Centos-Jenkins-DSL.git</url> </hudson.plugins.git.UserRemoteConfig> </userRemoteConfigs> <branches> <hudson.plugins.git.Branchcode-style> <name>master</name> </hudson.plugins.git.Branchcode-style> </branches> <configVersion>2</configVersion> <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations> <gitTool>Default</gitTool> </scm>
-
- As you can see Job DSL from couple lines of code creates a lot of XML.
Parameters
parameters {
booleanParam 'NOTIFY_QA', true, ""
}
-
- In the parameters section, we added a checkbox variable ‘NOTIFY_QA’ with the default value true and empty description.
Configure block
-
- On rare occasions, you will come across plugins that are not yet integrated with Job DSL plugin or misconfigured (e.g. Slack) but do not worry there is a solution for everything. In this case, you can use configure block, which will directly insert XML nodes into config.xml.For the purpose of this demo, I am using Validating String Parameter Plugin – it will add a string parameter which must be filled. Otherwise, the job will fail.
To use the configure block you first have to know what the XML for the given plugin will look like. You have two options:- Ask Google; someone already had a similar problem.
- Add this parameter to your job and check config.xml located in JENKINS_HOME.If you are using Centos-Jenkins-DSL project, then you will find this file in /var/lib/jenkins/jobs/JOB_NAME/config.xml.
In our case it will look like this:
- On rare occasions, you will come across plugins that are not yet integrated with Job DSL plugin or misconfigured (e.g. Slack) but do not worry there is a solution for everything. In this case, you can use configure block, which will directly insert XML nodes into config.xml.For the purpose of this demo, I am using Validating String Parameter Plugin – it will add a string parameter which must be filled. Otherwise, the job will fail.
<properties> <hudson.model.ParametersDefinitionProperty> <parameterDefinitions> <hudson.plugins.validating__string__parameter.ValidatingStringParameterDefinition> <name>IMPORTANT_GIT_TAG</name> <defaultValue></defaultValue> <failedValidationMessage>Warning!</failedValidationMessage> <description>Some big description of this parameter</description> </hudson.plugins.validating__string__parameter.ValidatingStringParameterDefinition> </parameterDefinitions> </hudson.model.ParametersDefinitionProperty> </properties>
configure { project ->
project / 'properties' / 'hudson.model.ParametersDefinitionProperty' / parameterDefinitions << 'hudson.plugins.validating__string__parameter.ValidatingStringParameterDefinition' {
name('IMPORTANT_GIT_TAG')
defaultValue("")
failedValidationMessage('Warning!')
description('''
Some big description of this parameter
'''.stripIndent().trim()
)
}
}
After you compare your Groovy code with this XML, you will have an idea of how it is working.
Steps
steps {
shell(readFileFromWorkspace('jobs/scripts/test.sh'))
gradle {
useWrapper true
tasks 'clean test'
switches '''
-Dhttp.proxyHost=proxy.example.com
-Dhttps.proxyHost=proxy.example.com
-Dhttp.proxyPort=80
-Dhttps.proxyPort=80
'''.stripIndent().trim()
}
}
First build step, shell, will be added with a script located in repository defined in scm section. readFileFromWorkspace is a very handy method; it’s built into DslFactory class and allows you to easily load files from the workspace of your Jenkins job.The next build step is to configure the Gradle plugin to use a Gradle wrapper, perform clean and test tasks with a couple of Gradle switches to be invoked. A small tip for those who wish to add multiline strings; use stripIndent and trim methods, otherwise you will get additional spaces, etc. in your Jenkins job fields.
Post-build Actions, aka publishers
publishers {
chucknorris()
}
For demonstration purposes, the job is configured to invoke Chuck Norris plugin after job completion.
Reusability
-
- Now it’s time for some refactoring. Some of this configuration will be common for other Jenkins jobs and we want the process of creating new ones to be as painless.
Methods
-
- You can find the source code for this part of the post in src/main/groovy/JobBuilder directory.We will start by extracting following into methods:
- Adding variable from Validating String Parameter Plugin.
- scm configuration.
- Gradle configuration.
First one will look like this:
- You can find the source code for this part of the post in src/main/groovy/JobBuilder directory.We will start by extracting following into methods:
package jobBuilder.Utils
class Param {
static Closure requiredString(
String _name,
String _defaultValue="",
String _regex=".+",
String _failedValidationMessage="You must set this!",
String _description="") {
return {
it / 'properties' / 'hudson.model.ParametersDefinitionProperty' / parameterDefinitions << 'hudson.plugins.validating__string__parameter.ValidatingStringParameterDefinition' {
name(_name)
defaultValue(_defaultValue)
regex(_regex)
failedValidationMessage(_failedValidationMessage)
description(_description.stripIndent().trim())
}
}
}
}
This is how we will use it inside our job Closure:
configure requiredString('IMPORTANT_GIT_TAG')
SCM configuration:
package jobBuilder.Utils
class Scm {
static void git(context, String repository,String gitBranch) {
context.with {
git {
remote {
url repository
branch gitBranch
}
}
}
}
}
and example of how to use it:
scm {
Scm.git(delegate,repository,branch)
}
-
- Now few important notes about the two of them, the first one, is used for creating configuration block, we are passing only plugin parameters, and we are returning Closure. The second one, on the other hand, receives context as a first parameter, in this case, it’s SCM block, by using a delegate.The last one, the Gradle configuration, is very similar to SCM, but in case you want to see how it looks like you can find it here.
Base class
- After the code is extracted into methods, it’s time to build a class that will use them and will be responsible for creating new jobs.
package jobBuilder
import javaposse.jobdsl.dsl.DslFactory
import javaposse.jobdsl.dsl.Job
import static jobBuilder.Utils.Param.requiredString
import jobBuilder.Utils.Steps
import jobBuilder.Utils.Scm
class BaseJobBuilder {
String name
String repository
String branch
String gitTag
String script
String gradleTasks
Job job
void build(DslFactory dslFactory) {
this.job = dslFactory.job(name) {
logRotator(2, 10, -1, -1)
scm {
Scm.git(delegate,repository,branch)
}
parameters {
booleanParam 'NOTIFY_QA', true, ""
}
configure requiredString(gitTag)
steps {
shell(dslFactory.readFileFromWorkspace('jobs/scripts/'+script))
Steps.gradle(delegate, gradleTasks)
}
publishers {
chucknorris()
}
}
}
}
A few notes about this class:
- You can move invoking requiredString method into parameters block.
- To use readFileFromWorkspace from class or method, you need to pass DslFactory.
- For demonstration purposes, I’m invoking our utility methods in two ways, one by calling class directly and one by importing class with the method – your choice.
If you are using our repository then you can see how I declare this class and create a new job at the bottom of single_jobs.groovy script:
new BaseJobBuilder( name: "example-job-1", repository: "git@github.com:mwpolcik/Centos-Jenkins-DSL.git", branch: 'master', gitTag: 'IMPORTANT_GIT_TAG', script: 'test.sh', gradleTasks: 'clean test' ).build(this)
The result is pretty simple, right now to define a new job we just need to instantiate BaseJobBuilder class with different parameters. Only a few lines of code and we get brand new Jenkins job. Configuring such entity is also very straightforward, let’s say for example that we’ve decided that Chuck Norris plugin is not as funny as we thought at the beginning. All we need to do right now is to delete one line of code, run your seed job and all of our builds are free from this plugin!
Source
The initial project Centos-Jenkins-DSL is based on examples from this repository:
Summary
I hope this post was helpful and that any doubts you had about using Job DSL have been dispelled. In case you haven’t looked at Appliscale GitHub yet I highly encourage you to do so. Besides the project for this post, you can find out how we automated our on-call procedure and a profiler for Erlang functions.
If you have any questions, please drop a comment below or send us a message.
