Jenkins High Availability (HA) - a simple practical guide

Last Updated on January 13, 2024 by cscontents

Table of Contents

Introduction

Jenkins is one of the most popular pipeline tool. It is widely used for creating various pipeline especially in DevOps world. Popularity of Jenkins also due to the below reason –

It is an open source tool.
It is freely available to use.

Staring from small personal project to big enterprise project, everywhere Jenkins can be used as a pipeline tool.

Now, when we are to use any tool like Jenkins which can have a huge impact on your software delivery, we must ensure its availability 24×7, that is Jenkins should be available all the time to support the software delivery process.

Coming to the question, how do we make sure we have Jenkins which is always available? In other words, we can say how to make our Jenkins highly available?

In this article we will be discussing on the solution of the above questions, that is how to make Jenkins highly available.

Prerequisite

To complete this tutorial, you would need below –

Prerequisite knowledge

Knowledge/ experience on Jenkins
Concept of High Availability (HA)
Knowledge/ experience on HAProxy

Prerequisite setup

Total 3 machines are needed.

Two separate machines for two Jenkins instances. We will be using Linux machine.
Another Linux machine for HAProxy.
One NAS (Network-Attachaed Storage) drive is needed which should be mounted in both the Jenkins machine. If you are using any cloud platform, then for that cloud platform there must be some NAS drive offering.
- g., EFS in AWS, Azure Files in Microsoft Azure etc.

In all the Linux machine the user should have sudo privileges to install the software packages.

Disadvantages of using single instance of Jenkins

Below are few disadvantages –

Due to use of single instance of Jenkins server we will have single point of failure which is very risky. And in production environment we can’t tolerate this kind of risk.
If the single instance of Jenkins goes down, all the task will be blocked which are dependent on Jenkins as pipeline tool. So, until & unless Jenkins server is brought back all the task will be waiting in queue.
If we depend on single instance of Jenkins, and your tasks are blocked due to some issue in that Jenkins then obviously your client will not be happy, and you may lose future business deal.

How to make Jenkins highly available?

The obvious solution or answer to the above questions is to have at least two instances of Jenkins and one NAS drive should be mounted in both the Jenkins machine so that Jenkins job related data is available to the instance of Jenkins.

It is due to fact that we can’t rely on a single instance/ server of Jenkins. If anything happens to the one instance we can continue our tasks using the other Jenkins server.

Jenkins High Availability (HA) – Explanation with Failover

In our Jenkins High Availability architecture –

HAProxy – It will be at frontend. We will be using HAProxy IP to access the Jenkins dashboard from browser.
Jenkins – Two instance of Jenkins will be configured as backend. One of them will be in Active mode & other one will be in passive mode. Basically, it will be Active-Passive model.

We will have below –

Jenkins 1 – Active Jenkins instance, HAProxy will always send traffic to this instance as long as it is up & running.

Jenkins 2 – Passive Jenkins instance, HAProxy will send traffic to this instance only when Jenkins 1 is down.

Below is a simple diagram which explains our Jenkins High Availability or Jenkins HA architecture.

Jenkins High Availability (HA) – implementation guide

Below is the step-by-step guide –

Step 1: Install two instances of Jenkins on two separate machines

In this tutorial we are using two RHEL machines where we will install Jenkins. You need to execute below commands one by one.

sudo wget -O /etc/yum.repos.d/jenkins.repo

https://pkg.jenkins.io/redhat-stable/jenkins.repo

sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
sudo yum upgrade
sudo yum install java-11-openjdk
sudo yum install jenkins
sudo systemctl daemon-reload
sudo systemctl start jenkins
sudo systemctl status jenkins

If you have Linux machine of other flavor like ubuntu, centos etc. then please head over to below article for Jenkins installation. Once you complete the Jenkins installation come back here.

Jenkins Installation on various OS – a simple practical guide

Once you see both the Jenkins instance up & running, open your browser and check whether the URL of both the Jenkins are accessible or not. Jenkins URL is in below format.

http://<IP_address_of_your_machine>:8080

Note: Port 8080 should be allowed in your machine network. Inbound rule should be added with port 8080.

Now, follow the given instruction in Jenkins UI and complete Jenkins setup.

Step 2: Mount the NAS drive in both Jenkins machine

In step 1, we have installed two instances of Jenkins on two separate machines. Now, we need to mount a NAS drive at /var/lib/jenkins/job directory on both the machine. This is because all the jenkins job related configuration & build information are stored in the /var/lib/jenkins/job directory.

Step 3: Install HAProxy

In this step we will install HAProxy on a different machine. In our case we will be using one Ubuntu machine where HAProxy will be installed.

Execute the below commands.

sudo apt update

sudo apt install haproxy

Step 4: Configure HAProxy to point to the Jenkins instances

To configure HAProxy we need to edit the /etc/haproxy/haproxy.cfg file.

Open the /etc/haproxy/haproxy.cfg file by running below command.

sudo vi /etc/haproxy/haproxy.cfg

Once you open the haproxy.cfg file, delete or remove all the content from this file.

Now copy & paste the below content in the haproxy.cfg file and edit the IP address of jenkins1 & jenkins 2 (enter your IPs).

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main
    bind *:80
    default_backend jenkins_servers
       reqadd X-Forwarded-Proto:\ http
    
backend jenkins_servers
       server jenkins1 45.228.89.101:8080 check
       server jenkins2 80.201.179.58:8080 check backup

Note: Port 80 should be allowed in your machine network. Inbound rule should be added with port 80. Your HAProxy machine should be able to communicate with both Jenkins instance over port 8080. Also, both Jenkins instance should allow inbound communication from the HAProxy machine.

Step 5: Start the HAProxy

Now, start the HAProxy by running below command.

systemctl start haproxy

Check the status of HAProxy,

systemctl status haproxy

Step 6: Add a Cron Job in Jenkins-2 to Reload

Though we are using one NAS drive to sync the job related data between the two Jenkins instance. But if we create/edit any job in the dashboard of Jenkins-1, that change or edit would not be automatically reflected in the dashboard of Jenkins-2. To reflect those changes, we need to reload the Jenkins 2.

There are few ways through which we can reload any Jenkins instance –

From browser if we hit this URL → http://<Jenkins_machine_IP>:8080/reload, it would be reloaded.
From the Jenkins machine CLI, if we run this command, curl -u user:api_token http://<Jenkins_machine_IP>:8080/reload, it will be reloaded.
If we restart the Jenkins instance then also it gets reloaded.

We will follow the second approach (point number 2) to reload Jenkins-2, and we need to reload it every time when we do any change in Jenkins-1.

To avoid the manual reloading of Jenkins-2, we would create one cron job in Jenkins-2 which will run a shell script every 1 min to reload it. Now, to reload Jenkins using the above-mentioned curl command we need a user & API token.

Below are the steps which you can follow –

Step 6.1: Create an API Token

You need to create an API token for the Jenkins user which will be used for reloading the Jenkins instance using curl command.

Once you log into Jenkins dashboard, in the top right corner you would see the username and with that you would see a dropdown.

Click on that dropdown.
In the dropdown, you would see ‘Configure’ option, click on that ‘Configure’ option

Once you open the above-mentioned ‘Configure’ page, you need to create an API Token there. Copy & paste the API token in any notepad.

Step 6.2: Create a Shell Script

Now we will create a shell script. In our case we will create the shell script in /root directory. You may create in any other place also. Execute the below commands.

vi jenkins_reload.sh

Copy & paste the below content (replace the user & API token with your details).

#!/bin/bash
curl -X POST http://localhost:8080/reload -u admin:11nh69f9065929u3a83a65abe3ba78dfbg

Save the file.

Step 6.3: Create a Cron Job

In this step we will create a cron job to run the above shell script every 1 min. Execute the below commands.

sudo vi /etc/cron.d/jenkins_reload

Copy & Paste the below content.

*/1 * * * * root /bin/bash /root/jenkins_reload.sh

Save the file.

Step 7: Access the HAProxy URL from browser

Finally, we can access our highly available Jenkins instances through the HAProxy IP address or HAProxy URL (if DNS is configured). Basically, you need to access the below URL from browser.

http://haproxy_ip_address:80

Note: Jenkins instances should be up & running. If any one of them is down then also we will be use Jenkins since we have two Jenkins instance.

Thank you.

If you are interested in learning DevOps, please have a look at the below articles, which will help you greatly.