Jenkins Pipeline, Terraform & Ansible - Guide for End to End Automation

Last Updated on August 2, 2023 by cscontents

Table of Contents

Introduction

Jenkins pipeline, Terraform & Ansible are 3 most powerful open-source tools widely used for achieving automation.

In this article, we will be discussing end-to-end automation on Microsoft azure cloud using Terraform, Ansible & Jenkins.

Before starting let’s discuss very briefly the importance of automation.

DevOps practices are powered by automation. Automation lies at the heart of DevOps methodology. So as DevOps engineers, we must focus on automation and try to bring automation as much as possible in the software delivery process.

Below are two key benefits of automation –

Reduce manual effort & time.
Reusability of automation solutions, scripts, etc.

So, we must focus on the above two benefits when you develop any automated solution. That is our automated solution must reduce manual effort & time, also we should be able to reuse such concepts, solutions, scripts, etc in future projects. This is a bit of basic knowledge about automation that we must remember.

Key points to focus on to achieve end-to-end automation

Automate Infrastructure Deployment (IaC): When we think about end-to-end automation, we first should focus on Infrastructure as Code (IaC). Using IaC we will be able to automate the infrastructure deployment. For example, Terraform, Pulumi, etc. can help us here.
Automate Software Installation: Once we have done with infrastructure deployment, the next thing is software provisioning. We should focus on automating the process. For example, tools like Ansible, Chef, Puppet, etc. can help us here.
Automate Application Codebase Integration & Deployment (CI/CD): We should focus on CI/CD pipeline which will automate the integration process & deployment/ delivery. Using a pipeline, the codebase will be continuously integrated & deployed on the target environment, hence it is called Continuous Integration & Continuous Deployment/ Delivery.
Automate Testing: When we develop any application it has to go through a lot of testing & testing is a very crucial activity in the SDLC lifecycle. We should focus on automating these tests which will include unit testing, integration testing, functional testing, etc. For example, tools like Selenium, Junit, etc can help us in automating it. Also, we should focus on integrating this automated testing with the CI/CD pipeline.

Prerequisite

To complete this tutorial, you would need below –

Prerequisite knowledge

Knowledge/ experience on –

Terraform
Ansible
Jenkins
Java-based application
Web server

Prerequisite setup

One Linux machine where Jenkins is installed & Jenkins dashboard is accessible from a browser. On the same machine below software tools should be installed.
- Terraform
- Ansible
Another Linux machine where the below software packages should be installed. And on the same machine, we will be deploying our application.
- Java
- Tomcat web server. It should be up and running.
We will be using the Microsoft Azure cloud platform to host the infrastructure, so you need one Microsoft Azure account with proper permission. In case you don’t have a Microsoft Azure account you can use AWS, GCP, etc. cloud platforms to host your infrastructure.

Technologies used in this guide:

Terraform v1.3.7 – Infrastructure as Code (IaC) tool
Ansible [core v2.13.7] – Configuration management tool
Jenkins – Pipeline tool.
Java
Tomcat – Web server
Microsoft Azure Cloud to provision the infra

Tasks that Need to be performed

Below are the tasks which will be performed sequentially in this end-to-end automation demo project –

Task 1: Infrastructure Deployment

Deploy one VM on the Azure cloud using Terraform and the process will be automated using a Jenkins pipeline.

Task 2: Install Application Dependency Software

Install Java, and Tomcat web server using Ansible, and this will be automated using Jenkins pipeline.

Task 3: Deploy Sample Application on Web Server

Deploy one sample Java application on the Tomcat web server using Ansible and the process will be automated using a Jenkins pipeline.

How many pipelines will be used to accomplish the above 3 tasks?

This is a very crucial decision, how many pipelines we should have? There are multiple options.

Option 1: Using 3 separate pipelines & finally integrating them

In our demo project for the above tasks, we can have 3 Jenkins pipelines and connect one pipeline with the other. Once the first pipeline finishes, it automatically triggers the second pipeline and automatically triggers the third pipeline.

Option 2: Using two pipelines & finally integrating them

In this case, the pipeline will consist of terraform script. And in the terraform script we will be using the ‘terraform provisioner’ block. In the ‘terraform provisioner’ block we will call our ansible playbooks for dependency software installation. Once this infra pipeline finishes it will trigger another pipeline which will build the application & deploy it on the target machine.

Our Solution & Approach in this demo project: Option 2

In the previous section, we saw there is two option –

Using 3 pipelines where each pipeline will have 3 separate tasks & finally, we need to integrate them
Using 2 pipelines where we will be using ‘terraform provisioner’ & decreases the number of pipelines from 3 to 2 pipelines.

In this tutorial, we will see Option-2 where we will have 2 pipelines.

We will see the individual pipelines in Jenkins. Once you understand the individual pipelines you can easily integrate these two pipelines to create an end-to-end flow.

Now, we will see individual Jenkins pipeline in detail.

Pipeline 1: To provision infra & install dependency software

One pipeline for infrastructure deployment & application dependency software installation using Terraform & Ansible.

We will deploy one virtual machine (VM) on Microsoft Azure. For this, we have already written a simple & practical guide, please follow the below article. The below article tells you how to deploy a VM using Terraform & it won’t use ‘terraform provisioner’.

Reference article for pipeline 1: Jenkins pipeline script to deploy infrastructure using Terraform

We recommend you review the above article before proceeding to the next steps. Because pipeline-1 will be almost similar to what is mentioned in the above article. The only change is we need to add the Ansible playbooks. Below are the changes discussed.

We have decided to use ‘terraform provisioner’ which calls ansible playbook to install the dependency software.

We will use a Tomcat web server, which requires Java as a prerequisite. To automate the installation of JDK & Tomcat ansible will be used. Ansible Playbook will install Java, and Tomcat and do the required configuration.

Right now, below is our directory structure.

ubuntu@ubuntu01:~/end-to-end-poc$ tree
.
├── install-java.yml
├── install-tomcat.yml
├── inventory.yml
├── main.tf
└── variables.tf

0 directories, 5 files

We have two ansible playbooks –

install-java.yml file

It will install JDK on the target machine.

You can get the Ansible playbook for JDK installation from the below article.

https://cscontents.com/automation-of-java-installation-using-ansible/#Ansible_Playbook_for_installation_of_Java

install-tomcat.yml file

It will install Tomcat on the target machine.

You can get the Ansible playbook for Tomcat installation from the below article.

https://cscontents.com/automation-of-tomcat-installation-using-ansible/#Ansible_playbook_for_tomcat_installation

Once Terraform deploys the VM, it will call the above Ansible playbooks one by one.

inventory.yml

In this inventory file, we need to just put the target/remote machine name where Ansible will perform the tasks. The content of this inventory.yml file is below.

myTFVM

Terraform scripts – main.tf file & variables.tf

Clone the below GitHub repository for the terraform scripts.

GitHub Repo – https://github.com/sk617/terraform-with-provisioner.git

Run the below command to clone the above repository (before doing that make sure you have ‘git’ installed on your machine)

git clone

https://github.com/sk617/terraform-with-provisioner.git

Pipeline 2: To deploy the application

This pipeline will be for building a sample Java application using Maven and deploying it on a web server. We will be using a Tomcat web server.

You can follow the below article to set up this pipeline. It will guide you to create a working application deployment pipeline.

Reference article for pipeline 2: Jenkins pipeline script to build & deploy application on web server

Steps to integrating pipeline-1 & pipeline-2

Below are the steps.

Step 1: Create two Jenkins pipeline jobs

Here we have two pipeline jobs pipeline 1 & pipeline 2. The first job will be the “parent” job, and the second job will be the “child” job. At first, the parent job will run, and the child job will be triggered after the parent job finishes.

Step 2: Configure the parent job to trigger the child job

In the parent job, you need to add a section called “Post-build Actions.” In this section, you need to add a “Build other projects” action. In the “Build other projects” action, you need to specify the name of the child job.

Step 3: Save the parent job and the child job

Once you have saved the parent job and the child job, you can run the parent job. When the parent job finishes, the child job will be triggered.

That’s all, you have an end-to-end setup using Jenkins pipeline, Terraform & Ansible.

Thank You.

If you are interested in learning DevOps, please have a look at the below articles, which will help you greatly.