Jenkins pipeline, Terraform & Ansible are 3 most powerful open-source tools widely used for achieving automation.
In this article, we will be discussing end-to-end automation on Microsoft azure cloud using Terraform, Ansible & Jenkins.
Before starting let’s discuss very briefly the importance of automation.
DevOps practices are powered by automation. Automation lies at the heart of DevOps methodology. So as DevOps engineers, we must focus on automation and try to bring automation as much as possible in the software delivery process.
Below are two key benefits of automation –
- Reduce manual effort & time.
- Reusability of automation solutions, scripts, etc.
So, we must focus on the above two benefits when you develop any automated solution. That is our automated solution must reduce manual effort & time, also we should be able to reuse such concepts, solutions, scripts, etc in future projects. This is a bit of basic knowledge about automation that we must remember.
Key points to focus on to achieve end-to-end automation
- Automate Infrastructure Deployment (IaC): When we think about end-to-end automation, we first should focus on Infrastructure as Code (IaC). Using IaC we will be able to automate the infrastructure deployment. For example, Terraform, Pulumi, etc. can help us here.
- Automate Software Installation: Once we have done with infrastructure deployment, the next thing is software provisioning. We should focus on automating the process. For example, tools like Ansible, Chef, Puppet, etc. can help us here.
- Automate Application Codebase Integration & Deployment (CI/CD): We should focus on CI/CD pipeline which will automate the integration process & deployment/ delivery. Using a pipeline, the codebase will be continuously integrated & deployed on the target environment, hence it is called Continuous Integration & Continuous Deployment/ Delivery.
- Automate Testing: When we develop any application it has to go through a lot of testing & testing is a very crucial activity in the SDLC lifecycle. We should focus on automating these tests which will include unit testing, integration testing, functional testing, etc. For example, tools like Selenium, Junit, etc can help us in automating it. Also, we should focus on integrating this automated testing with the CI/CD pipeline.
To complete this tutorial, you would need below –
Knowledge/ experience on –
- Java-based application
- Web server
- One Linux machine where Jenkins is installed & Jenkins dashboard is accessible from a browser. On the same machine below software tools should be installed.
- Another Linux machine where the below software packages should be installed. And on the same machine, we will be deploying our application.
- Tomcat web server. It should be up and running.
- We will be using the Microsoft Azure cloud platform to host the infrastructure, so you need one Microsoft Azure account with proper permission. In case you don’t have a Microsoft Azure account you can use AWS, GCP, etc. cloud platforms to host your infrastructure.
Technologies used in this guide:
- Terraform v1.3.7 – Infrastructure as Code (IaC) tool
- Ansible [core v2.13.7] – Configuration management tool
- Jenkins – Pipeline tool.
- Tomcat – Web server
- Microsoft Azure Cloud to provision the infra
Tasks that Need to be performed
Below are the tasks which will be performed sequentially in this end-to-end automation demo project –
Task 1: Infrastructure Deployment
Deploy one VM on the Azure cloud using Terraform and the process will be automated using a Jenkins pipeline.
Task 2: Install Application Dependency Software
Install Java, and Tomcat web server using Ansible, and this will be automated using Jenkins pipeline.
Task 3: Deploy Sample Application on Web Server
Deploy one sample Java application on the Tomcat web server using Ansible and the process will be automated using a Jenkins pipeline.
How many pipelines will be used to accomplish the above 3 tasks?
This is a very crucial decision, how many pipelines we should have? There are multiple options.
Option 1: Using 3 separate pipelines & finally integrating them
In our demo project for the above tasks, we can have 3 Jenkins pipelines and connect one pipeline with the other. Once the first pipeline finishes, it automatically triggers the second pipeline and automatically triggers the third pipeline.
Option 2: Using two pipelines & finally integrating them
In this case, the pipeline will consist of terraform script. And in the terraform script we will be using the ‘terraform provisioner’ block. In the ‘terraform provisioner’ block we will call our ansible playbooks for dependency software installation. Once this infra pipeline finishes it will trigger another pipeline which will build the application & deploy it on the target machine.
Our Solution & Approach in this demo project: Option 2
In the previous section, we saw there is two option –
- Using 3 pipelines where each pipeline will have 3 separate tasks & finally, we need to integrate them
- Using 2 pipelines where we will be using ‘terraform provisioner’ & decreases the number of pipelines from 3 to 2 pipelines.
In this tutorial, we will see Option-2 where we will have 2 pipelines.
We will see the individual pipelines in Jenkins. Once you understand the individual pipelines you can easily integrate these two pipelines to create an end-to-end flow.
Now, we will see individual Jenkins pipeline in detail.
Pipeline 1: To provision infra & install dependency software
One pipeline for infrastructure deployment & application dependency software installation using Terraform & Ansible.
We will deploy one virtual machine (VM) on Microsoft Azure. For this, we have already written a simple & practical guide, please follow the below article. The below article tells you how to deploy a VM using Terraform & it won’t use ‘terraform provisioner’.
Reference article for pipeline 1: Jenkins pipeline script to deploy infrastructure using Terraform
We recommend you review the above article before proceeding to the next steps. Because pipeline-1 will be almost similar to what is mentioned in the above article. The only change is we need to add the Ansible playbooks. Below are the changes discussed.
We have decided to use ‘terraform provisioner’ which calls ansible playbook to install the dependency software.
We will use a Tomcat web server, which requires Java as a prerequisite. To automate the installation of JDK & Tomcat ansible will be used. Ansible Playbook will install Java, and Tomcat and do the required configuration.
Right now, below is our directory structure.
ubuntu@ubuntu01:~/end-to-end-poc$ tree . ├── install-java.yml ├── install-tomcat.yml ├── inventory.yml ├── main.tf └── variables.tf 0 directories, 5 files
We have two ansible playbooks –
It will install JDK on the target machine.
You can get the Ansible playbook for JDK installation from the below article.
It will install Tomcat on the target machine.
You can get the Ansible playbook for Tomcat installation from the below article.
Once Terraform deploys the VM, it will call the above Ansible playbooks one by one.
In this inventory file, we need to just put the target/remote machine name where Ansible will perform the tasks. The content of this inventory.yml file is below.
Terraform scripts – main.tf file & variables.tf
Clone the below GitHub repository for the terraform scripts.
GitHub Repo – https://github.com/sk617/terraform-with-provisioner.git
Run the below command to clone the above repository (before doing that make sure you have ‘git’ installed on your machine)
Pipeline 2: To deploy the application
This pipeline will be for building a sample Java application using Maven and deploying it on a web server. We will be using a Tomcat web server.
You can follow the below article to set up this pipeline. It will guide you to create a working application deployment pipeline.
Reference article for pipeline 2: Jenkins pipeline script to build & deploy application on web server
Steps to integrating pipeline-1 & pipeline-2
Below are the steps.
Step 1: Create two Jenkins pipeline jobs
Here we have two pipeline jobs pipeline 1 & pipeline 2. The first job will be the “parent” job, and the second job will be the “child” job. At first, the parent job will run, and the child job will be triggered after the parent job finishes.
Step 2: Configure the parent job to trigger the child job
In the parent job, you need to add a section called “Post-build Actions.” In this section, you need to add a “Build other projects” action. In the “Build other projects” action, you need to specify the name of the child job.
Step 3: Save the parent job and the child job
Once you have saved the parent job and the child job, you can run the parent job. When the parent job finishes, the child job will be triggered.
That’s all, you have an end-to-end setup using Jenkins pipeline, Terraform & Ansible.
If you are interested in learning DevOps, please have a look at the below articles, which will help you greatly.
- Jenkins Pipeline as code – High level information
- Jenkins pipeline script to build Java application and push artifacts into repository
- Jenkins pipeline script to build & deploy application on web server
- Automation of Java installation – using Ansible
- Automation of Tomcat installation – using Ansible
- 10 frequently used ansible modules with example
- What is End-to-End Monitoring of any web application, and Why do we need it?
- What is “Monitoring” in DevOps? Why do we need to Monitor App/DB servers, Transactions etc.?
- DevOps Engineer or Software Developer Engineer which is better for you?- Let’s discuss
- How To Be A Good DevOps Engineer?
- How to do git push, git pull, git add, git commit etc. with Bitbucket