How to setup or install ELK stack on VM - a simple & practical guide

Last Updated on March 9, 2023 by cscontents

Introduction

ELK stack (Elasticsearch, Logstash & Kibana) is one of the most powerful monitoring tools available in the market. ELK stack is a combination of 3 open-source software: Elasticsearch, Logstash & Kibana. Since ELK stack is offered by Elastic company so this stack is also called Elastic stack.

Elasticsearch – it is a search & analytics engine. It is used to store data like logs, metric sets, etc.

Logstash – is a log processing tool. If you want to enrich or parse logs then Logstash will help you.

Kibana – it is a visualization tool. You can access the Kibana dashboard in the browser and visualize all the data and various charts.

This is a brief introduction to the ELK stack. And since you are learning ELK stack, which is a monitoring tool, then I hope you are aware of what is monitoring & its importance. If not you can head over to the below article to get a quick high-level understanding.

What is “Monitoring” in DevOps? Why do we need to Monitor App/DB servers, Transactions, etc.?

In this article, we will see the installation of the ELK stack on a virtual machine (VM) or how you can set up the ELK stack on a VM.

Prerequisite

To complete this tutorial you need below,

Prerequisite Knowledge

Hands-on experience with Linux CLI command.

Prerequisite setup

In this article, we will be discussing ELK stack installation on VM, you would need the below Linux VM(s). We will be using the ubuntu flavor of Linux.

Now, based on the requirement we can have ELK stack on a single VM or on 3 separate VMs.

If you are installing the ELK stack for learning purposes then you can go ahead with a single machine.
- 1 machine for Elasticsearch, Logstash & Kibana
  - RAM: min 4 GB
  - CPU: min 2 vCPU

If you want to install ELK stack for any small project or want to set up ELK stack for lower env in your project then you can go with 3 dedicated machines for Elasticsearch, Logstash & Kibana.
- - Elasticsearch machine:
    - RAM: min 8 GB
    - CPU: min 2 vCPU
  - Logstash machine
    - RAM: 2-4 GB should be fine
    - CPU: It depends on how many events/second Logstash will process. Logstash machine with 1 vCPU can process hundreds of events/second. But if you use high resource-consuming filters like grok, mutate, etc, then that single-core machine will process fewer events/second.
  - Kibana machine:
    - RAM: 2-4 GB should be fine
    - CPU: 2 vCPU
  - If you want to install an ELK stack for your production environment or want to set up a production-grade ELK stack, for this we will write another article.

Installation of ELK Stack on Ubuntu

Elasticsearch and Logstash need Java. So, before installing Elasticsearch and Logstash we need to install Java.

Java installation on ubuntu

To install Java on Ubuntu, follow the below guide.

https://cscontents.com/java-installation-on-ubuntu-rhel-centos/#Java_installation_on_Ubuntu

Elasticsearch installation on ubuntu

As of March 2023, the latest stable version of Elasticsearch is 8.6

Here, we will be installing elasticsearch 8.6.2

We will be using the Debian package to install Elasticsearch. Below are the steps you need to follow.

Step 1: Download the Elasticsearch Debian package

To download the Elasticsarch Debian package.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.6.2-amd64.deb

Step 2: Extract the package

sudo dpkg -i elasticsearch-8.6.2-amd64.deb

Step 3: Configure Elasticsearch

Once you have installed elasticsearch, you need to open the /etc/elasticsearch/elasticsearch.yml file and edit a few things.

Below is our /etc/elasticsearch/elasticsearch.yml file, you can check it & make the necessary changes in your /etc/elasticsearch/elasticsearch.yml file. Open the /etc/elasticsearch/elasticsearch.yml file in an editor.

vi /etc/elasticsearch/elasticsearch.yml

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
http.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
http.cors.enabled: true

http.cors.allow-origin: "*"

#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
discovery.type: single-node
#
# Bootstrap the cluster using an initial set of master-eligible nodes:

#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 05-01-2023 06:07:07
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
 enabled: false
 verification_mode: certificate
 keystore.path: certs/transport.p12
 truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["ELK-1"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
#http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

Step 4: Start & Enable ELasticsearch

Once you have edited the elasticsearch.yml file and saved it, run the below commands.

sudo systemctl daemon-reload

sudo systemctl enable elasticsearch

sudo systemctl start elasticsearch

Step 5: Check the status of Elasticsearch

Run the below command.

sudo systemctl status elasticsearch

To check the details of your elasticsearch instance you can run the below curl command.

curl http://localhost:9200

Output:

{
  "name" : "node-1",
  "cluster_name" : "my-application",
  "cluster_uuid" : "*****************",
  "version" : {
    "number" : "8.6.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "*******************************",
    "build_date" : "*******************",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Installation of Logstash on ubuntu

To install Logstash v8.5.2 you need to follow the below steps.

Step 1: Download the Logstash Debian package

Run the below command to download Logstash Debian package.

wget https://artifacts.elastic.co/downloads/logstash/logstash-8.6.2-amd64.deb

Step 2: Extract the package

Extract the downloaded package using dpkg.

sudo dpkg -i logstash-8.6.2-amd64.deb

Step 3: Start & enable logstash

To start & enable Logstash run the below command.

sudo systemctl start logstash

sudo systemctl enable logstash

Step 4: Check the status of Logstash

Run the below command to check the status.

sudo systemctl status logstash

Installation of Kibana on ubuntu

Follow the below steps to install Kibana.

Step 1: Download Kibana Debian Package

Download the Debian package for Kibana v8.6.2

wget https://artifacts.elastic.co/downloads/kibana/kibana-8.6.2-amd64.deb

Step 2: Extract the package

Extract the downloaded Debian package

sudo dpkg -i kibana-8.6.2-amd64.deb

Step 3: Configure Kibana

Now, edit the /etc/kibana/kibana.yml file

vi /etc/kibana/kibana.yml

There will be a default configuration of Kibana. But you need to ensure the below lines are there.

server.port: 5601
server.host: "0.0.0.0"
server.publicBaseUrl: "http://ip-of-your-machine:5601/"
elasticsearch.hosts: ["http://localhost:9200"]

Step 4: Start & Enable Kibana

Once you have edited the kibana configuration file, run the below commands.

systemctl daemon-reload

systemctl enable kibana

sudo systemctl start kibana

Step 5: Access Kibana Dashboard from the browser

To access the kibana dashboard from any browser, you need to enter the below URL.

http://ip-of-your-kibana-host:5601

Thank you.

If you are interested in learning DevOps, please have a look at the below articles, which will help you greatly.