What is "Monitoring" in DevOps? Why do we need to Monitor various logs, metric sets?

Last Updated on February 18, 2023 by cscontents

Table of Contents

Introduction

In the DevOps world, monitoring is a very important part. Here by monitoring I mean monitoring of application server, database server, system resource usage etc.

Let’s discuss the below question.

What is Monitoring?

Generally IT Engineer check logs when there is some issue or something not working as expected. Below are some built-in tool or command which we use to check the logs

grep
tail
cat / less / more etc.
vi / vim etc.
Windows Event Viewer

But we all know checking the logs using the above commands is a lengthy process and takes huge time. And sometimes it becomes so difficult to find the exact cause of an issue by reading the logs.

For example any business critical website which handles a lot of traffic, if anything happen to that site in the working hours or customer visiting hours, and you start looking at the log files, metrics to find the issue by using the above commands then probably you are not doing the right thing. You might be able to find the issue within the downtime, but there can be cases which are really difficult to find by reading the logs files, and doing this within the downtime may be a huge headache. That is why we need monitoring of the logs, metrics.

To solve the above issue we need some kind of tool which will help us in –

Fetching the logs from different sources to a centralized place
Correlating the logs and cross-reference them to give a holistic view
Log parsing, searching, filtering, highlighting etc. facility
Log retention as per our requirement
It needs to help us to detect anomaly detection and give us the alert in the right time.
Most importantly, we should be able to see some visualization using those logs, metrics data.

Together, all the above feature is called monitoring in DevOps, which is a very important part. We call the tool as ‘monitoring tool’ in general which will provide us all the above facility.

When we select any tool for this purpose, we need to think about the below things –

The tool should have an intuitive user interface so that various team member can use the dashboards.
It should be able to collect logs from various sources.
It should be able to fetch metric sets.
Furthermore, it should have some alerting facility etc.

In market there are many tools available, among them some are free, some are paid and some of them are SaaS (Software As A Service) offerings. So depending on your requirement and budget you can choose tools. It is always better to start with some free tools and do some POC and then take the decision. Doing some POC will surely help you to broaden your thinking.

Why do we need monitoring?

Generally, in the background of any business critical web application there are a lot of things running like

Many servers (application server, database server etc.)
Network Component (Routers, firewalls etc.)
Lots of logs (syslogs, access logs, service logs, audit logs etc.)
Metrics (performance metrics, transaction metrics, resource utilization metrics etc.)

If all the above looks good and the web application works fine, then nobody cares until something go wrong. That is why we need some sort of monitoring of the services, servers, database etc.

Thank You.

If you are interested in learning DevOps, please have a look at the below articles, which will help you greatly.