Logstash vs Fluentd - Let’s compare two popular log processing tools

Last Updated on February 18, 2023 by cscontents

Table of Contents

Introduction

Logstash and Fluentd are two of the most popular log processing tool in the market. Both are open source tool.

Logstash was developed by Elastic company.
Fluentd was developed by Treasure Data.

If we speak about importance about these tools, these tools are often used with the Elastic stack to set up monitoring stack.

Speaking from business perspective, monitoring is one of the key-aspect which will save you from losing your customer. You must set up monitoring stack for monitoring your software tech stack.

If you would like to know about monitoring in DevOps, then please head over to below article.

What is “Monitoring” in DevOps? Why do we need to Monitor App/DB servers, Transactions etc.?

In this article we will compare these two tools.

Logstash vs Fluentd – comparison

SL No.	Logstash	Fluentd
1	It is an open source log processing agent.	It is an open source log processing agent.
2	It was developed by Elastic company.	It was developed by Treasure Data.
3	Logstash is mainly used with Elastic stack which is known as ELK stack.	Fluentd is also used with Elastic stack which is known as EFK stack.
4	Logstash is just a log processing tool.	Fluentd is not only a log processing tool, but it can also work as logging agent.
5	Logstash need filebeat as logging agent to collect the logs.	Fluentd itself can collect logs & process logs. Also, Fluentd has Fluent-bit which is an ultra-light weight logging agent.
6	Logstash has many input & output plugins. But Logstash wins in this case over Fluentd.	Fluentd has many input & output plugins.
7	Logstash has many filter plugins using which logs can be processed. For example, grok	Fluentd has many filter/parser using which logs can be processed. For example, grok parser
8	Logstash does not provide us unified logging layer since it can only work as log processor.	Fluentd provides unified logging layer. Since log collection & processing can be done by Fluentd itself.
9	If you can afford a dedicated server for log processing, then it is better you go with Logstash.	If you can’t afford one dedicated server for log processing then you go with Fluentd. But you need to keep an eye on the resource consumption. Since Fluentd will work as log collector as well as log processor, so it will consume more resource.
10	Below is the log flow diagram using Elasticsearch, Logstash, Kibana & filebeat (ELK stack).	Below is the log flow diagram using Elasticsearch, Fluentd, Kibana (EFK stack).

Which one to use between ELK & EFK?

Depending on requirement & budget you can either go with ELK stack or EFK stack.

Case 1

If you have budget to accommodate a dedicated server for Logstash, then go with Elastic stack (ELK stack).

Case 2

If you are interested in unified logging layer, then go with Fluentd (EFK stack). But careful about resource consumption by Fluentd.

Otherwise, you would also need a dedicated server for fluentd where it will work as log processor. In this case fluent-bit will be installed on the target machine, and it will send the logs to fluentd aggregator server for processing.

Thank You.

If you are interested in learning DevOps, please have a look at the below articles, which will help you greatly.