Filebeat vs Fluentd - Let's compare two popular logging agent

Last Updated on February 27, 2023 by cscontents

Table of Contents

Introduction

Filebeat and Fluentd are two most popular logging agents available in the market. Logging agents like them are very important since they help us in setting up monitoring/observability. In today’s fast-paced world people don’t tolerate delay. And when it comes to business critical application, infrastructure etc. it is very important to take quick actions if there is something wrong. And this can be only achieved using proper monitoring tool. Using monitoring tool, we will be able to get alert and visualize the metrics, logs etc. and using those log, metrics data we will be able to take quick action if there is any issue OR in some case we will be able to anticipate the upcoming issue.

If you need some more detailed discussion on the requirement of monitoring , then please head over to below article.

What is “Monitoring” in DevOps? Why do we need to Monitor App/DB servers, Transactions etc.?

There are multiple monitoring tools out in the market. Depending on our use case and budget we can choose the proper monitoring tool. Below is a list of set of tools which we can consider.

Popular Monitoring tools

Elasticsearch, Logstash, Filebeat, Kibana
Elasticsearch, Fluentd, Kibana
Dynatrace
Datadog
Splunk etc etc.

Among all these tools, ELK stack is one of the most powerful and popular tool which is used for end-to-end monitoring.

When we talk about ELK stack below two popular options available with us.

First option, filebeat will collect the logs and then send them to Elasticsearch.
Second option is, filebeat will collect the logs and send the logs to Elasticsearch via Logstash.

We can use Fluentd also with Elasticsearch.

Using Fluentd as logging agent. Here Fluentd will collect the logs and then send them to Elasticsearch. If log parsing, processing etc is required then it will be done by Fluentd itself. In this case Logstash is not used.

Filebeat vs Fluentd – Comparison

Filebeat	Fluentd
It is an open source lightweight logging agent.	It is an open source logging agent, but it has Fluent-bit which is an ultra-lightweight logging agent.
Filebeat is maintained by Elastic company which manages the ELK stack.	Fluentd is maintained by CNCF.
Filebeat can send the logs to Logstash or Elasticsearch. If you want to parse the logs and enrich them, then you need to send the logs to Elasticsearch via Logstash. That is filebeat can work as log collecting agent only.	If you use Fluentd, it can collect logs as well as process the logs and then send them to Elasticsearch. That is fluentd can work as both “log collecting agent” and “log aggregator”. And due to this it is said that Fluentd provides you a unified logging layer.
On the target machines we need to install filebeat to collect the logs, and to process the logs (if required) we need to configure filebeat to send the logs to Logstash.	On the target machine we need to install Fluentd to collect the logs and if required that fluentd instance can process the logs. You can install fluent-bit in the target machine if there is resource constraint.
Since filebeat works as agent only it consumes very less resource of the target machine.	If we fluentd only for log collection then it consumes less resource. But if it collects the logs as well to parse them in that case it will consume more resource of the target machine. And due to this it is not recommended to use same instance of Fluentd as both ‘log collector’ and ‘log aggregator’ (in resource constraint case). If there is no issue with resource (memory, CPU) then you can go ahead to use fluentd as log collector & aggregator both.
Filebeat + Logstash = log collecting & parsing	(Fluentd as ‘log forwarder’) + (Fluentd as ‘log aggregator’) = log collecting & parsing, in case of resource constraint issue. Otherwise, single instance of Fluentd will work.
Filebeat is installed on target machine or deployed as daemonset in case of Kubernetes to fetch the logs.	Fluentd is installed on target machine and deployed as daemonset in case of Kubernetes to fetch the logs.
In case of Kubernetes , filebeat pod consumes comparably lesser system resource.	In case of Kubernetes, fluentd pod consumes comparably more system resource.
Below is the log flow, if we want to parse the logs using Logstash. If log parsing is not required, then we can directly send the logs to Elasticsearch.	Below is the log flow, if fluentd work as both log forwarder and log aggregator. If we use two instances of fluentd so that one instance can collect the log and another one can process it (to limit the resource consumption by fluentd on target machine).
If we buy license of Elastic stack or take subscription we can easily get support from the tech team of Elastic company. Also, we can get support from the community if we face any challenges or issue.	Support for fluentd is mostly community driven. There are commercial versions of fluentd available, for those good supports available.

These are some crucial points about filebeat and fluentd which must be considered before we choose between filebeat & fluentd.

Which one to select between Filebeat & Fluentd?

Which one to select it’s completely up to you and your requirement. But below are some perspective.

If you would like to use single tool for log collecting and processing then ‘Fluentd’ would be the choice.
If you are concern about resource consumption by the logging agent then ‘Filebeat’ would be good choice.
If you would like to use Elasticsearch & Kibana to the fullest, then ‘Filebeat’ is preferred.

Thank You.

If you are interested in learning DevOps, please have a look at the below articles, which will help you greatly.