What is FluentD, and how does it work with Kubernetes?

Known for its built-in reliability and cross-platform compatibility, FluentD addresses one of the biggest challenges in big data collection—the lack of standardization between collection sources.

With a decentralized ecosystem, FluentD offers a way to seamlessly collect logs from applications on Kubernetes. The free open-source data collector is positioned to support big data as well as unstructured and semi-structured data sets for better usage, understanding, and analysis of your log data. 

This post defines FluentD, shows examples of its use in business, and provides tips on how to get started with FluentD in Kubernetes.

What is FluentD?

A cross-platform software project developed for Treasure Data, FluentD, helps solve the challenge of big data log collection. Licensed under Apache License v2.0 and written in Ruby, the program bridges the gap between data collection sources by supporting both Linux and Windows.

FluentD tracks Windows event logs with the latest versions and helps unify the collection and consumption of data, providing a better understanding of how it can be used effectively for business. After logs are read and matched with a tail input plug-in and then sent to Elasticsearch, Cloudwatch, or S3, FluentD can collect, parse, and distribute the log files.

With a seamless sync between Kubernetes, FluentD promotes better monitoring and managing of services and infrastructure. Kubernetes allows you to fine-tune your performance as you look for faults. 

Who uses FluentD?

Companies such as Amazon Web Services, Change.org, CyberAgent, DeNA, Drecom, GREE, and GungHo use FluentD for its easy installation and customization with a plugin repository for most use cases. The program offers visualization of metrics, log monitoring, and log collection. Furthermore, as an open-source software project, its community of users is dedicated to making continuous improvements.

How does FluentD log? 

FluentD in Kubernetes helps collect log data from data sources using components that compile data from Kubernetes (or another source) to transform logs and then redirect data to give an appropriate data output result. In turn, data output plug-ins help collect and repurpose the data so that log data can be better analyzed and understood.

FluentD architecture 

FluentD is designed to be a flexible and light solution with hundreds of plug-in options to support outputs and sources. FluentD in Kubernetes offers a unifying layer between log types. As plug-ins are built and used with FluentD, more ways to analyze application logs, clickstreams, and event logs become available.

You can break FluentD down into several components.

Plugin ecosystem

One of FluentD’s greatest strengths is its extensibility. It has plugins available that allow integrations with most third-party applications (AWS, Azure, Elasticsearch, MySQL, and others). These allow FluentD to collect data, process it in the FluentD engine, and output it to the storage environment of your choice.

There are many types of plugins available to use:

• Input: Collect logs from various sources, including HTTP, REST APIs, and other sources.
• Output: Allow output forwarding to a wide range of destinations, such as SQL databases, cloud services, and other log ingestion services.
• Filtering: Helps custom processing and transformation of log data by removing events, adding fields, and hiding data.
• Parser: Helps FluentD parse data in different formats, such as JSON and CSV.
• Formatting: Allows users to create custom output formats based on their needs or use pre-existing formats.
• Buffer: Used to temporarily store input streams in memory or files.
• Storage: Store FluentD’s internal state in memory, storage, or key-value stores like MongoDB or Redis.
• Service discovery: Helps users extend service discovery to meet their unique needs.
• Metrics: Store FluentD’s internal metrics locally.

Buffering mechanism

FluentD’s buffering mechanism is what allows it to efficiently process large amounts of log data quickly and get data where it needs to go. It needs effective buffering to ensure all data gets processed and nothing gets lost in translation.

It does this with effective processing usage using chunks, memory storage, backpressure mitigation, and retry mechanisms.

• Chunks: Chunks are small groups of records (normally around 2 MB  each) created to allow for easier processing.
• Memory: The space FluentD uses to process chunks. It is normally in system memory because of its speed, but it is configurable based on the user’s needs.
• Backpressure mitigation: Process to help high-load environments by limiting the amount of memory used using mem_buffer_limit and file system buffering.
• Retry mechanism: Helps FluentD process failed processing requests and marks a chunk as irrecoverable if it can’t be processed.

Configuration files

FluentD uses a hierarchical file structure to handle system configuration. It uses configuration files that contain directives for input sources, output sources, matching, system directives, and routing.

One of FluentD’s strengths is its dynamic run configuration—you don’t need to reboot the entire system to enforce changes. This allows for easier configuration and testing of new environments.

FluentD also allows for complex routing based on your logs and unique situations. It offers tags and labels in configuration files to help direct output to the right source.

Resource optimization

Environments using FluentD can use a lot of resources—especially when processing large amounts of data or working in resource-constrained environments.

FluentD offers a few solutions to help these situations:

• Fluent Bit: A lightweight log forwarder ideal for edge nodes that connect to cloud infrastructure
• FluentD Forwarder: A stripped-down version of FluentD designed for resource-constrained environments

Why is FluentD important?

FluentD includes a unified logging layer, making logs accessible and usable as they are generated—allowing them to quickly view logs on monitoring platforms like LogicMonitor. On top of that, data sources can be decoupled to iterate data faster, creating avenues for more effective and efficient uses. Here are the top reasons why FluentD in Kubernetes is the best open-source software for data collection:

• Simple to set up and customize with plugins: FluentD features a 10-minute setup time and includes more than 500 plug-ins that support volumes of use-case scenarios. 
• Free and open source: FluentD in Kubernetes is available for use without restriction and is flexible enough to meet most company needs. 
• Reliability and high performance: More than 5,000 companies already depend on FluentD for its dependable and high-quality data collection results.
• Community support: The FluentD in Kubernetes community offers dedicated support for its growth and development through several resources, including GitHub and StackOverflow discussions, docs, a dedicated Slack channel, Twitter and Facebook pages, and a bug/feature tracker.
• Compatible: FluentD works to standardize and support cross-platform syncing of data for big data compatibility, analysis, and reuse.

Best practices for FluentD

FluentD is rated one of the easiest to maintain and install data collection tools compared to other choices like Scribe and Flume. Regardless of the tool, the goal is to get the fastest and most streamlined data-collecting experience. These best practices cover FluentD’s quick setup, which leads to quick optimization of logs and processing.

Avoid extra computations

FluentD is designed to be simple and easy to use, but adding extra computations to the configuration could make the system less robust, as it may struggle to maintain and read data consistently. It’s typically well-advised to streamline data as much as possible throughout data processing, and FluentD is no different. While FluentD is flexible enough to handle even demanding data requirements, maintaining a simple data-collecting system is best.  

Use multi-process input plugins

If you find that your CPU is overloading, try a multi-process. Multi-process input plug-ins allow the spin-off of multiple processes with additional configuration requirements. While multiple child processes take time to set up, they help prevent CPU overload and bottlenecks of incoming FluentD records.

Reduce memory usage with Ruby GC parameters

Ruby GC parameters tune performance and configure parameters. RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR lowers values and improves memory usage (default is 2.0). 

How does FluentD in Kubernetes work?  

FluentD is deployed in Kubernetes as a DaemonSet so that each node has one pod. As a result, logs are collected from K8s clusters and can be read from the appropriate directories created for Kubernetes. In addition, logs can be scrapped, converted to structured JSON data, and pushed to Elasticsearch via the tail input plug-in. 

FluentD in Kubernetes + LogicMonitor

FluentD allows the analysis of a myriad of logs regardless of the organization. The program’s flexibility and seamless cross-platform compatibility offer superb communication of real-time data analysis without the danger of integrating bad data or experiencing the torture of a slowdown. 

LogicMonitor is determined to provide effective solutions for teams using FluentD for logging.

LogicMonitor’s Envision platform offers a comprehensive hybrid observability solution for organizations that need help monitoring hybrid environments. Its integration with FluentD will allow your organization to unlock FluentD’s potential and take advantage of everything it can offer. 

Contact LogicMonitor to learn more about our Log Analysis today!