Sending Fluent Bit Logs

Fluent Bit is an open-source, lightweight, and high-performance log processor and forwarder designed to collect, filter, and deliver log data. Fluent Bit supports multiple input sources (e.g., files, system logs, Kubernetes logs) and a variety of output destinations, including cloud services, databases, and monitoring tools.

If you are already using Fluentbit to collect application and system logs, you can forward the logs to LogicMonitor using the LM Logs Fluentbit plugin. To download the plugin, go to the LM Logs Fluentbit plugin page from the LogicMonitor Github repository. This provides a LogicMonitor-developed plugin that contains the specific instructions for sending logs to LogicMonitor.

To download the plugin files tailored to your operating system architecture, please use the commands listed below:

Supported Operating Systems (OS)

• Linux-AMD
• Linux-ARM
• MS-Windows

Requirements

• A LogicMonitor account.
• A LogicMonitor API token to authenticate all requests to the log ingestion API. To know more, see API Tokens from the product documentation.

Downloading Fluent Bit Plugin

To download fluent bit plugin, use the following commands depending on your operating system architecture.

• For Linux-AMD:
  curl -L -o out_lm-linux.so https://github.com/logicmonitor/lm-logs-fluentbit/raw/master/plugins/out_lm-linux.so

• For Linux-ARM:
  curl -L -o out_lm-linux-arm64.so https://github.com/logicmonitor/lm-logs-fluentbit/raw/master/plugins/out_lm-linux-arm64.so

• For MS-Windows:
  curl -L -o out_lm-windows.so https://github.com/logicmonitor/lm-logs-fluentbit/blob/master/plugins/out_lm-windows.so

• For Other OS:
  curl -L -o out_lm.so https://github.com/logicmonitor/lm-logs-fluentbit/raw/master/plugins/out_lm.so

Installing the Plugin

1. Run the following command to get fluent bit installation script:
   curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
2. Run below command to install fluent bit on your machine:
   sudo apt-get install fluent-bit
3. Check fluentbit status with below command:
   sudo systemctl status fluent-bit
4. Copy the appropriate plugin file for your architecture from the LM Logs Plugin repository on LogicMonitor’s GitHub.

For Windows, do the following:

• Download the installation package from the Installation Packages section of the Fluent Bit official website, then run the .exe file to begin installation.

Configuring the Plugin

Use this configuration to specify which logs should be forwarded to LogicMonitor.

Create a fluent-bit.conf file, or edit the existing one, and add the following to the Fluent-bit configuration.

[SERVICE]
    Flush        5
 
[INPUT]
    Name        tail
    Path        <file_path>/tail.txt
 
[OUTPUT]
    Name logicmonitor
    lmCompanyName  <company_name_with_domain>
    Match *
    Workers 1
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    resourceMapping {"<event_key>": "<lm_property>"}
    includeMetadata <boolean_value>
    lmDebug <boolean_value>

Running out_lm fluentbit plugin

Option 1:

Execute the following command to utilize the previously created fluent-bit.conf file with the out_lm Fluent Bit plugin for log ingestion:

<path_to_installed_fluentbit>/fluent-bit -e <path_to_out_lm_plugin>/out_lm.so -c <path_to_file>/fluent-bit.conf

Here, 

• path_to_installed_fluentbit refers to the directory where Fluent Bit is installed on your system, typically located at /opt/fluent-bit/bin/fluent-bit.
• path_to_out_lm_plugin and path_to_file should point to the locations of out_lm plugin and configuration files, respectively.

Option 2:

Alternatively, perform the below steps to point to custom config file:

sudo nano /etc/systemd/system/fluent-bit.service

Update the ExecStart line to point to your custom config:

[Service]
ExecStart=/opt/fluent-bit/bin/fluent-bit -c /path/to/your/custom.conf

Reload systemd and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart fluent-bit

Configuration Properties

You can use the following properties to configure Fluent Bit plugin with LogicMonitor.

Mapping Resources

Map the sources generating the log data using the right format, so that logs are parsed correctly when they are sent to LogicMonitor Logs.

When defining the resource mapping for the Fluentbit event, the <event_key> in the incoming event is mapped to the LogicMonitor resource, which is the value of <lm_property>.

For example, you can map a “hostname" field in the log event to the LogicMonitor property “system.hostname" using:

resource_mapping {"hostname": 'system.hostname"}

Configuration Examples

The following are examples of resource mapping.

Mapping Host Information to a Resource Attribute

This example maps the host field from the log source to the system.ips attribute in LogicMonitor.

[SERVICE]
    Flush        5
 
[INPUT]
    Name         tail
    Path         /var/log/sample_logs/*.log
    Tag          application_logs
 
[OUTPUT]
    Name         logicmonitor
    Match        application_logs
    Host         logs.logicmonitor.com
    Port         443
    URI          /rest/log/ingest
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    Format       json
    tls          on
    resourceMapping {"host":"system.ips"}

In this configuration:

• The resourceMapping field specifies that the host field from logs should map to the system.ips attribute in LogicMonitor.

Mapping Application Name to a LogicMonitor Attribute

This example maps the app_name field from the log metadata to the system.application attribute in LogicMonitor.

[SERVICE]
    Flush        10
 
[INPUT]
    Name         tail
    Path         /var/log/app_logs/*.log
    Tag          app_logs
 
[FILTER]
    Name         modify
    Match        app_logs
    Add          app_name my_application
 
[OUTPUT]
    Name         logicmonitor
    Match        app_logs
    Host         logs.logicmonitor.com
    Port         443
    URI          /rest/log/ingest
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    Format       json
    tls          on
    resourceMapping {"app_name":"system.application"}

Here:

• The FILTER section adds the app_name field with the value my_application.
• The resourceMapping associates app_name with the system.application attribute in LogicMonitor.

Including Metadata and Debugging

This example demonstrates including additional metadata and enabling debug logging.

[SERVICE]
    Flush        5
    Log_Level    debug
 
[INPUT]
    Name         tail
    Path         /var/log/system_logs/*.log
    Tag          system_logs
 
[OUTPUT]
    Name         logicmonitor
    Match        system_logs
    Host         logs.logicmonitor.com
    Port         443
    URI          /rest/log/ingest
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    Format       json
    tls          on
    resourceMapping {"source":"system.type", "host":"system.hostname"}
    includeMetadata true

In this configuration:

• includeMetadata true ensures log metadata such as timestamps and hostname are forwarded.
• resourceMapping maps the source field to system.type and the host field to system.hostname.

HTTP conf file

This Fluent Bit configuration example sets up an HTTP input to receive log data and forwards it to a specified output, a LogicMonitor endpoint.

[INPUT]
    name http
    listen 0.0.0.0
    port 9880
 
[OUTPUT]
    Name logicmonitor
    lmCompanyName  <company_name_with_domain>
    Match test
    Workers 1
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    resourceMapping {"host":"system.ips"}
    includeMetadata <boolean_value>
    id <output_id>
    lmDebug <boolean_value>

Multiple conf file

This Fluent Bit configuration file is structured to collect logs from two different files, process them using a specified parser, and forward the logs to an output destination with LogicMonitor-specific parameters.

[SERVICE]
    Flush        1
    Log_Level    info
    Parsers_File parsers-multiple-config.conf
 
[INPUT]
    Name         tail
    Path         <file_path>/tail1.log
    Parser       json
    Tag          app1.logs
 
[INPUT]
    Name         tail
    Path         <file_path>/tail2.log
    Parser       json
    Tag          app2.logs
 
[OUTPUT]
    Name logicmonitor
    lmCompanyName  <company_name_with_domain>
    Match *
    Workers 1
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    includeMetadata <boolean_value>
    id <output_id>
    lmDebug <boolean_value>

This will also need a parser configuration parsers.conf which defines how Fluent Bit processes log data formatted as JSON:

[PARSER]
    Name         json
    Format       json
    Time_Key     time
    Time_Format  %Y-%m-%dT%H:%M:%S

TCP conf file

This Fluent Bit configuration file is designed to collect log data over TCP, process it, and forward it to an output destination with specific attributes like resource mapping and authentication.

[SERVICE]
    Flush        1
    Log_Level    info
    Parsers_File parsers.conf
 
[INPUT]
    Name        tcp
    Tag         tcp_log
    Listen      0.0.0.0
    Port        <port>
 
[OUTPUT]
    Name logicmonitor
    lmCompanyName  <company_name_with_domain>
    Match *
    Workers 1
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    resourceMapping {"host":"system.ips"}
    includeMetadata <boolean_value>
    id <output_id>
    lmDebug <boolean_value>

In this configuration, the parsers.conf should be similar to parsers.conf defined above.

Multiline conf file

This Fluent Bit configuration is designed to handle logs that span multiple lines (e.g., stack traces or long messages), parse them using a multiline parser, and forward them to a specified output.

[SERVICE]
    Flush        5
    parsers_file parsers_multiline.conf
[INPUT]
    Name        tail
    Path        tail.txt
    multiline.parser multiline-regex-test
[OUTPUT]
    Name logicmonitor
    lmCompanyName  <company_name_with_domain>
    Match *
    Workers 1
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    resourceMapping {"host":"system.ips"}
    includeMetadata <boolean_value>
    id <output_id>
    lmDebug <boolean_value>

Below configuration file defines a multiline parser named multiline-regex-test for processing logs that span multiple lines, using regular expressions to group related log lines.

[MULTILINE_PARSER]
    name          multiline-regex-test
    type          regex
    flush_timeout 1000
    #
    # Regex rules for multiline parsing
    # ---------------------------------
    #
    # configuration hints:
    #
    #  - first state always has the name: start_state
    #  - every field in the rule must be inside double quotes
    #
    # rules |   state name  | regex pattern                  | next state
    # ------|---------------|--------------------------------------------
    rule      "start_state"   "/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
    rule      "cont"          "/^\s+at.*/"                     "cont"

Troubleshooting

Enable debug logging by setting the debug property to true in fluent-bit.conf to see additional information in the Fluentd console. For more information on logs troubleshooting, see Troubleshooting.