Sending Kubernetes Logs and Events

LogicMonitor provides different methods for sending logs from a monitored Kubernetes cluster to LM Logs. The choice of method depends on the type of logs that you want to send. You can use any of the following methods to send logs from a monitored Kubernetes cluster to LM Logs.

• Using LogicMonitor Collector:
  • Using LogSource: LogSource is the recommended method to enable LM Logs. To use LogSource, you must have EA Collector 31.200 or a later version installed on your machine. For more information, see Kubernetes Event Logging LogSource Configuration or contact your Customer Success Manager. 
  • Using agent.conf: For Kubernetes events and Pod logs, configure the LogicMonitor Collector to collect and forward the logs from a monitored cluster or cluster group. For more information, see Sending Kubernetes Events and Pod logs using LogicMonitor Collector.
• Using lm-logs Helm chart: For Kubernetes logs, use the lm-logs Helm chart configuration which is provided as part of the LogicMonitor Kubernetes integration. For more information, see Sending Kubernetes Logs using lm-logs Helm chart.

Requirements

• LogicMonitor API tokens to authenticate all requests to the log ingestion API.
• LogicMonitor Collector installed and monitoring your Kubernetes cluster.

Sending Kubernetes Logs using lm-logs Helm Chart

You can install and configure the LogicMonitor Kubernetes integration to forward your Kubernetes logs to the LM Logs ingestion API.

Deploying

The Kubernetes configuration for LM Logs is deployed as a Helm chart.

1. Add the LogicMonitor Helm repository: 

helm repo add logicmonitor https://logicmonitor.github.io/k8s-helm-charts

If you already have the LogicMonitor Helm repository, you should update it to get the latest charts:

helm repo update

2. Install the lm-logs chart, filling in the required values:

helm install -n <namespace> \
--set lm_company_name="<lm_company_name>" \
--set lm_access_id="<lm_access_id>" \
--set lm_access_key="<lm_access_key>" \
lm-logs logicmonitor/lm-logs

Configuring Deviceless Logs for Kubernetes

Logs can be viewed in LM Logs even if the log is “deviceless” and not associated with an LM-monitored resource. Even without resource mapping, or when there are resource mapping issues, logs are still available for anomaly detection and to view and search.

For deviceless logs, log anomaly detection is done using the “namespace” and “service” fields instead of “Device ID”, when creating log profiles. To enable deviceless logs, set “fluent.device_less_logs” to “true”, when configuring lm-logs helmchart. For more information, see Send Kubernetes Logs to LM Logs.

Sending Kubernetes Events and Pod Logs using LogicMonitor Collector

You can configure the LogicMonitor Collector to receive and forward Kubernetes Cluster events and Pod logs from a monitored Kubernetes cluster or cluster group.

Requirements

• EA Collector 30.100 or later installed.
• You have already deployed LogicMonitor’s Kubernetes Monitoring.
• Access to the resources (events or pods) that you want to collect logs from.

Enabling the Events and Logs Collection

The following are options for enabling events and logs collection:

• Recommended—Modify the Helm deployment for Argus to enable events collection.
  For more information, see Kubernetes Events and Pod Logs Collection using LogicMonitor Collector.
  
• Alternatively—Manually add the following properties to the monitored Kubernetes cluster group (or individual resources) in LogicMonitor.

Configuring Filters to Remove Logs

The filtering criteria for Kubernetes Events are based on the fields “message”, “reason”, and “type”. For Kubernetes pod logs, you can filter the message fields. Filtering criteria can be defined using keywords, a regular expression pattern, specific values of fields, and so on. To configure filter criteria, uncomment to enable and then edit the filtering entries in agent.conf.

For example:

• To filter out INFO level pod logs to LogicMonitor, uncomment or add the line: logsource.k8spodlog.filter.1.message.notcontain=INFO
• To send Kubernetes events of type=Normal, comment out the line: logsource.k8sevent.filter.1.type.notequal=Normal

For more information, see Kubernetes Events and Pod Logs Collection using LogicMonitor Collector.

To configure filter criteria, configure the following agent.conf entries as applicable.

Collector agent.conf configurations

Helm-chart configurations

Troubleshooting

Kubernetes Logs

1. If you are not seeing Kubernetes logs in your LM Portal after a few minutes, it may be a resource mapping issue. Resource mapping for Kubernetes is handled by the Fluentd plugin.
   
2. If mapping is correct, verify that the log file path is mounted.  If the log file path is not mounted, edit the /k8s-helm-charts/lm-logs/templates/deamonset.yaml file to add the file path and volume.
   
   For example, if the path to mount is /mnt/ephemeral/docker/containers/, make the following edits:

• Add the file path:

name: ephemeraldockercontainers
  mountPath: /mnt/ephemeral/docker/containers/
  readOnly: true

• Add under volumes:

name: ephemeraldockercontainers
  hostPath:
    path: /mnt/ephemeral/docker/containers/

Kubernetes Pod Logs

If you have enabled pod logs collection and forwarding, but you are not receiving pod logs in LM Logs, restart the Collector and increase the polling interval to 3-5 minutes.