Log Alert Conditions

Log alerts are alert conditions based on log events and log pipelines. Typically log pipelines are applied to resources or resource groups, and the associated log alert conditions set the alert criteria, for example matching text in the log message. 

Log alert conditions use regular expression patterns to match ingested logs and trigger LogicMonitor alerts to send you notifications when these log events or anomalies occur.

With log alerts, you can speed up your investigation by adding alert conditions to track the logs you always want to know about or creating alerts on a detected anomaly.

Viewing Alert Conditions

On the Logs page, select the Pipeline Alerts icon to open the Alert Conditions page where you can review and manage alert conditions.

• Severity—Displays the level (Critical, Error, Warning) configured for the alert condition.
• Display As—Shows the configured display name of the alert condition.
• Logs query—Displays the regular expression pattern to match log events that will trigger the alert.
• Condition type—Displays the type of alert condition, such as basic or stateful alert.
• Pipeline—Displays the pipeline that the alert condition applies to.
• Description—Provides a short explanation of the alert condition.
• Active—Lets you toggle the alert condition on and off.
• Filter alert conditions associated with a specific pipeline.

Adding Alert Conditions

You can add alert conditions directly from any log event or anomaly listed in the Logs page, or from the Pipelines and Pipeline Alerts pages.

• From a log event or anomaly on the Logs page: Open the action menu at the beginning of the log line and select “Create alert condition” to open the Add Alert Condition dialog.
• From the Pipelines page: Select the Alert Conditions icon or count to open the Alert Conditions page, and select the plus sign to add an alert condition.

You can add the following alert conditions:

• A basic alert condition
• A stateful alert condition

Adding a Basic Alert Condition

In the Add Alert Condition dialog, enter information as follows:

1. Select the type of Alert severity level (Critical, Error, Warning) to generate when conditions are met.
2. Enter a Display name and a Description to be displayed in the list of alert conditions.
3. Select a Pipeline to apply the alert condition (unless prefilled).
4. Unless prefilled, enter a Logs query to filter the events in the pipeline. For more information see Writing a Filtering Query. Select the arrow to preview the results and refine your query before you save.
5. From Trigger alert, to activate an alert condition, do the following:
   1. for each matching log – Select this option if you want to generate an alert for each log matching a condition.
   2. only after matching logs within – Select this option to generate an alert if a certain number of logs match a condition within a defined time period. To generate an alert, there should be at least 1 matching log. You can select the following time intervals: 10 minutes, 30 minutes, 1 hour, 8 hours, and 1 day.

6. To define Alert clear conditions, enter the time that the alert will persist. Default is 60 minutes.
7. Toggle the Auto-clear after acknowledge switch on to clear the alert after it is acknowledged.
8. Toggle the Active switch on to activate the alert condition.
9. Select the Save icon to add the alert condition.

When you return to the Alert Conditions page, you can review the alert you created in the table.

Example

The pipeline Windows Servers (_resource.group.name="Windows") targets a subset of logs related to a “Windows” resource group. The alert triggers on specific keywords or values (eventcode 1100 and application log files). You do not need to include the resource group name in the query for the alert condition (however it would work).

Acknowledging Alerts

When log alerts conditions are matched, they trigger standard LogicMonitor alert notifications based on the alert settings (Critical, Warning, Error) and will route through the configured escalation chain.

You can also:

• View the log alert in the Logs graph, where it will display as a line alongside the alerting log event.
• Acknowledge or clear the alert in the Alerts list.

Limitations

Each pipeline can have no more than 90 alerts per minute. If the rate of alerts exceed this limit, they will be discarded and not processed.

Adding a stateful alert condition

To add a stateful alert condition, do the following:

1. From a log event or anomaly on the Logs page, open the Action menu at the beginning of the log line and select Create alert condition to open the Add Alert Condition dialog.
   Alternatively, from the Pipelines page, select the Alert Conditions icon or alert condition count to open the Alert Conditions page, and select the plus sign to add an alert condition.
2. In the Alert condition name field, enter a name for the alert condition.
3. From the Alert condition type dropdown list, select Stateful.
4. From the Alert severity dropdown, select the type of alert severity level, such as Critical, Error, or Warning when the conditions are met.
5. Enterthe description for the alert.
6. Select the toggle Existing Pipeline or New Pipeline to choose an existing pipeline or a new pipeline to create the alert condition.

7. Unless prefilled, enter a Logs query to filter the events in the pipeline. For more information, see Writing a Filtering Query.
8. Select the arrow to preview the results and refine your query before you save.
9. In the Alerting condition field, to add an alerting condition,type a query or select the existing queries by clicking inside the field. Use this condition to define the condition to generate an alert.

10. Under Alert clear conditions, to add a condition to clear an alert condition, do the following:
    • Type a query for Clear condition or select the existing queries from the list. Use this condition to define the condition to clear an alert.
    • For Clear after, enter the time that the alert will persist. Default is 60 minutes. You an choose time in days, hours, and minutes.
    • Toggle the Auto-clear after acknowledge switch on to clear the alert after it is acknowledged.
11. In the Alert grouping criteria table,to add alert grouping criteria, do the following:
    1. Select the add icon to add the alerting criteria.
    2. In the Alert grouping criteria and Clear grouping criteria fields, select or type the alerting and clearing criteria.
    3. Select Apply.

12. Toggle the Active switch on to activate the alert condition.
13. Select the Save icon to add the alert condition.