Sending Logs to Ingestion API
Last updated on 25 August, 2024If a log integration isn’t available or you have custom logs that you want to analyze, you can send the logs directly to your LogicMonitor account through the logs ingestion API. The received events are mapped to resources in LogicMonitor before they are further processed or stored.
Log Ingestion Endpoint
Endpoint | https://<account>.logicmonitor.com/rest/log/ingest |
Protocol | HTTPS:443 |
Headers | Authorization: <LMv1 token> |
Method | POST |
Content Type | application/json |
Content Encoding | gzip, deflate |
Resource Path
For the logs ingestion endpoint, https://<account>.logicmonitor.com/rest/log/ingest
:
- The base URL is
https://<account>.logicmonitor.com/rest
- The resource path is
/log/ingest
Where <account>
is the company name or account name for your LogicMonitor portal.
Note: The base URL for the logs ingestion endpoint is /rest (not /santaba/rest, which is the URL for the LogicMonitor REST API).
Authentication
All requests to the log ingestion API use LogicMonitor’s LMv1 API tokens for authentication. You can create the API tokens in your LogicMonitor account Settings under Users & Roles.
Every request must include an HTTP authorization header in the format:
Authorization: LMv1 AccessId:Signature:Timestamp
The Signature is a base64 and HMAC encoded hash based on your Access Key,
a timestamp in epoch milliseconds, and the resource path of the endpoint:
signature = base64(
HMAC-SHA256(
Access Key,
HTTP VERB
+ TIMESTAMP
+ POST DATA
+ RESOURCE PATH
)
)
Note: An API-only user that submits logs through the Log Ingestion API must have “Manage” permissions for Logs. For more information, see Roles.
Limitations
The log ingestion API expects an array of entries with the following constraints on the content size:
Description | Size |
Maximum content size per payload | 8 MB |
Maximum array size for a single log message | 32 KB |
Maximum array size for sending multiple logs/events in an array | – |
Rate limit for calls to the LM Logs ingestion endpoint (per account/API key) | 240K requests/min |
Example Request
Sending a log message for an Amazon EC2 instance to the ingestion endpoint using curl:
curl --location \
--request POST 'https://<account>.logicmonitor.com/rest/log/ingest' \
--header 'Authorization: <LMv1 token>' \
--header 'Content-Type: application/json' \
--data-raw '[{
"msg": "Generating example log message 31687",
"_lm.resourceId":{"<property>": "<value>"}
}]'
Parameters
The log ingestion endpoint expects an array of JSON objects as events with any JSON attribute. Reserved attributes are described in the following.
Parameter | Description |
message | msg | Msg | (Required) The log message. See Limitations. If an event does not contain this attribute, the event is not forwarded to the log processing pipeline. |
timestamp | date | _timestamp | Timestamp | eventTime | published_date | (Optional) The timestamp when the event occurred. Supported date formats are ISO8601 and Unix Epoch (in secs, ms, ns). If events don’t contain any of the default date attributes and you haven’t defined your own timestamp, the event will be assigned the current date as the timestamp. The timestamp can be either int or string . For more accurate nanosecond times, use string (to avoid rounding errors of int values in JSON). |
_lm.resourceId: {<property>: <value>} | _lm.resource.id is not required for log ingestion, only for mapping to a LogicMonitor resource. To map events to a unique LM resource, any resource property can be used, for example: deviceId, AWS ARN, Kubernetes pod Id.<property> is the LogicMonitor resource property to use for device mapping.<value> is the value of the resource property.If more than one property exists, only the first property will be mapped. Example 1 – deviceId known:_lm.resourceId: { system.deviceId: "<deviceId>" } Example 2 – in the AWS Lambda function using unique ARN: _lm.resourceId: { system.aws.arn: <arn> } |
To send additional information along with the log message, you can add more JSON items in the message. The information displays as “metadata” in the Logs table when reviewing logs. For more information, see Viewing Logs and Log Anomalies.
Note: Logs can have a severity associated with each event if it matches the criteria for an anomaly or matches an alert condition. This severity is not set during ingestion. For more information, see Log Alert Conditions.
Response Codes
The log ingestion API uses conventional HTTP response codes to indicate the success or failure of an API request.
Status Code | Description |
202 - Accepted | The request has been accepted for processing, but the processing has not been completed. |
207 - Multi-status | Some events in the batch have not been accepted for processing. See the “errors” property within the response for details. |
400 - Bad Request | The request is invalid. For example, it may be missing headers or the request body is incorrectly formatted. |
401 - Unauthorized | Authentication failed. The API key provided is not valid. |
402 - Payment Required | The LM Logs feature has not been enabled. Please contact Support. |
403 - Forbidden | The API key doesn’t have permission to perform the request. |
413 - Payload Too Large | The payload exceeds the maximum content size of 8 MB. |
429 - Too Many Requests | The number of requests exceeds the rate limit. |
500 - Server Error | Something went wrong on LogicMonitor’s end. |
502 - Bad Gateway | A dependency failed to respond within a reasonable time. |
Example: 202 – Accepted
Header:
X-Request-ID: 38b78dd6-3bc0-4cd9-8a15-6af552d49c3e
Body:
{
"success": true,
"message": "Accepted"
}
Example: 207 – Multi-status
Header:
X-Request-ID: 38b78dd6-3bc0-4cd9-8a15-6af552d49c3e
Body:
{
"success": false,
"message": "Some events were not accepted. See the 'errors' property for additional information.",
"errors": [
{
"code": 4001,
"error": "Resource not found",
"event": {
"_lm.resourceId": {
"system.deviceId": "kish"
},
"message": "test"
}
},
]
}
Custom Error Codes
Custom error codes provide insights into what went wrong when log ingestion fails. If events do not meet the log ingestion constraints, such as failed mapping to a LogicMonitor resource or field requirements, one of the error messages described in the following is returned.
Error Code | Description |
4001 - Resource not found | The query for the resource was attempted but the device was not found. |
4002 - More than one resource has been found | The event source must be mapped to only one LogicMonitor resource. |
4003 - Insufficient information for device lookup | The information is not enough for resource lookup. For example, the _lm.resourceId is not correctly specified and device mapping cannot be performed. |
4004 - Missing message field | LogicMonitor will not accept logs without a message field. |
4005 - Event too large | A single event may not be larger than 1 MB. |
4006 - Event too old or future | The event falls outside the configured ingestion time window, ±3 hours. |
Example: Failures
Header:
X-Request-ID: 38b78dd6-3bc0-4cd9-8a15-6af552d49c3e
Body:
{
"success": false,
"message": "Some events were not accepted. See the 'errors' property for additional information.",
"errors": [
{
"code": 4003,
"error": "Insufficient information for device lookup",
"event": {
"message": "test"
}
},
{
"code": 4004,
"error": "Missing message field",
"event": {
"_lm.resourceId": {
"system.deviceId": "kish"
},
}
}
]
}