Sending Azure Logs

The Microsoft Azure integration for LM Logs is implemented as an Azure Function that consumes logs from an Event Hub and sends the logs to the LogicMonitor logs ingestion API. The following describes options for setting up the forwarding of Azure logs.

Requirements

• An Azure Cloud Account created in your LogicMonitor portal.
• LogicMonitor API tokens to authenticate all requests to the log ingestion API.
• The Azure CLI tools installed on the machines that will forward logs.
• A “User Administrator” role in Azure to create the managed identity which will access the Azure resources logs.
• Azure devices can only send logs to the Event Hubs within the same region. Each Azure region requires a separate Azure Function deployment.

Azure Templates

You can use the Azure Templates provided by LogicMonitor to do the following:

• Configure and deploy the Azure resources required to forward activity logs.
• Create a managed identity to access the Azure resources logs.
• Forward the logs to the LM Logs API.

Deploying Resources

The Azure Templates deploy a resource group named lm-logs-{LM_Company_name}-{resource_group_region}-group. The group has the following resources:

• Event Hub
• Azure Function
• Storage Account
• App Service plan

Once the Azure Function and Event Hub are deployed, the Azure Function listens for logs from the Event Hub. 

 Select the Deploy to Azure button below to open the Azure Template and run the deployment.

 When deploying, you need to provide the following details in the template:

Parameter	Description
Region	(Required) The location to store the deployment metadata. Predefined in Azure but you can change the value. For a list of Azure regions by their display names, see Azure geographies.
resource_group_region	(Required) Enter the region where you want to create the resource group and deploy resources like Event Hub, Function app, and so on. For a list of the Azure regions in plain text, run the following command from PowerShell with the Azure CLI tools installed: az account list-locations -o table
LM_Company_name	(Required) Your LogicMonitor company or account name in the target URL. This is only the <account> value, not the fully qualified domain name (FQDN). Example:  https://<account>.logicmonitor.com
LM_Domain_Name	The domain of your LM portal. Note:  If LM_COMPANY_DOMAIN is not provided or is set as empty string, by default, it is set to "logicmonitor.com". The supported domains for this variable are as follows: – lmgov.us – qa-lmgov.us – logicmonitor.com
LM_Access_Id	(Required) The LM API tokens access ID. You should use an API-only user for this integration.
LM_Access_Key	(Required) The LM API tokens access key.
Azure_Client_Id	(Required) The Application (client) ID used while creating the Azure Cloud Account in your LogicMonitor portal.  Note: This ID should have been created when you connected the Azure Cloud Account. The ID can be found in the Azure Active Directory under App Registrations.
Enable Activity Logs	(Optional) Specify whether or not to send Activity Logs to the Event Hub created with the Azure Function. Can be “Yes” (default) or “No”.
Azure_Account_Name	(Optional) Use this field to establish mapping between the Azure account logs and the Cloud account resource. The Azure Account name can be retrieved from the system.displayname field in the Cloud Account Info tab.

If the deployment is successful and Enabled Activity Logs is set to “Yes”, logs should appear in the LM Logs page. These logs will be mapped to the Azure Cloud Account created in the LogicMonitor portal. If logs are not being forwarded, see Enabling Debug Logging.

If Enable Activity Logs was set to “No”, you need to manually configure forwarding of logs to the Event Hub. LogicMonitor also provides templates for this. See Sending Azure Resource Logs in the following.

Sending Azure Resource Logs

Once the Azure Function and Event Hub are deployed, Azure Function listens for logs from the Event Hub. The next step is to configure your resources and resource groups to send their logs to the Event Hub. For most Azure resources, this can be done by updating their diagnostic settings. 

Without the managed identity, you can still manually configure diagnostics settings of Azure resources to forward their logs to the Event Hub. This is only required if you are forwarding resource logs with the template provided.

Creating a Managed Identity

Select the Deploy to Azure button below to open the Azure Template and run the deployment. This creates the Managed Identity with the User Administrator role.

When deploying, you need to provide the following details in the template:

Parameter	Description
resource_group_region	(Required) The region where you created the resource group and deployed resources like the Event Hub, Function app, and so on.  For a list of the Azure regions in plain text, run the following command from PowerShell with the Azure CLI tools installed: az account list-locations -o table Note: The resource group and the resources within it must be in the same region as that of the Event Hub created when you deployed the Azure Function.
LM_Company_name	(Required) Your LogicMonitor company or account name in the target URL. This is only the <account> value, not the fully qualified domain name (FQDN). Example:  https://<account>.logicmonitor.com

Updating Diagnostic Settings

Select the Deploy to Azure button below to open the Azure Template and run the deployment. This lets you configure resource level log forwarding to the Event Hub. This template updates the diagnostic settings of selected resources in the resource group.

When deploying, you need to provide the following details in the template:

Parameter	Description
Resource Group	(Required) The resource group from where you want to forward logs to the Event Hub. For a list of Azure regions by their display names, see Azure geographies.
Subscription ID	(Required) The ID for the subscription which consists of all the resource groups.
LM_Company_name	(Required) Your LogicMonitor company or account name in the target URL. This is only the <account> value, not the fully qualified domain name (FQDN). Example:  https://<account>.logicmonitor.com
Force Update Tag	(Optional) Changing this value between template deployments forces the deployment script to re-execute.
Deployment Location	(Required) Select the region where this deployment is configured.

Sending Virtual Machine Logs

For virtual machines (VMs), the diagnostic settings will not be updated through the described template deployments. To forward system and application logs from VMs, you need to install and configure diagnostic extensions on the VMs. The following describes how to set up the log forwarding for Linux and Windows VMs.

Sending Linux VM Logs

Do the following to forward system and application logs from Linux VMs:

1. Install the diagnostic extension for Linux on the VM.

2. Install the Azure CLI.

3. Sign in to Azure using the Azure CLI: az login

4. Download the configuration script using the following command:

wget https://raw.githubusercontent.com/logicmonitor/lm-logs-azure/master/vm-config/configure-lad.sh

5. Run the configuration to create the storage account and configuration files needed by the diagnostic extension:

./configure-lad.sh <LM company name>

6. Update lad_public_settings.json to configure types of system logs and their levels (syslogEvents) and application logs (filelogs) to collect.

7. Run the following command to configure the extension:

az vm extension set --publisher Microsoft.Azure.Diagnostics --name LinuxDiagnostic --version 3.0 --resource-group <your VM's Resource Group name> --vm-name <your VM name> --protected-settings lad_protected_settings.json --settings lad_public_settings.json

Sending Windows VM Logs

Do the following to forward system and application logs from Windows VMs:

1. Install the diagnostic extension for Windows on the VM.

2. Install the Azure CLI using PowerShell: 

Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile .\AzureCLI.msi; Start-Process msiexec.exe -Wait -ArgumentList '/I AzureCLI.msi /quiet'; rm .\AzureCLI.msi

3. Sign in to Azure using the Azure CLI: az login

4. Download the configuration script using the following command:

Invoke-WebRequest -Uri https://raw.githubusercontent.com/logicmonitor/lm-logs-azure/master/vm-config/configure-wad.ps1 -OutFile .\configure-wad.ps1

5. Run the configuration to create the storage account and configuration files needed by the diagnostic extension:

.\configure-wad.ps1 -lm_company_name <LM company name>

6. Update wad_public_settings.json to configure types of event logs (Application, System, Setup, Security, and so on) and their levels (Info, Warning, Critical) to collect.

7. Run the following command to configure the extension:

az vm extension set --publisher Microsoft.Azure.Diagnostics --name IaaSDiagnostics --version 1.18 --resource-group <your VM's Resource Group name> --vm-name <your VM name> --protected-settings wad_protected_settings.json --settings wad_public_settings.json

Metadata for Azure Logs

The following table lists metadata fields for the Azure Logs integration with LM Logs. The integration looks for this data in the log records and adds the data to the logs along with the raw message string. 

Property	Description	LM Mapping	Default
time	The timestamp (UTC) for the event.	timestamp	Yes
level	Severity level for the event. Must be one of “Informational”, “Warning”, “Error”, or “Critical”.	severity	Yes
operationName	Name of the operation that this event represents.	activity_type	Yes
resourceId	Resource ID for the resource that emitted the event. For tenant services, this is in the form /tenants/tenant-id/providers/provider-name. When deploying the Azure integration using a template, you can add this field as a metadata key parameter to look for in the log record.	azure_resource_id	No
category	Log category for the event. Typical log categories are “Audit”, “Operational”, “Execution”, and “Request”.	category	Yes
ResourceType	Indicates from where the Azure logs are coming from. It also indicates where the service is deployed.	ResourceType	Yes

Troubleshooting

Follow these steps to troubleshoot issues with your Azure logs integration.

1. Confirm that the install process provisioned all the required resources: an Event Hub, a resource group, a storage account, and an Azure Function.

2. Confirm that logs are being sent to the Event Hub:

• Navigate to your Event Hub in the Azure portal and check that the incoming messages count is greater than 0.
• You can also check this for specific agents or applications by looking in their Azure Logs folder. For example, if you are running a Windows VM with a IaaSDiagnostics extension, its logs will be in the following Azure Logs directory (with version and wadid specified):
  C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.Diagnostics.IaaSDiagnostics<VERSION><WADID>\Configuration

3. Confirm that the Azure Function is running and forwarding logs to LogicMonitor. For more information, see Enabling debug logging.

• If the Function App is running and receiving logs, but you are not seeing the logs in LogicMonitor, confirm that the LM_Access_Key or LM_Access_Id provided is correct.
• If the Function App is not being executed, but logs are sent to the Event Hub, try to run the Azure function locally and check if it receives any log messages:
  • If the local function receives logs, stop and run the function on the Azure Portal. You can check its logs using the Azure CLI.
  • If the local function does not receive logs, check its connection string and the shared access policy of the Event Hub.

4. You can use PowerShell to send a test event from the log-enabled VM. On the configured device, enter the PowerShell prompt and run the following command: eventcreate /Id 500 /D "test error event for windows" /T ERROR /L Application

After the command runs, you should see the event in the LM Logs page.

Updating Template Parameters

You may need to update the template after deployment, for example to change credentials or parameters. You can do this in the Function App configuration by navigating to Function app lm-logs-{LM_Company_name}-{resource_group_region} → configuration → edit.

Enabling Debug Logging

You can enable Application Insights in the Function App to check whether it is receiving logs. For more information, see Enable streaming execution logs.

You can configure the application logging type and level using the Azure CLI webapp log config command. Example:

az webapp log config --resource-group <Azure Function's Resource Group name> --name <Azure Function name> --application-logging true --level verbose --detailed-error-messages true

After configuring application logging, you can see the logs using Azure CLI webapp log tail. Example:

az webapp log tail --resource-group <Azure Function's Resource Group name> --name <Azure Function name>

Removing Azure Resources

The Azure templates you ran to set up log ingestion create several resources, including the Event Hub which sends logs data to LM Logs. If needed, you can remove the LM Logs integration to stop the flow of data and associated costs.

Do the following:

1. In your Azure portal, navigate to the monitored VM > Activity log > Diagnostic settings > Edit setting (for the Logs Event Hub) and select Delete.

2. Delete the Event Hub which has the name and region name that you created during setup. This will cut off the logs flow from Azure to LM Logs.

3. You can remove all other resources created by template deployment such as the Function App, Managed Identity, App Insight, and Storage account. The names of these will following the Event Hub naming convention from the template. You can remove each item individually, or if they are in a resource group, you can remove the entire group.