You can use the Processor in the OpenTelemetry Collector to modify and enhance data that is sent to the Collector.

Using a Processor, you can leverage the following capabilities:

• Data Sampling—Choose what proportion of your application traces you want to send into LogicMonitor
• Data Scrubbing and Filtering—Remove or mask any data you do not want to store in LogicMonitor
• Span Aliasing—Rename spans to any chosen naming convention
• Custom Tagging—Attach further context and information to data

You can include a given Processor in the pipelines section within your collector configuration file, similar to the following probabilistic sampling:

service:
  extensions: [lmextension,health_check]
  pipelines:
    traces:
      receivers : [ otlp ]
      processors: [probablistic_sampling, batch]
      exporters : [otlphttp]
    metrics:
      receivers: [prometheus, hostmetrics]
      exporters: [lmexporter]

Data Sampling

You can make sampling changes for the OpenTelemetry Collector configuration directly in your LogicMonitor portal for both probabilistic (head-based) or tail-based sampling.

Probabilistic Sampling

Probabilistic, or head-based, sampling allows you to set a percentage-based sampling decision to control volume. In your LM OTel Collector configuration file, find the listed processors and adjust the sampling_percentage value between 1-100, as shown in the following example:

 probabilistic_sampler:
    sampling_percentage: 20

See Probabilistic Sampling Processor from OpenTelemetry for more information.

Tail-based sampling

Tail-based sampling allows you to make a sampling decision after traces are processed based on rules or conditions that you set.

You can set the following rules or conditions:

• Numeric_attribute: sample based on a number (For example, keeping data above a certain latency threshold)
• String_attribute: sample based on a string (For example, keeping data corresponding to a particular key-value pair)
• Rate_limiting: sample based on rate (For example, similar to head-based sampling above)

The following is an example of tail-based sampling:

tail_sampling:
    decision_wait: 10s
    num_traces: 10000
    policies:
      [
        {
          name: sample_error_http,
          type: numeric_attribute,
          numeric_attribute: {key: http.status_code, min_value: 300, max_value: 500}
        },
        {
          name: policy-2,
          type: string_attribute,
          string_attribute: {key: geo.address,values: [“US”,”CA”]}
        }
      ]

See Tail Sampling Processor from OpenTelemetry for more information.

Data Scrubbing and Filtering

You can modify ingested data if there is Personal Identifiable Information (PII) or other data that needs to be obfuscated before being stored in LogicMonitor.

The Attributes Processor supports the following actions:

• Insert: Adds custom key value pair metadata to a given span
• Update: Updates or changes an existing value
• Upsert: Inserts or updates a key depending on whether a key already exists
• Delete: Deletes an key value pair attribute from a span
• Hash: Hashes (SHA1) an existing value
• Extract: Extracts values using regex to target keys

The following example displays the insert, update, and upsert actions for the OpenTelemetry Attributes Processor (These actions require key, action, and value OR from_attribute):

processors:

  attributes/example:

    actions:

      - key: <key>

        action: {insert, update, upsert}
        value: <value>
        from_attribute: <other key>

The following example displays the delete and hash actions for the OpenTelemetry Attributes Processor (These actions require key and action):

processors:

  attributes/example:

    actions:

      - key: <key>
        action: {delete, hash}

The following example displays the extract action for the OpenTelemetry Attributes Processor (This action requires key, action, and pattern):

processors:

  attributes/example:

    actions:
      - key: <key>
        pattern:
        action: extract

See Attributes Processor from OpenTelemetry for more information.

Span Filtering

You can include and exclude the spans based on services and span names. Both service and span name can be provided at the same time but only those spans matching both will be included/excluded. Match Type can either be a strict or regex pattern.

The following example demonstrates how to include and exclude spans:

span/includeexclude:
  include:
    match_type: regexp
    services: ["productcatalog"]
    span_names: ["*Products*"]
  exclude:
    match_type: strict
    span_names: ["/ListProducts"]
  name:
    from_attributes: [net.peer.name]

A span is renamed to the value for the from_attribute if it meets the following requirements:

• Span name contains  “Products”
• Span not named “ListProducts”
• Service named “productcatalog”

Span Aliasing

The Span Processor is a separate process in OpenTelemetry and supports Span Aliasing. You can use the Span Process to make modifications to an existing span name.

The following attributes are supported:

• name: The name of the span to modify
• from_attributes: Specifies the key for the value that is used to create a new span name
• (Optional attribute) separator: A string, which is specified will be used to split values

The following example demonstrates how you can use the Span Processor to change the span name from the existing attribute values with a separator:

processors:
  span:
    name:
      from_attributes: ["http.method", "net.peer.name"]
      separator: "::"

In the example, you can do the following:

• Span name before processor: “HTTP GET”
• Attributes in the input span: { “net.peer.name”: “scientist-name-service”, “http.method”: “GET”}

After processing, the new span name is GET::scientist-name-service.

Custom Tagging

You can attach further context and information to trace data.

Custom Tagging using the Collector

You can pass custom tags to all spans from a given collector by applying a startup parameter. This can be done within the setup process or by modifying your configuration file.

Adding the OTEL_RESOURCE_ATTRIBUTES parameter allows you to set any attributes or tags with every span processed as demonstrated in the following example:

OTEL_RESOURCE_ATTRIBUTES="service.name=login"
OTEL_RESOURCE_ATTRIBUTES="application.name=logicmonitor"

Custom Tagging for Traces

Regardless of the instrumentation language, LogicMonitor requires the following custom tags to be specified during implementation if you want the traces to be mapped to existing monitored resources:

• IP 
• resource.type (which should be set to kubernetes-pod, cloud, or host)
• hoste.name (which should be the pod name for Kubernetes)

Capturing tags statically on your traces is important for mapping within LogicMonitor, but you may want to include additional attributes for your trace data to find more correlations and context while exploring and searching your data. For example, capturing metadata such as customer tier, product category, or geographical data provides important context to better understand what issue might be occurring.

You can set a key-value pair or group of pairs on any manually instrumented span. The following is an example for Java:

Span span = tracer.spanBuilder("/resource/path").setSpanKind(SpanKind.CLIENT).startSpan();
span.setAttribute("http.method", "GET");
span.setAttribute("http.url", url.toString());

See Span Attributes and Trace Semantic Versions from OpenTelemetry for more information.