What Are Spans in Distributed Tracing?

Distributed tracing is an essential process in the modern world of cloud-based applications. Tracing tracks and observes each service request an application makes across distributed systems. Developers may find distributed tracing most prevalent in microservice architectures where user requests pass through multiple services before providing the desired results.  

In this blog, we will explore the concept of spans within distributed tracing, delve into their composition and role in monitoring, and discuss best practices for effectively implementing span tracing to optimize the performance and reliability of cloud-based applications.

Introduction to span tracing

Span tracing is a critical component of distributed tracing, which is essential for monitoring and managing the performance of modern cloud-based applications. In distributed systems, particularly those utilizing microservice architectures, user requests often traverse multiple services before delivering the desired outcome.

Spans serve as the foundational elements of this tracing process, representing individual units of work within a trace. By breaking down each service request into smaller, time-measured individual operations, span tracing provides developers with granular visibility into the flow of requests across a distributed environment.

Understanding spans is crucial because they offer the detailed insights needed to diagnose performance bottlenecks, track the flow of requests, and ultimately optimize the reliability and efficiency of distributed applications.

Understanding distributed tracing 

Developers can acquire a comprehensive perspective of their software environment by combining distributed traces, metrics, events, and logs to optimize end-to-end monitoring and operations. Spans serve as the fundamental building blocks in distributed tracing and represent the smallest measure of work in the system.

DevOps engineers can set up distributed tracing across their operations by equipping their digital infrastructures with the necessary data collection and correlation tools, which should apply to the whole distributed system. 

The collected system data gives insightful information while offering the earliest signs of an anomalous event (e.g. unusually high latency) to drive faster responses.  

A closer look at spans in distributed tracing

A trace comprises a combination of spans, with each span serving as a timed operation as part of a workflow. Traces display the timestamp of each span, logging its start time and completion. Timestamps make it easier for users to understand the timeline of events that run within the software. Spans contain specific tags and information on the performed request, including potentially complex correlations between each span attribute. 

Parent Spans 

The parent, or root spans, occur at the start of a trace upon the initial service request and show the total time taken by a user request. Parent spans contain the end-to-end latency of the entire web request. For example, a parent span can measure the time it takes for a user to click on an online button (i.e., user request) for subscribing to a newsletter. During the process, errors and mistakes may occur, causing parent spans to stop. These spans branch out to child spans, which may divide into child spans of their own across the distributed system. It is important to note that parent spans may finish after a child span in asynchronous scenarios.  

Detailed visualization of parent-child references provides a clear breakdown of dependencies between spans and the timeline of every execution.

Developers should refer to every span – parent/root and subsequent child spans – in distributed tracing to gain a comprehensive breakdown of request performance throughout the entire lifecycle. 

Key components of a span

Every span contains specific descriptors that comprise the function and details of logical work performed in a system. A standard span in distributed tracing includes:

• A service/operation name – a title of the work performed
• Timestamps – a reference from the start to the end of the system process 
• A set of key:value span tags
• A group of key:value span logs
• SpanContext includes IDs that identify and monitor spans across multiple process boundaries and baggage items such as key:value pairs that cross process boundaries 
• References to Zero value or causally related spans

Span Tags

Essentially, span tags allow users to define customized annotations that facilitate querying, filtering, and other functions involving trace data. Examples of span tags include db.instances that identify a data host, serverID, userID, and HTTP response code. 

Developers may apply standard tags across common scenarios, including db.type (string tag), which refers to database type and peer.service (integer tag) that references a remote port. Key:value pairs provide spans with additional contexts, such as the specific operation it tracks. 

Tags provide developers with the specific information necessary for monitoring multi-dimensional queries that analyze a trace. For instance, with span tags, developers can quickly home in on the digital users facing errors or determine the API endpoints with the slowest performance. 

Developers should consider maintaining a simple naming convention for span tags to fulfill operations with ease and minimal confusion.  

Span Logs

Key:value span logs enable users to capture span-specific messages and other data input from an application. Users refer to span logs to document exact events and timelines in a trace. While tags apply to the whole span, logs refer to a “snapshot” of the trace.  

SpanContext

The SpanContext carries data across various points/boundaries in a process. Logically, a SpanContext is divided into two major components: user-level baggage and implementation-specific fields that provide context for the associated span instance. 

Essentially, baggage items are key:value pairs that cross process boundaries across distributed systems. Each instance of a baggage item contains valuable data that users may access throughout a trace. Developers can conveniently refer to the SpanContext for contextual metrics (e.g., service requests and duration) to facilitate troubleshooting and debugging processes.   

Best practices for effective span tracing

To maximize the benefits of span tracing, developers should follow best practices that enhance the accuracy and efficiency of their observability efforts. One key practice is to choose the right tags for each span, ensuring that they provide meaningful and actionable insights. Standardized tags such as http.method, db.type, and error help streamline database queries and filtering, making it easier to diagnose issues across distributed systems.

Managing span volume is another crucial aspect. In large-scale environments, excessive span data can lead to performance overhead and make traces harder to analyze. Developers should focus on capturing only the most relevant spans and data points, prioritizing critical paths and high-impact operations. By strategically reducing unnecessary spans, teams can maintain the performance of their tracing system while still gathering essential metrics.

Optimizing span data involves careful instrumentation, including the use of concise and consistent naming conventions for operations and services. Ensuring that each old or new span includes key-value pairs that accurately reflect the operation it represents will facilitate more precise monitoring and troubleshooting. Additionally, developers should regularly review and refine their span tracing setup, adjusting as their systems evolve to maintain optimal observability and performance.

Spans vs. traces: What’s the difference?

At its core, a trace represents a service or transaction under a distributed tracing structure. Spans represent a single logical structure within a given trace. Trace context is a significant component for traces within a distributed system as they provide components with easy identification through the use of unique IDs. 

Implementation of a trace context typically involves a four-step process:

1. Assigning a unique identifier to every user request within the distributed system
2. Applying a unique identification to each step within a trace
3. Encoding the contextual information of the identities
4. Transferring or propagating the encoded information between systems in an app environment

Traces capture the data of a user service request, including the errors, custom attributes, timelines of each event, and spans (i.e., tagged time intervals) that contain detailed metadata of logical work. Therefore, a trace ID refers to the execution path within a distributed system, while a span represents a single request within that execution path.  

Summary of spans in distributed tracing

Distributed tracing enables developers to track and observe service requests as they flow across multiple systems. A trace serves as performance data linked to a specific user request in a function, application, or microservice. Each trace comprises spans representing the smallest measurement of logical data and contains metrics that direct users to specific events. 

Specifically, a trace is the complete processing of a user request as it moves through every point of a distributed system (i.e., multiple endpoints/components located in separate remote locations).

Spans in distributed tracing provide IT specialists with granular control over data transferred between multiple end-users, improving the monitoring and diagnostics of IT operations. 

Advantages of spans and distributed tracing 

Modern digital operations involve complex technologies such as cloud, site reliability engineering (SRE), and serverless functions. Software managers and engineers typically accustomed to managing single services lack the technological capabilities to monitor system performance on such a scale. 

As such, remote online processes involve multiple user requests passing through distributed tracing to different functions and microservices, resulting in increased system speed and reduced delays in transforming code into products. 

Distributed tracing (and spans that serve as the essential logical measurement of work within these functions) optimizes observability strategies for developers within complex and remote app environments. 

Combining distributed tracing and a good understanding and implementation of spans allow software teams to pinpoint challenges or faults when managing user requests from multiple endpoints for expedited troubleshooting. Some immediate benefits of a distributed tracing and span-based approach include:

• Improved user experiences that lead to a more favorable business reputation and outcomes
• Holistic management of software systems that minimize downtime for maximum efficiency
• Creation of a proactive software environment that gives the company an edge over other companies in the increasingly competitive digital landscape 
• Accurate and responsive identification of user priorities so system managers can quickly determine the steps and measures to keep digital users/customers satisfied 

Developers may implement distributed tracing through various methods with differing difficulties. Choosing a method depends on the user’s current programming knowledge, infrastructure, and skill sets. Building a distributed tracing system from scratch provides the most flexibility and customization. 

Take your observability to the next level with LogicMonitor

At LogicMonitor, we help companies transform what’s next to deliver extraordinary employee and customer experiences. Our solutions empower you to achieve comprehensive monitoring and streamlined operations.