Distributed tracing

Where does a slow request actually spend its time? Your web app’s response time looks normal, the database is quiet, and the request still takes four seconds. A single request can authenticate against one service, load data from an internal API, enqueue a background job, and call a third-party payment API before the response comes back. When it is slow or fails, the cause often lives in a different service than the one that reported the problem. Distributed tracing follows that request as it crosses those boundaries and brings the work from every service together under one trace, so you see the whole journey instead of piecing it together from each service on its own. Think of a group chat: every service replies in the same thread, so the request reads as one conversation instead of five separate chats.

​

What is distributed tracing?

When each service reports on its own, you see that a web action is slow, but not that an inventory service down the call chain caused it. In AppSignal, each of these services is a separate app or namespace. To follow one request across them, you have to open each one and line up the timestamps. Distributed tracing removes those boundaries. As a request moves from one service to the next, each service tags its work with the same trace identifier and records which piece of work called it. AppSignal uses those identifiers to reassemble the full path of the request, across every service that took part. With a distributed trace, you can:

• Follow one request from its entry point through every service it reached.
• See where the time went, even when the slow part runs in a different service or application.
• Find which service an error started in, not just where it surfaced.

​

Spans, traces, and services

AppSignal describes a trace with three building blocks.

• Span: a single unit of work, such as an incoming HTTP request, a database query, a call to another service, or a block of code you instrument yourself. A span has a name, a duration, and attributes, and it records the span that started it. Spans are the smallest thing a trace is made of.
• Trace: all the spans that belong to one end-to-end operation. Because each span knows its parent, the spans in a trace form a tree that describes the request from start to finish.
• Service: a named component that takes part in a trace, such as a web app, a background worker, or an internal API. You name a service with the service_name option (or the APPSIGNAL_SERVICE_NAME environment variable), and AppSignal uses that name to tell services apart in a trace. A single trace usually crosses several services, and those services can report to different AppSignal applications.

Within AppSignal, a trace is a kind of sample: the detailed record of one request, stored so you can open it and read it span by span. AppSignal derives your performance metrics, such as response time and throughput, from these traces, so a performance chart and a single trace are two views of the same data.

​

Availability and setup

Distributed tracing is available for Python, Ruby, and PHP first. Other languages, such as Elixir and JavaScript, are not supported yet. Whether you have to turn anything on depends on how your integration reports to AppSignal. Python and Ruby can report either through the agent bundled with the integration or through a collector, so collector mode has to be configured. PHP and a custom OpenTelemetry setup only ever report through a collector, so there is nothing to switch on.

• Python and Ruby: set the collector_endpoint option to a hosted collector’s URL rather than a self-hosted one. See collector configuration for Python, collector configuration for Ruby, and hosted vs self-hosted collectors for the difference between the two.
• PHP: nothing to configure. The PHP package reports through a collector already, so distributed tracing works once your app reports to AppSignal.
• A custom OpenTelemetry setup: nothing to configure. Exporting to a collector is the only way these setups report, so traces connect on their own.

Distributed tracing is built on OpenTelemetry, so it is not limited to the integrations above. Any OpenTelemetry-instrumented application can join a trace by exporting its data to a hosted collector. You can do this through an AppSignal integration such as the PHP package, or a custom OpenTelemetry SDK setup.

​

How does it work in AppSignal?

AppSignal builds distributed traces on OpenTelemetry, which handles the part that makes distributed tracing possible: context propagation. When an instrumented service calls another service, OpenTelemetry attaches the trace identifier and the calling span to the outgoing call. It travels as headers on an HTTP request, or as metadata on a queued job, and the receiving service reads it and continues the same trace. Each service reports the portion of the trace it is responsible for, called its subtrace. AppSignal receives these subtraces independently, often from different processes or hosts, and reassembles them into one trace using the shared trace identifier and the parent and child links between spans. You never configure the connection between services by hand. If a library is instrumented, its calls join the trace automatically. AppSignal samples traces rather than keeping every one, and it makes that decision across services: when a trace is kept, the work from every service in it is kept too, so the trace arrives whole instead of as scattered fragments. Sampling favors the traces you most want to see, the ones with errors and the slowest requests. Gaps can still appear at the edges, when a service does not report or its data arrives too late. You explore the result on the trace page in two ways: the service map, which shows the services in the trace and the calls between them, and the trace timeline, which lays every span out over time. The trace page covers how to read both.

​

Cross-application support

A service in AppSignal is not the same thing as an application. Several services can report to one AppSignal application, and a single trace can also cross services that belong to entirely separate applications. For example, a web app, an inventory service, and an audit service might all report to one AppSignal application, each under its own service name, while a separate service reports as its own application. A single user action, such as a checkout, can produce a trace that includes services from both applications. AppSignal’s service map treats this as one trace regardless of where each subtrace was reported. It resolves the application that owns each service and links every card to that application’s view of the same trace. To follow a request across an application boundary, select the next service’s card. You move from one application to the next along the same trace, without losing the thread. This works because context propagates across process and application boundaries the same way it does between services, and AppSignal matches subtraces by their shared trace identifier, not by which application or host they came from.

​

Related