> ## Documentation Index
> Fetch the complete documentation index at: https://docs.appsignal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Distributed tracing

> Follow one request across services and applications: how AppSignal assembles each service's subtrace into one trace.

Where does a slow request actually spend its time? Your web app's response time
looks normal, the database is quiet, and the request still takes four seconds.

A single request can authenticate against one service, load data from an internal
API, enqueue a background job, and call a third-party payment API before the
response comes back. When it is slow or fails, the cause often lives in a
different service than the one that reported the problem.

Distributed tracing follows that request as it crosses those boundaries and
brings the work from every service together under one trace, so you see the whole
journey instead of piecing it together from each service on its own. Think of a
group chat: every service replies in the same thread, so the request reads as one
conversation instead of five separate chats.

## What is distributed tracing?

When each service reports on its own, you see that a web action is slow, but not
that an inventory service down the call chain caused it. In AppSignal, each of
these services is a separate app or namespace. To follow one request across them,
you have to open each one and line up the timestamps.

Distributed tracing removes those boundaries. As a request moves from one service
to the next, each service tags its work with the same trace identifier and
records which piece of work called it. AppSignal uses those identifiers to
reassemble the full path of the request, across every service that took part.

With a distributed trace, you can:

* Follow one request from its entry point through every service it reached.
* See where the time went, even when the slow part runs in a different service
  or application.
* Find which service an error started in, not just where it surfaced.

## Spans, traces, and services

AppSignal describes a trace with three building blocks.

* **Span**: a single unit of work, such as an incoming HTTP request, a database
  query, a call to another service, or a block of code you instrument yourself.
  A span has a name, a duration, and attributes, and it records the span that
  started it. Spans are the smallest thing a trace is made of.
* **Trace**: all the spans that belong to one end-to-end operation. Because each
  span knows its parent, the spans in a trace form a tree that describes the
  request from start to finish.
* **Service**: a named component that takes part in a trace, such as a web app,
  a background worker, or an internal API. You name a service with the
  `service_name` option (or the `APPSIGNAL_SERVICE_NAME` environment variable), and
  AppSignal uses that name to tell services apart in a trace. A single trace
  usually crosses several services, and those services can report to different
  AppSignal applications.

<Frame caption="The spans in a trace form a tree. Each span records the span that started it, from the root span down.">
  <img src="https://mintcdn.com/appsignal-715f5a51/ydExjxcWacNDpILG/assets/images/diagrams/distributed-tracing/span-tree.png?fit=max&auto=format&n=ydExjxcWacNDpILG&q=85&s=ada68a41a3078ae6ab60c8bf9bf578a3" alt="A trace's spans arranged as a tree, from the root span down through parent and child spans" width="1672" height="941" data-path="assets/images/diagrams/distributed-tracing/span-tree.png" />
</Frame>

<Note>
  AppSignal builds traces on top of OpenTelemetry, so these terms line up with
  OpenTelemetry's. AppSignal groups each trace under an
  [action](/appsignal/terminology#actions) and a
  [namespace](/application/namespaces), based on the endpoint, job, or task the
  trace belongs to.
</Note>

Within AppSignal, a trace is a kind of
[sample](/appsignal/terminology#samples): the detailed record of one request,
stored so you can open it and read it span by span. AppSignal derives your
performance metrics, such as response time and throughput, from these traces, so
a performance chart and a single trace are two views of the same data.

## Availability and setup

Distributed tracing is available for Python, Ruby, and PHP first. Other
languages, such as Elixir and JavaScript, are not supported yet.

<Note>
  Distributed tracing is an **experimental feature only available** when using
  the AppSignal hosted collector.
</Note>

Whether you have to turn anything on depends on how your integration reports to
AppSignal. Python and Ruby can report either through the agent bundled with the
integration or through a collector, so collector mode has to be configured. PHP
and a custom OpenTelemetry setup only ever report through a collector, so there
is nothing to switch on.

* **Python and Ruby**: set the `collector_endpoint` option to a hosted
  collector's URL rather than a self-hosted one. See [collector configuration
  for Python](/python/configuration/collector), [collector configuration for
  Ruby](/ruby/configuration/collector), and [hosted vs self-hosted
  collectors](/collector/hosted-vs-self-hosted) for the difference between the
  two.
* **PHP**: nothing to configure. The PHP package reports through a collector
  already, so distributed tracing works once your app reports to AppSignal.
* **A custom OpenTelemetry setup**: nothing to configure. Exporting to a
  collector is the only way these setups report, so traces connect on their
  own.

Distributed tracing is built on OpenTelemetry, so it is not limited to the
integrations above. Any OpenTelemetry-instrumented application can join a trace
by exporting its data to a hosted collector. You can do this through an
AppSignal integration such as the PHP package, or a [custom OpenTelemetry SDK
setup](/opentelemetry/installation).

## How does it work in AppSignal?

AppSignal builds distributed traces on OpenTelemetry, which handles the part
that makes distributed tracing possible: **context propagation**. When an
instrumented service calls another service, OpenTelemetry attaches the trace
identifier and the calling span to the outgoing call. It travels as headers on
an HTTP request, or as metadata on a queued job, and the receiving service
reads it and continues the same trace.

Each service reports the portion of the trace it is responsible for, called its
**subtrace**. AppSignal receives these subtraces independently, often from
different processes or hosts, and reassembles them into one trace using the
shared trace identifier and the parent and child links between spans. You never
configure the connection between services by hand. If a library is instrumented,
its calls join the trace automatically.

<Frame caption="One trace spans several services. Each service reports a subtrace, and AppSignal assembles the subtraces into a single trace.">
  <img src="https://mintcdn.com/appsignal-715f5a51/ydExjxcWacNDpILG/assets/images/diagrams/distributed-tracing/trace-across-services.png?fit=max&auto=format&n=ydExjxcWacNDpILG&q=85&s=c2b8f1d2a9e12f272fdc4583604a4103" alt="One trace across three services, each reporting its own subtrace and the spans within it" width="1672" height="941" data-path="assets/images/diagrams/distributed-tracing/trace-across-services.png" />
</Frame>

AppSignal samples traces rather than keeping every one, and it makes that
decision across services: when a trace is kept, the work from every service in it
is kept too, so the trace arrives whole instead of as scattered fragments.
Sampling favors the traces you most want to see, the ones with errors and the
slowest requests. Gaps can still appear at the edges, when a service does not
report or its data arrives too late.

You explore the result on the [trace page](/trace-page) in
two ways: the **service map**, which shows the services in the trace and the
calls between them, and the **trace timeline**, which lays every span out over
time. The trace page covers how to read both.

### Cross-application support

A service in AppSignal is not the same thing as an application. Several services
can report to one AppSignal application, and a single trace can also cross
services that belong to entirely separate applications.

For example, a web app, an inventory service, and an audit service might all
report to one AppSignal application, each under its own service name, while a
separate service reports as its own application. A single user action, such as a
checkout, can produce a trace that includes services from both applications.

AppSignal's service map treats this as one trace regardless of where each subtrace was
reported. It resolves the application that owns each service and links every
card to that application's view of the same trace. To follow a request across an
application boundary, select the next service's card. You move from one
application to the next along the same trace, without losing the thread.

This works because context propagates across process and application boundaries
the same way it does between services, and AppSignal matches subtraces by their
shared trace identifier, not by which application or host they came from.

## Related

<CardGroup cols={2}>
  <Card title="Trace page" href="/trace-page">
    Read one trace: its timeline, span details, tags, related logs, and triggers.
  </Card>

  <Card title="Performance / tracing" href="/performance-tracing">
    Inspect the spans and events inside a single trace.
  </Card>

  <Card title="Custom instrumentation" href="/custom-instrumentation">
    Add your own spans to the trace when the defaults are not enough.
  </Card>

  <Card title="Namespaces" href="/application/namespaces">
    How AppSignal groups traces by endpoint, job, and task.
  </Card>

  <Card title="Labs" href="/labs">
    Try out distributed tracing and other experimental features.
  </Card>
</CardGroup>
