Distributed tracing

Introduction

Featureobserve In one binary, the request log line and its request id tell you what a request did. Across services, one click in the web app becomes an HTTP request to crm-api, a deals.move message to the deals service, a few queries, a deal.won event to two more services, and a queued mail. A trace ties them together: each step is a span with a start, a duration, attributes and a parent, and the spans of one click form one tree.

Navree records traces with OpenTelemetry, the open standard for them:

• every request, outgoing call, query, job, scheduled task, cache call and mail is a span, named and labeled with OpenTelemetry's conventions; messages, gRPC calls and GraphQL operations are spans too, with their packages;
• the trace context travels with every hop: HTTP calls in and out, queued jobs, and messages and gRPC calls with their packages;
• spans go to any OpenTelemetry backend over OTLP;
• log lines written inside a request or job carry its trace id;
• Lighthouse links each issue to the trace where it happened.

Propagation is always on, so trace ids appear in your logs and in the headers your app sends from the first request. Spans are built only when something reads them: the OTLP export, the stdout exporter, or SpanRecorder in a test. An app that exports nothing pays for a trace id per request and nothing else.

GET /api/v1/deals/{id}/move             crm-api         182 ms
+- send deals.move                      crm-api         171 ms
   +- process deals.move                deals           164 ms
      +- SELECT deals                   deals             2 ms
      +- UPDATE deals                   deals             3 ms
      +- publish deal.won               deals             2 ms
         +- process deal.won            notifications    38 ms
         |  +- INSERT notifications     notifications     2 ms
         |  +- job mail.send            notifications   310 ms
         |     +- mail send             notifications   305 ms
         +- process deal.won            activities       12 ms
            +- INSERT activities        activities        2 ms

Spans are made from Navree's telemetry events: when a request is answered, a query runs or a job attempt ends, ObserveModule turns the event into a span of the request's or job's trace.

Exporting traces

Tracing is part of ObserveModule, in the observe.tracing section. OTLP export needs the facade's otel feature. It is opt-in, not in full, because it adds the OpenTelemetry SDK, tonic and prost:

[dependencies]
navree = { git = "https://github.com/NavreeHQ/navree", features = ["otel"] }

otel turns on observe and adds the exporter; the default features stay on.

Then point the section at an OTLP endpoint, usually an OpenTelemetry Collector:

[observe.tracing]
endpoint = "http://127.0.0.1:4317"

That's all a service needs. Its spans carry service.name from micro.service (or app.name), are batched in the background and sent over OTLP/gRPC. Without the otel feature, an endpoint in the configuration fails the boot with a message that names the feature, and one from OTEL_EXPORTER_OTLP_ENDPOINT is ignored with a warning.

An invalid value fails the boot with a message that names the key: a sample_ratio outside 0 to 1, a timeout_ms or max_queue_size of 0, an endpoint that isn't an http:// or https:// URL.

As environment variables: APP_OBSERVE__TRACING__ENDPOINT=http://otel-collector:4317, APP_OBSERVE__TRACING__SAMPLE_RATIO=0.2. Navree also reads OpenTelemetry's standard variables for any setting the observe.tracing section doesn't set: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL (grpc or http/protobuf), OTEL_EXPORTER_OTLP_HEADERS and OTEL_RESOURCE_ATTRIBUTES (key=value,key=value), OTEL_SERVICE_NAME and OTEL_TRACES_SAMPLER_ARG.

In code, inject Arc<TracingOptions> (from navree::observe): ObserveModule provides the section as it was read, with the variables applied and the service name resolved.

Locally

Run a backend that accepts OTLP, such as Jaeger's all-in-one image, and point every service at it:

docker run --rm -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one

APP_OBSERVE__TRACING__ENDPOINT=http://127.0.0.1:4317

Click through the web app, then open http://localhost:16686 and pick crm-api: each request is a trace, with the spans of every service it reached.

Without a backend, set observe.tracing.stdout = true and read the spans in the terminal:

{
  "name": "GET /api/v1/deals/{id}",
  "kind": "server",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "a3ce929d0e0e4736",
  "parent_span_id": "00f067aa0ba902b7",
  "start": "2026-10-26T08:18:20.201542Z",
  "end": "2026-10-26T08:18:20.384068Z",
  "status": { "code": "unset" },
  "attributes": {
    "http.request.method": "GET",
    "http.response.status_code": 200,
    "http.route": "/api/v1/deals/{id}",
    "navree.request_id": "01a17749fe5f71caaee8974e482b5659",
    "url.path": "/api/v1/deals/0199c1d2-6f7e-7a31-b5d4-2f0c8e9a1b23"
  },
  "links": []
}

What is traced

Navree names spans and sets attributes the way OpenTelemetry's semantic conventions say, so backends recognize HTTP, database, messaging and RPC spans and draw them accordingly.

The first seven come with the core modules; the messaging, gRPC and GraphQL spans come with microservices, gRPC and GraphQL.

• A request's span covers the whole request. Queries, outgoing calls, cache calls and mail sent while it runs are its children; so are its log lines, through their trace id.
• An outgoing call is one span for all its attempts: http.request.resend_count says how many retries it took.
• A job attempt is one span. The queries and calls of the handler are its children, and the job itself is a child of the span that dispatched it (see context propagation).
• Each scheduled task run starts a new trace.
• Notifications, exceptions and log lines are telemetry events but not spans.

Some rules hold for every span:

• No values from your data. db.query.text is the statement with its placeholders, never the bound parameters (a literal you write into the SQL yourself stays in it); URLs lose their query string and credentials; payloads, bodies and headers are never recorded.
• Errors mark the span. A 5xx response, a failed outgoing call (a 5xx, or no response), a failed job attempt, a failed scheduled task or a failed query sets the span's status to error, with the error's message. A 4xx is a client's mistake, not the server's: an HTTP server span keeps status unset for it, as the conventions say.

Context propagation

Navree uses W3C Trace Context (traceparent and tracestate) on every hop, so spans from different services join the same trace, whatever language a service is written in:

• Incoming HTTP. navree-http reads traceparent and tracestate on every request. A valid header continues the caller's trace with a new span; a missing or malformed one starts a new trace. The observe.tracing keys enabled, trust_incoming and sample_ratio apply even without ObserveModule.
• Outgoing HTTP. Inside a request or job, every HttpClient request carries a traceparent for a new client span of the current trace, and the current tracestate. A request that sets its own traceparent header keeps it.
• Queued jobs. A job dispatched inside a request or job records that span's traceparent with the job (the jobs.trace_parent column, added by the queue's migrations). Each attempt continues that trace, whatever trust_incoming says, since the value came from your own app. Queued listeners and queued mail are jobs, so they follow. Jobs dispatched outside a request or job, and failed jobs retried with navree queue:retry, start a new trace.

An event published through the outbox keeps the context of the code that emitted it: the outbox stores it with the row, so the publish span sits under the request that changed the deal even when a relay on another instance sent it.

Browsers can join too. The app's CORS settings allow the traceparent and tracestate request headers, so a front end instrumented with OpenTelemetry's web SDK starts the trace at the click, and the backend's spans continue it.

Public edges

By default an incoming traceparent is trusted: the request's spans join the caller's trace, and the caller's sampling decision holds. That's right between your own services and for your own front end. On an API that strangers call, a client could pick trace ids or force sampling on. Set observe.tracing.trust_incoming = false on the edge service: each request then starts a new trace, sampled by your own sample_ratio, whose span keeps a link to the client's span so you can still follow it.

Your own code in a trace

Everything your handlers and jobs do through Navree is already in the trace: their queries, outgoing calls, cache calls, mail and the jobs they dispatch. Spans you open yourself with the tracing crate stay in your logs, where each line carries the trace id; they are not exported as spans.

To show users a reference they can quote to support, read the current trace id:

use navree::observe::current_trace_id;

let reference = current_trace_id().map(|id| id.to_string()).unwrap_or_default();

It is None outside a request, job or scheduled task run, and when tracing is off. The same value is navree::telemetry::current_trace_id(); TelemetryContext::current() (in navree::telemetry) gives the whole context: the request or job id, the trace, the user and the workspace.

Work you spawn onto another task leaves the request's context behind, since it is a task-local value. Wrap the future to keep it:

use navree::telemetry::TelemetryContext;

tokio::spawn(TelemetryContext::propagate(async move {
    // queries and calls here belong to the request's trace
    refresh_forecast(&db, deal_id).await
}));

Sampling

Recording every trace is right in development and for most apps. At high volume, record a share of them with observe.tracing.sample_ratio:

• the decision is made when a trace starts, from its trace id, so every service makes the same decision for the same trace;
• a request that arrives with a trusted traceparent follows the caller's decision, and a queued job follows the request that dispatched it, so traces are never cut in half;
• unsampled requests still get a trace id: it is in the logs and propagated to the next service, but none of their spans are built or exported.

A ratio below 1.0 drops error traces too, since nobody knows at the start that a request will fail. To keep every trace with an error and a share of the rest, keep sample_ratio = 1.0 and let an OpenTelemetry Collector make the decision at the end of each trace (tail sampling).

Logs and traces

Every log line written inside a request or job carries the trace id and the id of the request's or job's span. In the JSON format, they sit next to the request id:

{
  "timestamp": "2026-10-26T08:18:20.384068Z",
  "level": "INFO",
  "target": "crm::deals",
  "fields": { "message": "deal moved", "deal_id": "0199c1d2-6f7e-7a31-b5d4-2f0c8e9a1b23" },
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "a3ce929d0e0e4736",
  "span": {
    "method": "POST",
    "name": "request",
    "path": "/api/v1/deals/0199c1d2-6f7e-7a31-b5d4-2f0c8e9a1b23/move",
    "request_id": "01a17749fe5f71caaee8974e482b5659"
  }
}

Search your logs for a trace id to get the lines of every service that took part, or start from the logs and open the trace. To pass the request id on as well, add .with_request_id(&request_id) to an HttpClient request: it sends X-Request-Id, and the other Navree service uses it as its own request id. In development, Logbook shows the same ids on its entries.

Lighthouse

Lighthouse stores the trace id with every exception, slow request, slow query and failed job it groups into an issue. Tell it where your traces live:

[observe.tracing]
endpoint = "http://otel-collector:4317"
trace_url = "https://grafana.example.com/explore?traceId={trace_id}"

Each occurrence of an issue then links to its trace. Your own pages can build the same link: tracing_options.trace_link(&trace_id) on the injected TracingOptions returns trace_url with the id filled in, or None when trace_url isn't set.

In production

• Export to a Collector, as an agent on each host or a gateway for the cluster, rather than straight to a vendor. The Collector batches, retries, samples by tail, and lets you change backends without redeploying services.
• Export never slows requests. Spans are queued and sent in the background, in batches of up to 512 every two seconds. When the queue is full (max_queue_size), new spans are dropped and counted in the navree_tracing_spans_dropped_total metric on /metrics. A failed or timed-out export is logged as a warning under the navree::tracing target.
• Shutdown flushes. On Ctrl-C or SIGTERM, Navree exports what's left in the queue before the app stops.
• Resource attributes identify each process: service.name, service.instance.id (unique per process), service.version (the app.version setting, when you set it), plus observe.tracing.resource.
• TLS to an https:// endpoint uses rustls with the ring provider, as the rest of Navree does.

Testing

SpanRecorder (from navree::observe::testing) collects finished spans in memory. Add its module to the app, then assert on the tree. This test sends a request with a traceparent to a route that calls another service through a faked HttpClient:

use navree::axum::http::Method;
use navree::http_client::{FakeResponse, HttpClient};
use navree::observe::testing::SpanRecorder;
use navree::prelude::*;
use navree::testing::TestApp;
use serde_json::json;

#[tokio::test]
async fn the_rates_call_joins_the_request_trace() {
    let spans = SpanRecorder::new();
    let (client, server) = HttpClient::fake();
    server.on(Method::GET, "https://rates.example.com/*", FakeResponse::new(200));
    let app = TestApp::builder()
        .config("observe.metrics", false)
        .module(ObserveModule::from_config())
        .module(HttpClientModule::from_config())
        .module(RatesModule)
        .module(spans.module())
        .override_value(Arc::new(client))
        .build()
        .await;

    app.client()
        .get("/api/v1/rates/eur")
        .header("traceparent", "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01")
        .send()
        .await
        .assert_status(200);

    let request = spans.find("GET /api/v1/rates/{currency}");
    let call = spans.find("GET");
    assert_eq!(request.trace_id.to_string(), "4bf92f3577b34da6a3ce929d0e0e4736");
    assert_eq!(call.trace_id, request.trace_id);
    assert_eq!(call.parent_span_id, Some(request.span_id));
    assert_eq!(call.attributes["url.full"], json!("https://rates.example.com/v1/eur"));

    app.shutdown().await.unwrap();
}

spans.all() returns every finished span, spans.find(name) the first with that name (and fails the test with the names it has when there is none), spans.find_all(name) every one, and spans.clear() forgets them. Each RecordedSpan has its name, kind (SpanKind), trace_id, span_id, parent_span_id, start, end, status (SpanStatus::Unset or SpanStatus::Error { message }), attributes (JSON values by name) and links. The recorder sees the spans of sampled traces whether or not an endpoint is configured, so tests never export anything.

To assert on the events themselves (requests, queries, jobs, mail), use TelemetryRecorder.