Distributed request tracing
Trace a request across microservices using trace IDs and span metadata.
In a monolith, one request hits one server. You log "request took 500ms" and you're done. In microservices, one user action bounces across 5–10 services, databases, and caches. A single slow request turns into a mystery: was it the API gateway, service A, service B, the database, or the network between them? Distributed tracing gives you a thread through the maze.
This template shows a request entering at the API gateway with a unique trace ID, flowing through two services that call each other and a database, and exiting back to the client. Each step logs its own span (duration) under the same trace ID. A monitoring dashboard shows you the whole timeline and surfaces the slowest span — usually the culprit.
When to use this template
- Designing a microservices system — add tracing from day one. Retrofitting is much harder once services are tangled.
- Debugging "the system is slow" complaints — when a customer says the homepage loads slowly, pull up the trace and pinpoint the cause in 30 seconds instead of 30 minutes.
- Capacity planning — trace data shows which services are being hit most and where latency compounds. Optimize the bottlenecks first.
How to adapt it
Expand this template to your own service topology:
- Add parallel calls — if Service A calls Service B and Service C in parallel, draw two arrows leaving Service A simultaneously and converging at the client. Parallel spans take the longest single path, not the sum.
- Include async work — if a span enqueues a background job and returns immediately, show the response arriving before the job finishes. Explain that the trace captures the user-facing part; async jobs have separate traces.
- Show error propagation — if a database timeout causes Service A to fail, add a
Xmark or 'timeout' label on the database span and chain it forward: Service A catches the timeout, retries, and succeeds. Tracing captures the whole story.
Mermaid code
Copy it anywhere Mermaid is supported — GitHub, Notion, or your docs.
sequenceDiagram
actor User
participant Client
participant API Gateway
participant Service A
participant Service B
participant DB
User->>Client: Click button
Client->>API Gateway: GET /orders (trace_id: abc123)
API Gateway->>Service A: Forward with trace_id
Service A->>Service B: Query user profile (trace_id)
Service B->>DB: SELECT user WHERE id=42
DB-->>Service B: User data
Service B-->>Service A: Profile (span: 5ms)
Service A->>DB: SELECT orders (trace_id)
DB-->>Service A: Orders (span: 12ms)
Service A-->>API Gateway: Aggregated response
API Gateway-->>Client: JSON (total: 45ms)
Client-->>User: Render orders
Frequently asked questions
- What is distributed tracing and how does it solve microservice debugging?
- Distributed tracing assigns a unique trace ID to each user request, passes it through every service call, and logs how long each step takes (spans). When a user reports 'my orders page is slow', you can look up the trace ID and see exactly which service is the bottleneck — was it the API gateway, user-profile lookup, or database query? Without tracing, debugging is guesswork across logs.
- What is the difference between trace ID, span ID, and span duration?
- A trace ID follows one user request across all services (e.g., abc123). A span ID identifies one operation within a service (e.g., 'query user database'). Span duration is how long that operation took (e.g., 5ms). Multiple spans make up one trace. A slow trace is usually one span with a long duration; tracing tells you which one.
- How do I get trace data into my logs and monitoring system?
- Add middleware to your API gateway that generates trace IDs (UUID or similar), includes it in response headers, and logs it. Every downstream service receives the trace ID in request headers and includes it in their own logs. Your log aggregator (Datadog, New Relic, Grafana, etc.) correlates all logs with the same trace ID into one visualization. Query by trace ID to see the full request journey.
- What's the difference between distributed tracing and APM?
- APM (Application Performance Monitoring) is the umbrella — it includes metrics, logs, and traces. Tracing is the slice that follows requests. You often start with APM metrics ('95th percentile latency is 200ms'), then use tracing to debug ('which service caused this latency?'). Tools like Datadog, Honeycomb, and Jaeger do both.
Related templates
Observability stack architecture
Metrics, logs, traces, dashboards, and alerting for production systems.
API gateway request routing
Gateway routing requests to microservices based on path and headers.
API rate limiting sequence
Client, gateway, and limiter handling a 429 with Retry-After.