After a Thursday deploy, p95 response time on a support assistant goes from 2.1 seconds to 4.8, and the token bill climbs with it. Every request still returns HTTP 200. The application logs show a prompt in and an answer out, and nothing in between. LLM observability is the practice of recording each step of an LLM request as a span in a single trace, with token counts, cost and timing attached, so a regression points to one step. Without it, the team above can only guess which of five steps got slower.
One Request, Five Spans
Received
And Rerank
Assembly
Call
Response
← Scroll to see full pipeline →
What should one LLM observability trace contain?
One trace per user request, one child span per step, and the same attribute names everywhere. The OpenTelemetry GenAI semantic conventions supply those names: gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and the cache-read counters. Use them even with a single vendor, because every backend that reads the spec can then parse your traces without custom mapping.
Retrieval and prompt assembly
Record query latency, the number of chunks returned and the final prompt size in tokens. Prompt size belongs here, not only on the model span, because this is where it grows. A RAG pipeline that starts returning twelve chunks instead of five shows up as input tokens climbing on every downstream call.
The model call
Record model, token counts, finish reason and two timings: time to first token (TTFT) and total duration. Streaming makes TTFT the number users feel. Total duration is the one your concurrency limits feel.
1import time 2from opentelemetry import trace 3from opentelemetry.sdk.trace import TracerProvider 4from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter 5 6provider = TracerProvider() 7provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter())) 8trace.set_tracer_provider(provider) 9tracer = trace.get_tracer("support-bot") 10 11def fake_stream(): # stands in for a streaming API response 12 time.sleep(0.05) 13 for tok in ["Refund", " issued", "."]: 14 time.sleep(0.01) 15 yield tok 16 17with tracer.start_as_current_span("chat gpt-4o-mini") as span: 18 span.set_attribute("gen_ai.operation.name", "chat") 19 span.set_attribute("gen_ai.request.model", "gpt-4o-mini") 20 start, first, n_out = time.perf_counter(), None, 0 21 for tok in fake_stream(): 22 first = first or time.perf_counter() 23 n_out += 1 24 span.set_attribute("gen_ai.usage.input_tokens", 1840) 25 span.set_attribute("gen_ai.usage.output_tokens", n_out) 26 span.set_attribute("app.ttft_ms", round((first - start) * 1000)) 27 span.set_attribute("app.tenant_id", "acme-blr") 28# console output includes: "app.ttft_ms": 60, "gen_ai.usage.output_tokens": 3
The GenAI conventions are still marked as in development, so pin the semantic-conventions version you emit and expect a rename or two. The app.* attributes are your own: tenant, feature and prompt version are what turn a trace into a cost report.
How do you account for tokens per tenant and feature?
Token accounting is where LLM observability pays for itself. Sum token attributes from model spans, grouped by the tenant and feature attributes you set on the parent span. Price input, cached input and output separately, since providers bill them at different rates. A prompt version attribute is worth adding too: when a template change doubles average input tokens, the cost report names the version, not just the week.
Read the spec before writing the cost query. It says gen_ai.usage.input_tokens should include cached tokens, so the cache-read count is a subset of it, not an addition to it.
Bill (input minus cache read) at the full rate and cache read at the discounted rate. Adding the two double-counts every cached token and inflates the report.
How do you set a latency budget for an LLM request?
Start from what the user waits for, then divide it among spans. With streaming, that is time to the first visible token, which covers retrieval, assembly and model TTFT. A budget written per span is what makes a regression attributable: the deploy above either blew the retrieval allowance or the model allowance, and the trace says which. Count tokens consistently across those spans, using the same tokenizer logic covered in tokens and the context window.
An example split for a 1.5-second p95 target: 250 ms for retrieval, 150 ms for reranking, 50 ms for prompt assembly and 1,000 ms for model TTFT. Alert on each span’s p95 against its own allowance, not only on the end-to-end number.
- LLM observability records each step of a request as a span in one trace, with tokens, cost and timing attached.
- The OpenTelemetry GenAI conventions define shared attribute names such as gen_ai.request.model and gen_ai.usage.input_tokens.
- Under the spec, input token counts include cached tokens, so cost queries must subtract cache reads before applying the full rate.
- Tenant, feature and prompt-version attributes on the parent span turn raw traces into a per-team cost report.
- A latency budget split per span, with time to first token as the user-facing target, makes every regression attributable to one step.
Conclusion
Roll out LLM observability in order: the model span first, then retrieval, then tools, and set budgets once a week of traces exists. Traces tell you what got slower or costlier. Whether answers got worse is a separate question, covered in metrics for agent reliability and accuracy.
Frequently Asked Questions
What is LLM observability?
LLM observability is the practice of recording every step of an LLM request, such as retrieval, prompt assembly, the model call and tool use, as spans in a single trace. Each span carries token counts, cost and timing, so a slowdown or a cost increase can be traced to one specific step.
What are the OpenTelemetry GenAI semantic conventions?
They are standard attribute names for telemetry from generative AI calls, such as gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Using them lets any compatible backend read your traces without custom mapping. The conventions are still marked as in development, so pin the version you emit.
What is time to first token?
Time to first token, or TTFT, is the delay between sending a request and receiving the first streamed token of the response. With streaming, it is the latency users actually perceive. Total generation time matters separately for throughput and concurrency, but TTFT is the right target for a user-facing latency budget.
How do I track LLM cost per customer?
Attach a tenant or customer ID attribute to the parent span of each request, then sum input, cached and output tokens from the model spans beneath it. Price each token type at its own rate, and subtract cached tokens from input before applying the full input rate, because input counts include them.
Do I need a dedicated LLM observability tool?
Not always. If you already run an OpenTelemetry backend, emitting GenAI attributes may be enough for tracing and cost reports. Dedicated tools such as Langfuse or Arize Phoenix add prompt-version views, evaluation workflows and cost dashboards that general-purpose tracing backends do not provide by default.