Advanced12 min readAI for Business

Observability for LLM apps: tracing, costs, latency, quality drift

Extend ordinary observability with multi-step traces, attributable cost, prompt and model versions, evaluated quality signals, privacy controls, and workload-derived alerts.

What you should be able to do

Keep ordinary service telemetry, then add model, prompt, retrieval, tool, cost, and evaluated-quality context. Capture raw prompts or outputs only for a justified purpose with access, minimisation, retention, and deletion controls.

Saved only in this browser.
In this article

LLM applications retain ordinary software failure modes and add probabilistic output, model and prompt drift, tool behavior, retrieval quality, and usage-sensitive cost. Some failures produce stack traces; others appear only in evaluated outputs, user reports, or changed distributions.

A successful API status does not establish that an answer is supported by its sources. Existing observability platforms can carry semantic evaluations and tool traces when you instrument them; the missing piece is the application’s definition of success and failure, not necessarily a new vendor.

Production LLM systems need additional semantic data on top of ordinary traces, metrics, and logs. Start with the OpenTelemetry generative-AI semantic conventions and extend only where your product needs more detail. The event shape and incident below are illustrative designs, not measurements from a deployed service.

The observability stack

For a source-grounded support assistant, join four records: the request outcome, the model call, the retrieval decision and the validated answer. Give the request a trace ID and record the prompt, model, source/index and policy versions. A retrieved document ID is useful only if the responder is authorised to inspect it; trace search must enforce the same tenant boundary.

Keep transport status separate from task status. “HTTP success,” “schema valid,” “source supported” and “delivered” describe different events. A source-support failure can occur after a perfectly healthy model call.

Call-level instrumentation

Every LLM call should produce an approved metadata record. The commented payload and identity fields below are optional sensitive fields, not defaults:

{
  "call_id": "uuid",
  "timestamp": "2026-08-04T14:23:45Z",
  "trace_id": "uuid",  // for grouping into traces
  "span_id": "uuid",   // for parent-child relations
  "feature": "summarize_document",
  "prompt_version": "v3.2",
  "model": "provider-model-revision",
  "provider": "anthropic",
  "input_messages": null,  // optional; approved sampled/redacted payload only
  "output_message": null,  // optional; approved sampled/redacted payload only
  "input_tokens": 1842,
  "output_tokens": 384,
  "total_tokens": 2226,
  "cost_usd": 0.0084,
  "latency_ms": 2340,
  "first_token_ms": 1240,  // streaming
  "status": "success",
  "error": null,
  "subject_ref": "pseudonymous_ref",  // optional, purpose-specific lookup
  "tenant_ref": "tenant_scoped_ref",
  "metadata": {
    "session_id": "...",
    "experiment_arm": "v3_test"
  }
}

This is an illustrative application event, not a mandatory schema. Capture the minimum needed for a defined operational purpose. Raw prompts and outputs are opt-in sensitive payloads, not default telemetry.

Key implementation choices:

Where to log. Options:

  • A dedicated observability tool (Helicone, LangSmith, Phoenix, Braintrust, Arize).
  • A general observability platform with LLM extensions (Datadog LLM Observability, Sentry).
  • Your own logs/database.

Choose from requirements: OpenTelemetry export, trace and tool-call visualization, data location, self-hosting, redaction, access control, retention/deletion APIs, model-price maintenance, evaluation support, and total cost. A dedicated tool, an existing APM, or a small owned implementation can each be correct; test export and deletion before committing.

How to instrument. Options:

  • A proxy that sits between your app and the LLM provider (Helicone’s model).
  • A wrapping SDK in your application code.
  • A wrapper class that you call manually for each LLM invocation.

Proxies centralize instrumentation but add a network hop and failure domain. SDK wrapping keeps instrumentation in process but couples code to the integration. Manual wrappers can be precise but require coverage tests. Measure overhead and failure behavior for the selected approach.

A pragmatic approach: SDK wrapping at the boundary where your app calls the LLM. One place to instrument: everything else flows through.

What to log. Apply data classification and retention before enabling payload capture. Under GDPR, Article 5 data minimisation and storage-limitation principles still apply to observability data.

  • Prefer prompt version, hashes, token counts, policy decisions, and derived metrics over raw content.
  • If payload capture is justified, sample it, redact it before export, encrypt it, restrict access, and set a short retention period.
  • Do not keep an automatically retrievable raw original merely because a redacted copy was logged; that recreates the sensitive-data store and needs its own lawful purpose and controls.
  • Don’t log credentials, even in error paths.
  • For streaming, log both first-token-latency and total-latency.

Trace-level instrumentation

A single user action often involves many LLM calls. Without trace-level instrumentation, you have a thousand call logs and no way to know which calls were part of which user action.

Implementation:

Trace ID generation. Generate a unique trace ID at the start of a user request. Pass it through all subsequent calls.

Parent-child spans. Within a trace, each call has a span ID and (optionally) a parent span ID. This creates a tree showing the call hierarchy.

Operation naming. Each span is named (“summarize_document”, “extract_entities”, “tool_call:search”). The trace shows the full operation chain.

A trace view in your UI:

Trace abc-123 (12.3s total)
├─ classify_intent (450ms) [small-model-revision]
├─ retrieve_documents (1.2s) [embedding + search]
├─ generate_response (8.5s) [generation-model-revision]
│   ├─ tool_call: search_internal (320ms)
│   ├─ tool_call: lookup_customer (180ms)
│   └─ generate_final_text (7.5s)
└─ judge_response_quality (2.1s) [claude-haiku-4-5]

Now you can see what your system actually did for this user request. You can find slow calls, expensive calls, failed calls, in context.

Candidate implementations include dedicated LLM-observability products and general APM systems using OpenTelemetry. Verify trace propagation, tool-call representation, export, redaction, deletion, access control, and failure behavior in a representative trial rather than relying on a product list.

Application-level metrics

For this support workflow, the main views are: requests by final outcome; unsupported-claim rate in an approved reviewed sample; retrieval and validation failures by version; end-to-end latency; and cost per completed, accepted answer. Break those down by relevant language, document type and tenant where authorised. Report the sample denominator and excluded traffic with quality figures.

Do not average away a small forbidden-disclosure slice. A lower mean latency or cheaper call does not compensate for an unauthorised answer. Likewise, a rise in refusals is not automatically a defect: check whether the traffic includes more out-of-scope requests.

Quality monitoring

The hardest layer: automated quality assessment.

For offline evaluations, you run on a defined dataset. For online quality monitoring, you assess production traffic.

Approaches:

LLM-as-judge on a sample. Define a sample from traffic volume, risk slices, privacy constraints, detection target, and budget. Run a calibrated judge on eligible examples, preserve human adjudication for disputed or high-impact cases, and track the result by prompt/model version. Judge models have position, verbosity, and self-preference biases; they are measurements to validate, not ground truth.

Implicit signals. Track regenerations, abandonments, error rates, time-to-completion, follow-up message frequency. These are weak signals but cheap. Use as leading indicators.

Explicit user feedback. Thumbs up/down and issue reports can reveal failures, but participation is selective and satisfaction does not prove factual accuracy.

Pattern signals. Phrase or repetition detectors can flag responses for review, including refusals. A refusal can be correct for an out-of-scope request; evaluate the signal against the task and policy before classifying it as an error.

An implementation record should state the sampling rule, excluded data, rubric, judge version, human calibration set, uncertainty, aggregation window, alert threshold, and cost ceiling. Derive change thresholds from repeated baseline runs and the severity of missed regressions; a copied percentage has no statistical or business meaning for a different workload.

Cost observability

Retries, loops, unexpectedly long inputs or outputs, routing changes, and provider-price changes can raise cost quickly. Detect each mechanism directly instead of assuming a multiplier.

Cost observability layers:

Per-call cost. Every call’s cost computed at log time. Aggregations available immediately.

Budget alerts. Daily, weekly, or monthly budgets per feature and tenant, with warning and enforcement thresholds chosen from forecast variance and business criticality.

Anomaly detection. Compare cost and usage with the correct traffic-mix baseline, and separately cap pathological single-run loops, tokens, tools, retries, duration, and spend.

Cost attribution. Per-feature, per-tenant, per-user costs. Find the heavy hitters.

Forecast. Based on current trajectory, what will the bill be at end-of-month?

A useful dashboard view: a single panel showing today’s cost vs the rest of this week vs last week, broken down by feature.

Set budget limits from measured traffic and business criticality. Prefer staged controls (alert, throttle, downgrade a non-critical path, then circuit-break), because a universal “shut off at 10×” rule may react far too late or take down an essential workflow.

Latency observability

LLM latency is more nuanced than typical APIs:

Total latency. From request to final response.

Time-to-first-token (TTFT). For streaming, when does the user see the first character? This measures the start of a streamed reply; its importance relative to completion time depends on the task and interface.

Time-to-last-token (TTLT). How long until the response is complete?

Tokens-per-second. Output rate. Some models stream slower than others.

Tool call latency. For agent flows, time spent in tool calls vs LLM calls.

Track all of these. Different optimization strategies target different metrics.

For user-facing chat, TTFT is one important interaction metric; total completion time, output rate, interruptions, task success, and accessibility also matter. Set objectives from observed user behavior rather than assuming one latency metric dominates every interface.

For batch: total latency matters; throughput matters more.

For agents, measure tool, queue and model time separately before deciding which component to optimise.

Error observability

LLM-specific errors:

API errors. Rate limits, auth failures, server errors. Same as any API.

Validation errors. Structured output didn’t conform to schema. Track frequency by feature.

Content filter errors. Provider blocked the request. Track to detect prompt issues.

Tool errors. Specific tools failing. Track per tool.

Quality signals. A judge flags a possible defect. Preserve the rubric and calibration; confirm consequential findings before treating a score as ground truth.

Hallucination signals. Detection of likely hallucinations (model claimed something not in the source). Hard to detect automatically but can be approximated.

Cost errors. Calls that cost much more than expected. Often indicate a bug.

Each gets its own dashboard. Each can trigger alerts.

Debugging tools

When something breaks, you need to find it and understand it. The debugging surface includes:

Trace search. Find an event by trace ID, approved pseudonymous subject reference, or timestamp without exposing broader tenant data.

Call inspector. Show metadata by default. Reveal only approved, minimized request/response fields under role checks, access logging, purpose limitation, and retention rules; many deployments should never retain the full payload.

Trace timeline. For complex flows, see the chain of calls visually.

Controlled replay capability. Can an approved, minimized input be re-run in an isolated environment with tools disabled or sandboxed? Never replay production calls into live side effects, and do not assume retained payloads are justified merely because replay is useful.

Diff view. Compare two calls (different versions of the same prompt, different models) side by side.

Search by approved content or derived fields. Search for defined patterns without turning the telemetry system into an unrestricted corpus of customer prompts. Authorization, minimization, indexing, retention, and access logging apply to search as well as storage.

Products differ materially in these capabilities. Verify them in a representative trial and include integration, storage, privacy review, migration, and operating work in the comparison; list price alone does not prove buying is cheaper.

Privacy and PII

LLM observability logs are sensitive. Inputs may contain personal data; outputs may quote it. Raw payload capture is sometimes proposed for debugging, but it is not automatically necessary or lawful; define the purpose and consider synthetic reproduction, derived fields, or short-lived controlled samples first.

Practices:

Pseudonymization/tokenization. Replace direct identifiers with purpose-specific tokens and protect any re-identification lookup separately. If re-identification remains possible, the records are still personal data rather than anonymous “non-PII.”

Redaction at log time. Detect and redact PII before it lands in observability storage. Specific patterns (emails, phone numbers, SSNs) replaced with placeholders.

Tenant isolation. Multi-tenant observability data isolated per tenant. One tenant’s data not visible to another.

Access controls. Who can view raw inputs/outputs? Logged.

Retention policies. Delete records at the approved retention limit, subject to applicable requirements. Moving them to cold storage is still retention, not deletion.

Deletion and restriction workflow. Make telemetry records discoverable by the identifiers approved for that purpose and implement the action counsel defines across primary storage, indexes, exports, and backups. Do not translate the colloquial “right to be forgotten” into a blanket promise; GDPR erasure has conditions and exceptions.

The system needs controls proportionate to the personal data and purpose. The exact combination varies, but tenant isolation, authorized access, minimization, retention, deletion/restriction handling, and auditability must be decided before payload telemetry is enabled. The European Commission summarizes the conditions and exceptions for erasure requests.

Alerting

Thresholds and signals that warrant alerts:

Cost.

  • Spend or forecast exceeds the feature’s measured budget envelope.
  • Per-request cost exceeds a workload-specific ceiling.
  • Rate of change exceeds the baseline band for the same traffic mix.

Latency.

  • Tail latency breaches the product’s measured service objective.
  • Time to first token or completion breaches the interaction-specific objective.
  • Tool timeouts or queue time depart from the baseline band.

Error rate.

  • Error rate breaches the workload’s error budget.
  • A specific error type departs from its baseline band, including validation errors and rate limits.

Quality.

  • A calibrated metric changes beyond its repeated-run variance or a safety slice records any disallowed outcome.
  • User feedback negative rate > baseline.
  • Regeneration rate > baseline.

Pattern.

  • Specific bad phrases appearing more often.
  • Sudden change in input distribution.

Each alert should identify the feature, time window, observed and expected value, affected slice, trace links, and runbook. Do not diagnose a runaway in alert text unless loop or retry telemetry supports that diagnosis.

A practical setup

Start with synthetic traces and test both the useful diagnostic path and the prohibited access path. An exporter outage should follow a defined bounded-buffer/drop policy, not fill disk indefinitely or block a safety-critical action without a decision. Record missing-telemetry counts so a blank dashboard is not mistaken for absence of failures.

This invented incident shows what the records need to explain:

ObservationTrace evidenceSupported conclusion
Customer answer fails source checkAnswer references source S1 but converts dispatch to arrivalThe claim is unsupported
Retrieval span contains S1 and S2S2 records approval pending; both versions were currentThese sources were available to generation
Model call is HTTP success, prompt v4Schema parser succeedsTransport and shape worked; meaning failed
Answer validator blocks deliveryFinal status held-for-reviewNo answer was sent on this path
Same tenant’s authorised reviewer opens traceAccess event recorded; other tenant receives denialThe expected access boundary is exercised

This evidence supports a generation/validation investigation; it does not prove that retrieval is always correct or that a prompt change caused the failure. Compare a controlled reproduction before making that attribution. Replays use synthetic or approved minimised inputs with external side effects disabled.

The observability acceptance record ties each signal to its purpose, retention and response. Test export and deletion, cross-tenant denial, trace propagation through a queue, missing telemetry, and a known unsupported-answer fixture before enabling a live diagnostic workflow.

Read next

Continue through the same learning path with the next practical articles.