Before you start
What you will be able to do
- Distinguish a model response, a tool attempt and an observed external outcome.
- Correlate a failed run without logging credentials or entire private documents.
- Turn an incident into an outcome-based regression test.
Concepts covered
What moves after a write times out?
- A timeout is not a negative receipt. The caller did not receive a response; the remote write may still exist. Do not let a model sentence settle that uncertainty.
- Stop blind retries. Retain the approved action and the original operation identifier. Investigation must not create a second effect.
- Ask the target through authorised read access. Look up the original operation using the target API's contract. A local checkpoint alone cannot tell you what committed remotely.
- Receipt-found path: inspect the evidence. The target returns an observable result for the operation. A missing or delayed lookup response follows a different path.
- Compare, rather than merely count, effects. Check the destination and content against the approved action. An object in the wrong queue is not a successful completion.
- Confirm the match or escalate the discrepancy. A matching receipt establishes the effect without another creation call. A mismatch needs investigation, not a success message.
- Alternative path: the lookup is inconclusive. Report the unresolved outcome instead of inventing success or absence. Retry only under a verified idempotency contract; otherwise escalate.
Start with the effect the user wanted, not the agent’s final sentence. If an assistant says it created a ticket, the first question is whether the intended ticket exists in the correct queue with the approved content. The conversation is evidence about what the agent reported; the target system is evidence about what happened.
This guide combines primary-source mechanisms with a recommended diagnostic design. The trace is fictional and deliberately redacted, not production data. Sources were checked on 14 September 2026; human editorial review is pending.
A trace is not a success score
The word trace has related but different uses. In agent evaluation, it often means the recorded sequence of messages, tool calls and results. In distributed tracing, an OpenTelemetry trace joins instrumented operations, called spans, across an application. A span records a unit of work, its timing and associated metadata. The two records can be correlated; neither automatically contains the other.
| Evidence | Question it answers | What it cannot establish alone |
|---|---|---|
| Model response | What answer or action did the model propose? | Whether an action ran or an answer is supported |
| Tool event | What operation was attempted and what response was observed? | Whether a timed-out write committed remotely |
| Span and correlated logs | Where did the request spend time or encounter an error? | Whether the task was useful or correct |
| State checkpoint | What application state was saved for recovery? | Whether a remote service deduplicates re-execution |
| Outcome check | Does the target state meet the task’s criteria? | Why the agent took a particular route |
| Evaluation result | Did a trial satisfy defined checks? | Whether the grader or test environment was correct |
Anthropic’s evaluation guide explicitly separates the trajectory from the outcome. A grade and the underlying record should be read together: a low grade can expose a bad agent decision, but it can also expose an ambiguous task, a faulty grader or a contaminated test environment.
Work one failure from beginning to end
Illustrative task: prepare a policy answer, obtain approval, and create a helpdesk ticket. The
approved action is action-leave-a; the target API accepts caller-provided request identifiers.
Names here describe an application design, not a vendor’s required event schema.
| Event order | Record | Interpretation |
|---|---|---|
| Request | Run run-leave-a has an authenticated principal and a specific destination queue |
The host knows the actor and allowed target |
| Retrieval | A policy revision and passage reference are retained | There is inspectable evidence behind the draft |
| Approval | Reviewer approval is bound to action-leave-a and the exact proposed content |
An approval for a different action is not reusable |
| Dispatch | create_ticket is sent with request identifier request-leave-a |
A write attempt exists; completion has not yet been established |
| Transport error | The client times out before receiving a response | The external effect is unknown |
| Model response | “The ticket has been created” | Unsupported completion claim; it must not become the run outcome |
| Reconciliation | An authorised lookup returns a ticket for request-leave-a with matching queue and content |
The effect is confirmed by the target, not by the earlier model sentence |
| Terminal record | The run records the verified ticket reference and corrects its user-visible report | No second creation request is needed |
If the lookup is unavailable or inconclusive, the run remains unresolved. Eventual consistency, meaning a read may lag behind a completed write, is one reason a missing lookup result may be insufficient. Do not invent a successful receipt, and do not let another agent “fix” the problem by creating the ticket again.
Record the evidence needed to distinguish causes
The following is a proposed minimum record for this design. Collect it at the application and tool boundaries, where observable requests and results exist. Do not require access to a model’s private reasoning.
| Record | Purpose | Handling constraint |
|---|---|---|
| Run ID, parent run ID, span correlation | Connect the main run, workers and tool attempts | Keep unbounded identifiers out of metric labels |
| Model identifier, prompt revision, tool-schema revision | Identify which deployed behaviour was exercised | Store configuration references, not API credentials |
| Authorised principal and scope reference | Establish whose permissions were applied | Restrict access; identifiers can still identify a person |
| Tool name, attempt ID and logical operation ID | Distinguish a retry from a new intended effect | Reuse the logical ID only for the same operation under the API contract |
| Attempt start/end time, observed status, error category | Measure the observed attempt duration and identify failed calls | Separate queue and execution timings require instrumentation at those boundaries; preserve timeout and unknown-outcome states |
| Approval reference and immutable action reference | Prove what was reviewed before dispatch | Compare the exact action again when execution resumes |
| Source revision, passage reference and bounded evidence | Diagnose missing or stale grounding | Do not copy an entire private corpus into telemetry |
| Terminal reason and outcome reference | Distinguish completed, denied, cancelled, exhausted and unresolved | Never infer success from the absence of an exception |
Per-run identifiers belong in traces and logs, not metric labels. Each distinct label combination identifies a separate time series, so a run ID label separates otherwise comparable operations into per-run series. This increases metric cardinality, the number of distinct combinations a metric tracks. See OpenTelemetry’s metric cardinality guidance.
Keep the detailed evidence in access-controlled records with an explicit retention policy. Redact before export. A transcript, tool argument or retrieved passage can contain confidential material even when the field is called “debug”. Hashing a guessable identifier is not the same as removing personal information.
Diagnosis before changing the prompt
| Symptom | Inspect first | Likely intervention if confirmed |
|---|---|---|
| No supporting passage | Query, authorised corpus and returned references | Repair retrieval or ingestion; do not ask the model to fill the gap |
| Wrong answer despite the right passage | Passage-to-claim support and draft transformation | Improve generation constraints and evidence grading |
| Valid tool syntax, denied operation | Host decision and target-service permissions | Correct an incorrectly scoped integration, or preserve an intentional denial |
| Repeated model calls without progress | Repeated state, query or action signatures | Add bounded termination and a user-visible blocker |
| Duplicate external object | Operation IDs, retry history and target deduplication | Fix action identity and the write boundary |
| Slow run with healthy tools | Span relationships, model-call count and approval waits | Separate review latency from inference and unnecessary sequential work |
| Worker succeeds, parent answer fails | Handoff data and final acceptance criteria | Fix the handoff or composition, not merely the worker prompt |
A denied operation can be a correctly functioning control. Do not remove authorisation checks or select a less restrictive provider simply to make the trace look successful.
Recovery is not replaying everything
Idempotency means repeating an operation under a defined contract does not introduce an additional intended effect. AWS describes caller-provided request identifiers as a way to express the same intent across retries. The service must implement the matching deduplication behaviour; putting an identifier into a local log does not provide it.
Preserve the original logical operation ID and compare the target and payload. Changed intent requires a new operation and, where applicable, a new approval. Also check how long the receiving service retains deduplication records before treating an old request as safe to repeat.
LangGraph’s persistence documentation describes re-execution after saved state. That is useful for recovery, but an external side effect may run again. A graph checkpoint and a tool service’s idempotency contract solve different problems. The local agent course deliberately limits its demonstrated deduplication to a local outbox; it does not claim exactly-once delivery to an external service.
Cost and limits of observability
Recording every prompt and tool response improves visibility but increases storage, access-control and privacy obligations. Recording only an aggregate duration is cheap but cannot explain why the wrong document was used. Choose bounded evidence references plus redacted failure records before deciding to retain all content.
Sampling reduces telemetry volume but can remove the very failed run needed for diagnosis. A sampling policy should account for errors and consequential actions; its coverage should be known to the investigator. Missing instrumentation is missing evidence, not proof that a step did not run.
Tracing is not a replacement for evaluation, security review or permission enforcement. Use outcome-based tests to compare changes, traces to investigate individual runs, and ordinary service metrics to watch operational health.
Turn the failure into a regression case
Create an isolated fixture in which a write commits but its response is lost. Require the agent system to preserve the operation identifier, avoid an unsupported success claim, reconcile using the target’s contract and avoid a duplicate effect. Add a second fixture where reconciliation cannot establish the outcome; it should finish as unresolved rather than claim success.
Grade the final target state and the accuracy of the user-facing report, not its exact wording. A different investigation route may be acceptable; a second ticket is not. Keep the environment isolated so leftover objects from an earlier trial cannot make a broken run pass.