Knowledge guide
AI Agent Observability Checklist: Reconstruct One Failed Run
Use this AI agent observability checklist to reconstruct one failed run across models, tools, approvals, side effects, usage, and errors.
Data accurate as of August 2026 based on market research
Introduction
A self-hosted agent passes this checklist when an operator can reconstruct one failed run from its trigger to its final outcome through a shared run or trace ID. The evidence should show model calls, retrieval, tools, approvals, external side effects, timings, reported usage, and errors without capturing sensitive content indiscriminately.
Introduce one safe failure, then follow it from alert to trace and structured events without correlating approximate timestamps. Use this as an acceptance test for a self-hosted AI system. Observability explains recorded behavior. It does not prove that an answer was correct, safe, or high quality.
Contents
Truth Box
The one-run reconstruction test
AI agent observability checklist
Implementation paths for a small team
Current standards context
Misconceptions about AI agent observability
FAQ
Conclusion
Truth Box
| Key Point | Insight |
|---|---|
| Pass test | One failed run can be reconstructed without joining events by approximate timestamps |
| Minimum evidence | Correlated trace, structured model and tool events, outcome status, side-effect record, and an actionable alert |
| Privacy default | Record approved metadata and redacted references; retain raw content only under an explicit data policy |
| Standards status | OpenTelemetry GenAI agent spans and metrics are Development conventions, not a frozen universal schema |
| Boundary | Observability supports investigation; it does not prove correctness, safety, or answer quality |
The one-run reconstruction test
Choose a sanitized incident or create a safe synthetic failure, such as a test tool rejecting a harmless request. Keep publishing, payments, account changes, and customer messages out of scope.
Preserve the run identifier. Record the trigger, environment, profile, session, agent, provider, model, and configuration version. Follow retrieval and tools in order, then find the approval decision, side effect, outcome, latency, retries, token-count provenance, and errors.
A distributed trace should preserve parent and child spans. W3C Trace Context defines traceparent and tracestate headers for correlation across HTTP service boundaries. Pass when a responder can move from the failure signal to the failed operation and verify what happened outside the agent process.
AI agent observability checklist
| Check | Evidence to record | Pass condition |
|---|---|---|
| Run identity | Trace ID, task or session ID, trigger, environment, profile, and timestamps | One identifier follows the agent, model, retrieval, MCP, and tool work |
| Span hierarchy | Parent and child spans for routing, models, retrieval, tools, approvals, subagents, and external actions | Order, nesting, failure location, and terminal status are visible |
| Version context | Agent, configuration or prompt, provider, model, tool schema, and knowledge versions | Changed behavior can be tied to the versions that produced it |
| Model calls | Provider, model, duration, finish status, retries, error type, and token-count type | Slow, failed, retried, and incomplete calls can be isolated |
| Retrieval | Query reference, document versions, result count, and source references | The influencing knowledge is identifiable without copying every private excerpt |
| Tools, MCP, and approvals | Tool or server, redacted argument summary, duration, status, retry, policy result, and approval | Failed, denied, skipped, timed-out, and completed actions are distinguishable |
| Side effects and outcome | External action ID, target, safe change reference, rollback or idempotency reference, and final outcome | The team can verify what changed and how the task ended |
| Governance and alerting | Allowlist, redaction, access, retention, deletion, backup, destination, alert owner, and runbook | Sensitive fields are controlled, and sustained user impact reaches an owner |
For retrieval-heavy systems, connect traces to the document versions used by your private knowledge system with Hermes Agent and MCP. Use the AI agent security checklist for access and approval controls, and the backup and recovery checklist for restoring trace and audit data.
Do not make private reasoning or hidden chain-of-thought a required telemetry field. Record exposed inputs, outputs, tool choices, approvals, policy events, errors, and external changes. Prompt and tool payload capture needs a deliberate policy because these fields can contain personal data or secrets.
Implementation paths for a small team
| Path | Best fit | Important caveat |
|---|---|---|
| Hermes Langfuse plugin | The Hermes plugin documentation describes opt-in turn, API-call, tool, usage, and cost observations for cloud or self-hosted Langfuse | It can export messages, assistant output, tool calls, and sanitized or truncated tool data. Treat the destination as sensitive. Export is fail-open, so monitor telemetry silence |
| Hermes hooks or outbound webhooks | Teams sending selected events to a private backend | Use observer-only events. Some Hermes hooks can block or transform execution, while outbound webhooks are notify-only. Payloads can contain prompts, arguments, results, paths, or secrets |
| OpenTelemetry Collector and an existing backend | Multi-service stacks needing common propagation and storage control | The team still maintains the backend, retention, dashboards, and a versioned contract for agent fields |
| Framework-native or managed tracing | One framework where setup speed matters more than backend control | Coverage, defaults, exporter destinations, data location, and maintenance duties vary |
Keep a documented local evidence contract across products. Use stable operation names, separate a single run from its longer session, and retain enough version context to compare changed behavior.
Current standards context
OpenTelemetry's GenAI agent-span and metric documents are marked Development. The former website page says the conventions moved to a separate repository and is no longer maintained there. Pin the checked commit or release in implementation notes, and review it before changing production field names.
Tracing defaults also differ by framework. The OpenAI Agents SDK documents tracing as enabled by default and says generation and function spans can store sensitive inputs and outputs. Microsoft Foundry tracing is off by default. Check the actual framework, exporter, and deployment configuration instead of assuming one default.
Treat token telemetry as operational evidence. Record whether counts came from the provider, instrumentation, or an enabled offline counter, and whether they represent used or billable tokens. Do not assume they reproduce the final invoice. Link outcome labels, human feedback, or chosen evaluators to the trace so a reviewer can assess the result as well as the execution path.
Misconceptions about AI agent observability
| Misconception | Correction |
|---|---|
| More logs mean better observability | Uncorrelated logs create search work. A smaller set of structured events tied to a run or trace ID is more useful |
| Every prompt, response, and reasoning step should be stored | Raw content can expose customer data, credentials, and private files. Capture approved metadata and redacted references by default, and do not claim access to hidden chain-of-thought |
| Self-hosting keeps telemetry private automatically | Privacy still depends on exporter destinations, network egress, dashboard accounts, retention, backups, and who can query the data |
FAQ
What is the minimum useful AI agent observability setup?
Use one correlated run or trace ID, structured model and tool events, an outcome status, a side-effect audit record, and one actionable alert that links to investigation steps.
Can Hermes Agent send traces to a self-hosted dashboard?
Yes. The opt-in plugin accepts a self-hosted Langfuse base URL. It can export message and tool content as well as usage data, so review capture, redaction, access, and retention settings. Because export is fail-open, verify trace arrival and monitor telemetry silence.
Should prompts and responses be logged?
Only when the data policy permits it. Prefer approved metadata, redaction, document IDs, pseudonymous references, or short controlled samples over unrestricted raw capture.
Does OpenTelemetry solve the whole observability problem?
No. W3C Trace Context standardizes cross-service HTTP propagation. OpenTelemetry provides instrumentation, telemetry conventions, and Collector-based processing and export, while its GenAI conventions remain Development. Teams still need outcome labels, privacy rules, retention, dashboards, alerts, runbooks, and quality evaluation.
How long should AI agent telemetry be retained?
Use the shortest period that supports incident review. Base it on data sensitivity, storage volume, access controls, deletion capability, and tested recovery needs rather than a universal interval.
Conclusion
Run one controlled failure and require a path from alert to trace, structured evidence, side-effect record, and final outcome. If any step depends on timestamp guessing, fix correlation before adding dashboards.
For help planning and testing a private observability path, see Nguyen LNP's self-hosted AI systems service. You can also browse more AI implementation guides.
Sources and Further Reading
Hermes Agent, built-in plugins
OpenTelemetry GenAI agent spans, commit pinned
OpenTelemetry GenAI metrics, commit pinned
OpenTelemetry, moved GenAI conventions notice
OpenTelemetry Collector
W3C Trace Context Recommendation
OpenAI Agents SDK, tracing
LangSmith, observability concepts
Langfuse, observability best practices
Arize Phoenix, tracing overview
Google Cloud, agent observability
AWS, Amazon Bedrock AgentCore observability
Microsoft Foundry, observability
Microsoft Foundry, trace data
Prometheus, alerting practices
Grafana, alerting best practices
Need help applying this?
See the related service page: Self-hosted AI system deployment or email [email protected].