Knowledge guide
AI Agent Evaluation Checklist Before Production
Use this AI agent evaluation checklist to test outcomes, tool calls, permissions, failures, graders, and evidence before production.
Data accurate as of September 2026 based on market research
Introduction
An AI agent is ready for production only when evidence shows that it completes the intended task, uses permitted tools, fails safely, and leaves a reviewable trace. Check the final state, tool path, policy decisions, and external changes. Make the release decision from recorded evidence, not from a successful demo.
For implementation help, see the self-hosted AI systems service. Pair this guide with the AI agent observability checklist and AI agent security checklist.
Contents
Truth Box
What an AI agent evaluation must prove
Build a representative evaluation set
AI agent production release gate
Choose the right grader
Test security, failure, and recovery
Industry and platform context
Common misconceptions
FAQ
Conclusion
Truth Box
| Key Point | Insight |
|---|---|
| Grade the real outcome | Verify the database, file, ticket, message, or external record instead of trusting the agent's summary |
| Inspect the trajectory | Tool choice, arguments, order, retries, approvals, and stop conditions can fail even when the answer looks right |
| Use mixed graders | Code checks suit exact conditions, model graders suit nuanced rubrics, and people calibrate judgment |
| Test unsafe paths | Denied requests, poisoned content, missing data, timeouts, and duplicate events belong in the suite |
| Keep release evidence | Save the task set, versions, traces, scores, failed cases, reviewer decision, and rollback trigger |
What an AI agent evaluation must prove
Anthropic distinguishes a transcript, also called a trace or trajectory, from its outcome. The transcript is the complete trial record, including outputs, tool calls, reasoning, intermediate results, and other interactions. The outcome is the final state in the environment.[1] Google Cloud and Microsoft Foundry make a similar distinction between final-response or system evaluation and trajectory or process evaluation.[3][4]
Use five evaluation layers:
| Layer | Question | Evidence |
|---|---|---|
| Outcome | Did the requested result exist in the real system? | State query, file hash, record ID, or verified artifact |
| Trajectory | Did the agent use suitable tools and arguments without wasteful or forbidden steps? | Trace with tool calls, inputs, outputs, retries, and stop reason |
| Control | Did permissions, approvals, and policy checks hold? | Denial events, approval record, unchanged restricted state |
| Resilience | Did the agent fail safely when tools or data were wrong? | Injected-failure run, recovery result, and idempotency evidence |
| Operations | Can the team detect, stop, investigate, and restore the workflow? | Alert, runbook, owner, disable path, rollback test, and retained trace |
NIST's voluntary AI RMF recommends testing before deployment and during operation, with documented test sets, metrics, deployment-like conditions, safety, security, and repeatable evaluation processes.[5][8]
Build a representative evaluation set
Start with manual checks, recent defects, and costly edge cases. Anthropic gives 20 to 50 simple tasks as a useful early range, not a universal minimum.[1] OpenAI recommends mixing production, historical, expert, and synthetic data according to the objective.[2]
Each case needs a clear contract:
| Field | What to record |
|---|---|
| Input | User request, starting state, available data, and relevant permissions |
| Expected outcome | The state or artifact that should exist after the run |
| Allowed actions | Tools, data scopes, destinations, and side effects the case permits |
| Forbidden actions | Changes that must never occur, even if the final answer is useful |
| Failure setup | Missing field, unavailable tool, stale record, hostile document, or duplicate event |
| Pass evidence | Deterministic assertion, rubric, trace fields, and reviewer role |
Include normal tasks, ambiguity, unsupported requests, denied permissions, malformed tool output, partial writes, cancellation, and duplicate delivery. Repeat important cases and isolate each trial so shared state does not distort results.[1]
Do not apply one pass rate to every suite. Anthropic recommends low initial pass rates for capability evals and nearly 100 percent for regression evals.[1] Set release thresholds by the suite's purpose and the impact of failure.
AI agent production release gate
Use this table for a go or stop decision.
| Gate | Pass condition |
|---|---|
| Purpose | The agent has one defined job, owner, users, and measurable completion state |
| Test set | Cases cover ordinary work, boundaries, failures, hostile input, and recovery |
| Outcome checks | Critical results are verified in the external system, not from the final message |
| Tool checks | Required tools and arguments are correct; irrelevant or forbidden calls are absent |
| Permission checks | Read, write, execute, and network access match the narrow task |
| Approval checks | High-impact actions pause for a named reviewer and cannot bypass the gate |
| Reliability checks | Repeated trials expose variance, loops, duplicate writes, and retry behavior |
| Trace checks | A reviewer can reconstruct the request, tools, policy decisions, side effects, and stop reason |
| Recovery checks | Operators can disable the agent, revoke access, restore state, and identify affected records |
| Release record | Versions, results, failures, residual risk, approver, and rollback trigger are saved |
A forbidden side effect, missing authorization, sensitive-data exposure, unrecoverable partial write, or missing stop control should block release.
Choose the right grader
| Method | Best use | Main caution |
|---|---|---|
| Deterministic code check | Schemas, calculations, permissions, exact state, file paths, and tool arguments | Can reject valid variation if the assertion is too rigid |
| LLM-as-a-judge | Tone, completeness, groundedness, and other rubric-based qualities | Needs clear criteria and calibration against human review |
| Human review | High-impact judgment, ambiguous cases, and grader calibration | Slower and harder to apply to every run |
Use code for verifiable facts and model graders for rubric-based judgment. Calibrate model graders against expert review.[1] AWS, MLflow, and LangSmith support trace-based or mixed evaluation methods.[7][10][11]
Do not force one exact trajectory when several safe paths are valid. Check mandatory steps, forbidden steps, outcome state, and efficiency separately. Read failed traces to distinguish an agent failure from a broken fixture or unfair grader.
Test security, failure, and recovery
OWASP's Agentic Top 10 includes goal hijacking, tool misuse, identity and privilege abuse, memory poisoning, cascading failures, and human-agent trust problems.[6] Turn those categories into harmless sandbox tests.
Place hostile instructions inside a test document or tool response. Try an out-of-scope account, forbidden destination, expired approval, malformed output, interrupted write, duplicate event, and missing field. Pass only when policy is enforced outside the model, restricted state stays unchanged, and the trace explains the stop.
Industry and platform context
Current platform guidance follows a common pattern. Google documents final-response and trajectory evaluation, but the cited feature is Preview and Pre-GA.[3] Microsoft documents system, process, and tool-call evaluators; Task Adherence and several system evaluators are in public preview and are not recommended by Microsoft for production workloads.[4] AWS, MLflow, and Arize document trace-based evaluation.[7][10][12]
These products can run scorers, but they do not define your business contract. Your team still decides what success, unacceptable behavior, residual risk, and approval mean.
Common misconceptions
| Myth | Correction |
|---|---|
| A good final response proves the agent worked | Verify the real state and inspect the tool path |
| An LLM judge can grade everything | Use deterministic assertions for exact facts and calibrate subjective graders with people |
| Passing pre-production tests finishes the job | Monitor production, convert failures into regression cases, and re-run the gate after meaningful changes |
FAQ
What is the first step in an AI agent evaluation checklist?
Write the agent's job, expected final state, permitted actions, forbidden actions, and release blockers in plain language. Then build cases that test each condition.
Should I evaluate the final answer or the tool trajectory?
Evaluate both. The answer can look correct while the agent used stale data, the wrong tool, excessive permissions, or an unsafe side effect.
How many AI agent evaluation cases are enough?
No universal number applies. Start with real tasks and failures that cover the highest-impact behavior. Anthropic suggests 20 to 50 simple tasks can be useful early, then expand the suite as failures and use cases appear.[1]
Can LLM-as-a-judge replace human review?
No. It can scale rubric-based scoring, but people should define the criteria, calibrate the judge, review disagreements, and own high-impact decisions.
When should the evaluation suite run again?
Run it after changes to the model, prompt, tool schema, permissions, data source, routing, memory, or approval policy. Keep monitoring after release and add real failures as regression cases.
Conclusion
Release only when the evidence packet proves the intended outcome, permitted trajectory, enforced controls, safe failure, and tested recovery. If a critical result exists only in the agent's own final message, the evaluation is incomplete.
For help scoping and testing a private agent workflow, contact [email protected] or review the self-hosted AI systems service. More implementation guides are available in the AI article library.
Sources and Further Reading
[1] Anthropic, Demystifying evals for AI agents
[2] OpenAI, Evaluation best practices
[3] Google Cloud, Evaluate Gen AI agents
[4] Microsoft Foundry, Agent Evaluators for Generative AI
[5] NIST, AI RMF Core
[6] OWASP, Top 10 for Agentic AI Applications
[7] AWS, Amazon Bedrock AgentCore Evaluations
[8] NIST, Generative AI Profile
[9] Google Cloud, A methodical approach to agent evaluation
[10] MLflow, Evaluating production traces
[11] LangChain AgentEvals with LangSmith, Trajectory evaluations
[12] Arize AX, Agent trajectory evaluations
[13] OpenAI, How evals drive the next chapter in AI for businesses
[14] LangChain, Agent Evaluation Readiness Checklist
Need help applying this?
See the related service page: Self-hosted AI system deployment or email [email protected].