An agent can produce a correct-looking answer while completing the task incorrectly. Testing AI agents therefore requires more than comparing the final text with a reference answer. You need to inspect the trajectory, tool calls, and final state of the environment.
That is the main difference between evaluating a chatbot and evaluating an agent.
What is an AI agent eval?
An eval is a repeatable test for an AI system. You provide a task, run the agent in controlled conditions, and grade the result against criteria written before the run.
Anthropic uses several useful terms:
- a task is one test case with inputs and success criteria,
- a trial is one attempt at that task,
- a grader scores one aspect of performance,
- a trace or trajectory records messages, tool calls, and intermediate results,
- the outcome is the actual environment state after the trial.
The outcome matters more than the model's claim. A booking agent can say that a flight was booked. The eval should check whether the reservation exists in the database with the requested dates and passenger details.
Why final-answer tests miss agent failures
An agent acts across multiple steps. At any point it can:
- choose the wrong tool,
- call the right tool with incorrect parameters,
- ignore the returned result,
- repeat an operation with a side effect,
- stop too early,
- reach a good result through an unsafe path.
A grader that sees only the final message will miss many of these failures.
The preprint “When Agents Implement Systems” examines agent-built systems and defects that existing tests failed to detect. It is a case study, not a universal benchmark, but its limitation is familiar: a green test suite proves only the scenarios represented by the suite.
Build a task set from real work
The first useful eval does not need hundreds of cases. Twenty realistic examples are usually more informative than a thousand synthetic prompts with no relationship to production.
Include:
- common tasks,
- ambiguous requests,
- missing information,
- tool failures,
- attempts to perform forbidden actions,
- cases where the correct behavior is to ask a human.
Each task needs a concrete success condition. “The answer is good” is not one. “The agent created exactly one issue in the correct project and changed no other record” can be tested.
Use more than one grader
A single aggregate score can hide the failure you care about.
Deterministic graders
Use code for facts that code can verify: process exit status, a database row, output shape, number of changed files, or the absence of a forbidden call.
Model graders
Model-as-judge grading is useful for writing quality, reasoning relevance, or alignment with a flexible brief. Give the grader a specific rubric and examples. Do not use it in place of an assertion when the result is mechanically verifiable.
Trajectory graders
Grade how the agent reached the result. Did it read the source before editing? Did it wait for approval before using a write tool? After a failure, did it change strategy or repeat the same call?
Cost and latency graders
Two versions may have the same success rate while one consumes five times more tokens and performs a dozen unnecessary actions. That difference will appear on the bill and in the user experience.
Run multiple trials
Model behavior varies. One success or failure says little about reliability.
For important tasks, run several trials and record:
- success rate,
- step count,
- cost,
- latency,
- human-escalation rate,
- failure category.
If results swing sharply between runs, the variance may matter more than the average.
Test side effects
The most damaging failures may not appear in the answer. An agent created the requested document and overwrote an existing one. It sent the right email twice.
Your test environment should support a state diff before and after the trial. For write tools, useful controls include:
- idempotency keys,
- dry-run mode,
- an isolated database or sandbox,
- an allowlist of operations,
- a complete tool-call log.
The guide to the AI agent environment covers this layer in more detail.
Use evals before and after launch
Run the suite when you change the model, prompt, tools, or memory. The goal is not to pass a benchmark once. It is to catch regressions.
Production will supply cases your team did not imagine. Every meaningful failure should become a new task in the suite. Over time, the eval grows from real system behavior rather than a hypothetical list of edge cases.
A minimum eval loop
You can start with:
- 20 realistic tasks,
- an isolated environment,
- deterministic checks of the outcome,
- a saved trace for every trial,
- three trials for critical cases,
- a comparison with the previous version.
If you cannot define success for a task, it may not be ready for autonomous execution. An agent that proposes a result for review is often the better product until the team can measure the outcome.
Sources
Want to build an AI agent, an MVP, or an app?
I design and build AI products from the first version to production: MVPs, React Native mobile apps, web apps, and AI agents. Tell me what you want to build and I will tell you where to start.