The demo passed. The product did not
An agent receives a customer request, checks the account, applies a policy, and sends a helpful reply.
The answer is accurate. The tone is good. The task looks complete.
Then someone checks the account and discovers that the promised refund was never created.
On another run, the refund is created twice. On a third, the agent spends six minutes searching irrelevant documents before asking a person to do the work manually.
If we evaluated only the final message, all three runs might look acceptable.
This is the trap in evaluating agentic products. A model response is one part of the experience. The product also includes instructions, tools, permissions, orchestration, memory, connected systems, human checkpoints, recovery controls, and the environment in which the work happens.
The agent can produce a fluent answer while the product fails.
Traditional usability research still matters. So do model benchmarks, software tests, security reviews, and operational monitoring. But none of them alone tells us whether an agent can complete the right work, within the right boundary, often enough to deserve its role.
Anthropic’s evaluation guidance defines an agent trial through its task, transcript or trajectory, graders, and final environment state. It makes a useful distinction between what an agent says happened and what actually exists in the environment after the run. Anthropic, Demystifying evals for AI agents
For product teams, that distinction is gold.
The strongest evaluation programme does not ask one broad question such as “Is the agent accurate?” It asks a set of narrower questions:
Did the user get the intended outcome?
Were important constraints respected?
Was the route through the work sensible?
Did the agent know when to ask for help?
Could a person understand and change the outcome?
Did failure remain contained and recoverable?
Does the product behave consistently across repeated attempts?
Are live signals improving the tests we run before release?
This guide turns those questions into 18 practical evaluation patterns.
What are you actually evaluating?
When a team says “the agent improved”, several different things may have changed.
The model
A new model may reason better, follow instructions more reliably, or use tools differently.
The agent harness
Prompts, orchestration, tool definitions, memory, retries, guardrails, and context selection shape behaviour around the model.
The product experience
Task framing, plan visibility, approvals, recovery, and feedback change what people can understand and control.
The environment
APIs, data quality, permissions, latency, and external-system behaviour affect whether the same agent succeeds.
The operating model
Human reviewers, escalation owners, policies, and support processes determine what happens when the product reaches its limit.
An evaluation should identify which layer it is testing and which version of that layer produced the result. Otherwise, a model change may get credit for a better tool, or the agent may get blamed for a broken test environment.
An agent score is evidence about a whole configured system, not a model in isolation.
Start with the Outcome
Before choosing a metric, define what good work means.
That definition is a product decision. It should come from user goals, domain requirements, operational constraints, and the acceptable cost of failure.
Pattern 01 · Evaluatable task contract
Write each evaluation task with a clear goal, starting state, available tools, relevant constraints, and success criteria.
“Handle this support case” leaves too much open. “Resolve the duplicate charge, create no more than one refund, record the policy used, and escalate if account identity cannot be verified” can be tested.
If a reasonable evaluator cannot agree on what success means, the product requirement is not ready.
Pattern 02 · End-state verification
Check the state of the world after the run.
For a booking agent, verify the reservation. For a coding agent, run the tests and inspect the changed files. For a research agent, check whether claims are supported by appropriate sources. For a workflow agent, inspect the records, messages, approvals, or artefacts it was meant to create.
The final answer is evidence. The final state is the outcome.
Pattern 03 · Constraint grader
Grade the boundaries separately from task success.
An agent can achieve the outcome while using prohibited data, exceeding a budget, contacting the wrong person, skipping an approval, or making an irreversible change too early.
Treat critical constraints as gates, not small deductions hidden inside an average score.
A useful eval describes the job and the boundary with enough precision to judge both.
Build Scenarios from the Real Job
A tidy set of average cases creates tidy results.
Real workflows contain ambiguity, missing access, conflicting evidence, stale data, unusual users, unavailable tools, and cases where the correct action is to do less.
Your scenario library should reflect that distribution without simply copying production logs and their existing biases.
Pattern 04 · Representative task set
Start with the jobs users actually bring to the product.
Segment them by intent, complexity, risk, user type, data condition, and tool path. Weight the release view to match expected use, but keep per-segment results visible so common easy tasks do not hide weak performance on important cases.
Update the set as real usage changes.
Pattern 05 · Boundary-case pair
Test when a behaviour should happen and when it should not.
If you test whether the agent escalates risky cases, also test similar cases that should remain autonomous. If you test whether it searches, test when searching is unnecessary. If you test policy refusal, test a nearby legitimate request.
Paired cases reveal over-triggering as well as under-triggering.
Pattern 06 · Exception scenario
Include missing information, conflicting policies, unavailable tools, permission failures, and uncertain external state.
Grade whether the agent classifies the exception, preserves valid work, routes it to the right owner, and asks a useful question.
An exception handled well is a successful product behaviour, even when the original task cannot be completed.
Pattern 07 · Adversarial journey
Test misuse, manipulation, and attempts to cross the agreed boundary.
The adversarial journey should include the full product, not merely a hostile prompt. A user may place misleading instructions in a document, exploit a tool response, manipulate memory, or pressure a human reviewer through the agent’s output.
Pair specialist security testing with product scenarios that show how the interface contains and communicates the risk.
A balanced scenario suite tests capability, restraint, and recovery.
Trajectory and Human Collaboration
Evaluate the path, not only the answer
Two agents may reach the same result through very different routes.
One uses the correct sources, makes three necessary tool calls, and stops. Another searches broadly, exposes sensitive data to an unnecessary tool, retries an irreversible action, and happens to land on the right state.
Both passed the outcome check. Only one behaved like a dependable product.
OpenAI’s agent evaluation tooling includes trace grading for end-to-end workflows, reflecting the need to inspect what happened across the run rather than scoring a single response. OpenAI, Introducing AgentKit
Pattern 08 · Tool-choice check
Evaluate whether the agent chose an appropriate tool for the situation.
Check required tools, prohibited tools, unnecessary calls, and fallbacks when the preferred tool is unavailable. A correct answer based on an unreliable source should not receive the same confidence as one grounded in the system of record.
Pattern 09 · Evidence-path review
Inspect what information supported important decisions.
Measure source quality, relevance, coverage, freshness, and whether uncertainty survives synthesis. For high-impact recommendations, sample traces and compare the presented evidence with what the agent actually used.
The goal is not to reveal private chain-of-thought. It is to verify operational evidence and decision inputs.
Pattern 10 · Plan-adaptation check
Test whether the agent changes approach for a good reason.
Useful adaptation responds to new evidence, a failed tool, or a changed constraint. Poor adaptation drifts from the goal, silently expands scope, or repeats the same failed action.
Grade the trigger, the new plan, and whether the product made a meaningful change visible to the user.
Pattern 11 · Efficiency budget
Set budgets for time, cost, tool calls, retries, and human attention.
Efficiency is not a contest to produce the fewest steps. The budget should protect the experience from waste while leaving room for difficult work. Track useful work separately from loops, redundant searches, repeated questions, and avoidable review.
A slower route may be better when it produces stronger evidence or safer recovery.
Measure the human-agent relationship
Agent quality includes how well the product collaborates with people.
An agent that completes many tasks but creates confusing approvals, poor handoffs, and unpredictable takeover may shift effort rather than remove it.
Pattern 12 · Uncertainty calibration
Compare expressed confidence and escalation behaviour with actual correctness.
The agent should not sound certain when evidence is weak. It should also avoid asking for approval whenever a case is merely unfamiliar.
Test whether confidence labels, caveats, and escalation triggers help users make better decisions.
Pattern 13 · Intervention value
Measure what happens after a person edits, rejects, pauses, or takes over.
Did the intervention prevent an error, improve the outcome, or merely add delay? Did the product present enough context for the person to act? Did the agent use the correction appropriately when work resumed?
High intervention volume can indicate prudent control or weak automation. Value separates the two.
Pattern 14 · Handoff quality
Evaluate whether the receiving person or agent can continue without reconstructing the case.
Grade the transferred goal, state, evidence, prior decisions, open questions, and requested action. Measure how often the receiver reopens sources or asks for information already available.
The quality of the packet matters more than the length of the transcript.
Human effort should change the outcome, not compensate for missing product context.
Reliability and Production Evidence
Test reliability, recovery, and change
A single successful run demonstrates possibility. A product needs repeatability.
Agent outputs vary, environments change, and small updates can improve one behaviour while damaging another. Evaluation needs to capture that variability instead of smoothing it into one impressive example.
Pattern 15 · Repeated-trial reliability
Run important tasks more than once.
Track first-attempt success and consistent success across repeated trials. The metric should match the product promise. A brainstorming tool may benefit from several attempts where one good result is enough. A customer-facing action agent needs to behave correctly again and again.
Anthropic distinguishes pass@k, the chance of at least one success across attempts, from pass^k, the chance that every attempt succeeds. The second view is especially useful when consistency matters. Anthropic, Demystifying evals for AI agents
Pattern 16 · Safe-failure test
Deliberately create tool failure, timeout, malformed data, missing access, and uncertain commitment state.
Grade whether the agent stops safely, avoids duplicate effects, communicates the situation accurately, preserves useful progress, and offers an appropriate recovery route.
“Did not complete” can be a pass when unsafe continuation would be worse.
Pattern 17 · Regression gate
Protect behaviours the product already performs reliably.
Run a regression suite for prompt changes, tool changes, model updates, policy changes, and workflow edits. Keep critical constraints and known severe failures as hard release gates.
Maintain a separate capability suite for harder tasks where the team is still improving. Mixing the two can hide a serious regression inside a higher overall score.
A release decision should show capability, consistency, constraints, and recovery together.
Connect offline evals to production
Offline evaluations are controlled, repeatable, and fast. Production reveals what people actually ask, which systems fail, and which consequences matter.
Neither should replace the other.
Pattern 18 · Production learning loop
Turn real failures and interventions into reviewed evaluation cases.
Sample user-reported problems, repeated overrides, failed commitments, unusual tool paths, long-running tasks, and high-impact near misses. Remove or protect sensitive data, then decide whether the case belongs in regression, capability, safety, or usability testing.
Connect each signal to the model, harness, workflow, tool, policy, and interface version that produced it.
NIST’s AI Risk Management Framework calls for deployed systems to be monitored, measurement approaches to remain connected to deployment context, and metrics to be updated as risks and impacts evolve. NIST, AI RMF Core
Production evidence should improve the evaluation suite, and the evaluation suite should improve production.
Common evaluation failure modes
Grading the response only
The final text looks good, while the environment state is wrong or a harmful action already happened.
Verify outcomes and consequential actions directly.
One average score
Easy common tasks hide weak performance on rare, high-impact cases.
Keep segment, risk, and critical-constraint results visible.
Happy-path dataset
Every tool works, every input is complete, and every policy is clear.
Add exception, boundary, recovery, and adversarial scenarios.
Vague human grading
Reviewers are asked whether a result is “good” without a shared rubric.
Define observable criteria, examples, and disagreement handling. Calibrate model-based graders against domain experts.
Broken eval, blamed agent
The task is ambiguous, the environment leaks state, or the grader rejects a valid solution.
Create a reference solution, inspect failed traces, and test the grader itself.
Optimising the benchmark
The team improves the score without improving user outcomes.
Refresh tasks, use hidden sets where appropriate, and compare offline gains with production evidence.
Ignoring human cost
The agent completes more tasks by generating more approvals, escalations, or correction work.
Measure intervention value, queue health, reconstruction effort, and total time across people and agent.
No eval owner
The suite becomes stale, noisy, and distrusted.
Give evaluation infrastructure a clear owner while enabling domain experts and product teams to contribute cases.
A practical evaluation stack
No single method catches every problem. Use several layers with different strengths.
Deterministic checks
Use database state, API state, tests, schema validation, permission checks, tool logs, policy assertions, and budget limits where correctness can be verified directly.
Model-based graders
Use explicit rubrics for qualities such as groundedness, completeness, tone, or handoff usefulness. Calibrate them regularly against human judgment and inspect disagreements.
Expert review
Use domain experts for high-impact decisions, ambiguous outputs, grader calibration, and cases where several answers may be valid.
Usability research
Observe whether people understand the plan, evidence, state, approval, interruption, and recovery controls. Test behaviour, not only stated trust.
Production monitoring
Track real outcomes, tool and system failures, latency, cost, stop behaviour, escalations, overrides, complaints, and affected users.
Experimentation
Use controlled comparisons for changes that can safely reach real traffic. Measure downstream outcomes as well as immediate engagement.
Anthropic describes these layers as complementary: automated evals for fast iteration, production monitoring for real-world evidence, and human review for calibration and subjective quality. Anthropic, Demystifying evals for AI agents
A Product Evaluation Review
Use these questions when defining an agent, preparing a release, or reviewing a model and workflow change.
Product promise
What job is the agent responsible for?
What does successful completion mean to the user?
Which failures would break the product promise?
Evaluation unit
Are we testing the model, harness, interface, environment, operating model, or configured system?
Are every relevant version and dependency recorded?
Does the test environment behave like production where it matters?
Outcomes and constraints
Can the end state be verified independently of the agent’s message?
Which constraints are hard gates?
Is safe refusal or escalation treated as a valid outcome?
Scenarios
Does the suite cover real task distribution and high-impact tails?
Are positive and negative boundary cases paired?
Are exceptions, unavailable tools, and adversarial journeys included?
Trajectory
Are tool choice, evidence, adaptation, and efficiency evaluated?
Can we inspect why a run passed or failed without exposing private reasoning?
Are retries and irreversible actions visible?
Human experience
Does uncertainty help the user calibrate their decision?
Do interventions improve outcomes?
Can a new owner continue from the handoff packet?
Reliability and operation
Are critical tasks run across repeated trials?
Does the release gate protect known good behaviour?
Do production failures become reviewed eval cases?
Evaluation Is Product Definition
Evaluation can look like a technical activity that begins after design.
For agentic products, it belongs much earlier.
The moment a team writes a useful task, it has to decide what the agent owns. The moment it defines an end-state check, it has to decide what completion means. The moment it adds a constraint grader, it has to decide which boundaries are non-negotiable. The moment it measures intervention, it has to decide what role the human should play.
Those are product decisions.
A good evaluation programme makes them concrete. It gives design, product, engineering, research, operations, and domain experts a shared language for quality.
It also keeps confidence honest.
One fluent run can show that an idea is possible. A dependable product requires broader evidence: the outcome is real, the path is appropriate, important constraints hold, people can intervene effectively, failures remain recoverable, and improvement in the lab survives contact with the world.
That is a much higher bar than model accuracy.
It is also the bar users experience.
