AI Agent Evals: Build a Test Set That Finds Real Failures

By Numan ·

Create an evaluation set for agent accuracy, tool use, refusals and recovery behavior.

AI Agent Evals: Build a Test Set That Finds Real Failures

An agent can produce a plausible final answer after taking the wrong path. Evaluation should inspect both the result and the actions. The OpenAI evaluation best-practices guide (https://developers.openai.com/api/docs/guides/evaluation-best-practices) describes evals as structured tests for variable model behavior. Its hosted Evals platform has a deprecation timeline, but the test-design principles remain useful with any evaluation runner.

Start from user tasks

Collect a small set of real requests. Include a normal case, missing data, ambiguous input, a permission denial, and a backend error. Write down the acceptable answer and the tool behavior you expect before running the agent. Avoid building a test set only from examples that already work.

Score the path as well as the answer

For each run, capture which tools were called, what arguments were used, whether the target was authorized, and whether the agent stopped when it should. A correct answer achieved by reading a private record is a failure. A refusal after a transient tool error may be a recoverable failure rather than a success.

Use a few clear labels

Mark each case correct, partly correct, or incorrect, with a reason. Record unsupported claims and wrong citations separately. If human review is needed, keep the rubric short enough that two reviewers can apply it consistently.

Make it continuous

Run the set when you change prompts, tools, models, or permissions. Add cases from actual incidents and user complaints. Keep the old cases so a fix in one area does not hide a regression elsewhere. The useful metric is performance on your own tasks, not a generic leaderboard score.

Primary source: OpenAI evaluation best practices (https://developers.openai.com/api/docs/guides/evaluation-best-practices).

Editorial note: Prepared with AI assistance and checked against the linked primary documentation on 15 November 2026.