Production AI cannot be validated by asking a few people whether the demo looks good. Generative systems are nondeterministic, model behavior changes across versions, retrieval adds another source of variability, and agents can take different tool paths even when the final answer looks similar.
Evaluation is the engineering layer that turns those uncertain behaviors into measurable release decisions. The goal is not to produce one impressive score. The goal is to define what success means for the actual application, test representative cases continuously, identify where failures occur, and stop regressions before they reach users.
This guide describes a provider-portable evaluation operating model. Specific vendor evaluation products change over time—for example, OpenAI's legacy Evals platform is being deprecated in late 2026—but the methodology of datasets, graders, traces, human calibration, and continuous regression testing remains fundamental.
For architecture context, see How to Evaluate an AI Development Company, AI Agent Architecture, and Enterprise RAG Architecture.
1. Define the failure cost before the metric
An eval should measure the failure that matters to the application.
A marketing draft with a weak sentence and an agent issuing the wrong refund are not equivalent errors. Before choosing a score, define:
- what the system is expected to do;
- what constitutes a minor, major, or unacceptable failure;
- which failures can be corrected by a human;
- which failures create financial, security, legal, or operational risk;
- what quality threshold is required before release.
This determines which cases deserve deterministic pass/fail gates and which can use softer preference or quality scores.
2. Build a representative evaluation dataset
Your test set should resemble production traffic, not the prompts the development team wishes users would send.
Include a mix of:
- common happy-path requests;
- edge cases;
- ambiguous inputs;
- historical production failures;
- adversarial or prompt-injection cases where relevant;
- permission-sensitive examples;
- negative cases where the correct behavior is to refuse, escalate, or say evidence is insufficient;
- long, messy, or partially incorrect user inputs;
- examples from different customers, domains, languages, or document types represented in production.
OpenAI's current evaluation guidance recommends task-specific datasets, production data where possible, edge/adversarial cases, continuous expansion from logs, and human calibration of automated metrics.
3. Keep development and held-out evaluation sets separate
If the team repeatedly tunes prompts against the same examples used for final scoring, the evaluation gradually becomes a training set.
Maintain separate sets for:
- development: cases engineers actively inspect and optimize against;
- regression: known failures that should never return;
- held-out evaluation: cases reserved for unbiased comparison;
- production monitoring: sampled real traffic used to discover new failure modes.
Version the dataset so every model, prompt, retrieval, or workflow change can be compared against the same benchmark.
4. Evaluate components before the end-to-end system
When a complex system fails, a single overall quality score is hard to debug.
Break evaluations along architecture boundaries:
| Component | Example evaluation questions |
|---|---|
| Intent/routing | Was the request classified or routed correctly? |
| Retrieval | Were the right documents/chunks found and authorized? |
| Generation | Was the answer correct, grounded, relevant, and complete? |
| Tool use | Was the correct tool chosen with correct arguments? |
| Agent workflow | Did approvals, handoffs, limits, and stopping conditions occur correctly? |
| Safety/policy | Did the system respect content, permission, and action boundaries? |
| Operational | Did latency, cost, retry count, and failure rate stay within budget? |
Component evals tell you what broke. End-to-end evals tell you whether the user outcome is acceptable. You need both.
5. Prefer deterministic graders when the requirement is deterministic
Use code-based or exact graders when success can be stated precisely.
Examples include:
- valid JSON schema;
- correct classification label;
- required field present;
- SQL or code passes tests;
- tool name matches expected tool;
- tool arguments match known values;
- no forbidden endpoint was called;
- a required citation is present;
- latency or cost remains below a threshold.
Deterministic graders are reproducible and inexpensive. Do not replace an exact assertion with an LLM judge just because the application uses an LLM.
6. Use model judges for qualities that require interpretation
Some properties are difficult to encode in simple rules: completeness, relevance, clarity, groundedness, tone, or whether a response follows a nuanced rubric.
For those cases, a model judge can score outputs against explicit criteria. Better judge prompts:
- define the criterion narrowly;
- include examples of strong, acceptable, and poor outputs;
- ask for structured scores or classifications;
- prefer pairwise comparisons when comparing two candidates;
- avoid asking one judge to score too many unrelated properties at once.
OpenAI's evaluation guidance notes that language models are generally stronger at discriminating between options than performing unconstrained open-ended judging, making pairwise or rubric-based evaluation useful patterns.
7. Calibrate automated graders against humans
An automated evaluator can be consistently wrong.
Before using a model judge as a release gate, compare its scores with domain-expert judgments. Measure agreement, inspect disagreements, and revise the rubric until the automated grader reflects the quality standard the business actually wants.
Human evaluation remains especially important for new tasks, subtle domain requirements, policy interpretation, and cases where there is no objective ground truth.
8. Evaluate RAG retrieval separately from RAG answers
A grounded answer cannot recover evidence that retrieval never supplied.
For RAG, evaluate at least two layers:
- retrieval quality: recall, ranking, precision, missed relevant documents, irrelevant context, authorization, and freshness;
- response quality: groundedness, relevance, completeness, citation quality, and handling of insufficient evidence.
Microsoft Foundry's current RAG evaluators explicitly separate document retrieval and retrieval-context quality from groundedness, relevance, and response completeness. That separation should exist in any provider stack.
9. Evaluate agent traces, not only final answers
An agent can arrive at a correct answer through an unsafe or wasteful path.
Trace-level evaluation should check:
- which tool was selected;
- whether tool arguments were correct;
- whether a handoff occurred at the right point;
- whether the agent requested approval before a sensitive action;
- whether retries were bounded and idempotent;
- whether unnecessary tools or subagents were invoked;
- whether the run stopped within its configured budget;
- whether policy or guardrail violations occurred.
OpenAI's current agent-evaluation guidance recommends trace grading specifically because end-to-end traces expose model calls, tool calls, guardrails, and handoffs. That provides a much clearer diagnosis than final-output grading alone.
10. Add safety and authorization tests as first-class evals
Safety should not be a separate manual checklist run after quality evaluation.
Build test cases for:
- prompt injection;
- data exfiltration attempts;
- cross-tenant access;
- unauthorized tool use;
- malformed tool arguments;
- requests to bypass approval;
- sensitive-data handling;
- policy boundary cases.
For agents and RAG systems, evaluate whether the application architecture blocks the unsafe action, not merely whether the model says it should not do it.
11. Measure quality together with latency and cost
A change that improves quality by one point but triples latency may not be a production improvement.
Record:
- end-to-end latency;
- model latency;
- retrieval/reranking latency;
- tool-call count;
- retry count;
- input/output tokens;
- model and tool cost;
- failure/timeout rate.
Compare these metrics on the same evaluation set. The winning architecture is the one that meets the required quality and risk threshold within acceptable operational budgets.
12. Turn evals into release gates
Evaluation becomes operationally useful when a regression can stop a release.
A release pipeline can enforce gates such as:
- no critical safety failures;
- tool-selection accuracy above a threshold;
- RAG retrieval recall not below baseline;
- groundedness above the production threshold;
- no regression on high-risk cases;
- latency and cost within agreed budgets;
- human approval on any intentionally changed risk boundary.
Not every metric needs a hard threshold. Some can be trend indicators. But critical business invariants should have explicit release criteria.
13. Compare versions, not isolated scores
An eval score has meaning when it is attached to a versioned system.
Record the versions of:
- model and model settings;
- system/developer prompts;
- tool definitions;
- retrieval configuration;
- embedding/reranking models;
- agent routing and workflow code;
- evaluation dataset;
- grader configuration.
This turns "the score dropped" into an actionable comparison between known configurations.
14. Feed production failures back into the dataset
The best evaluation set is never finished.
Monitor user feedback, failed traces, escalations, support tickets, low-confidence retrievals, unusual tool sequences, and operator overrides. When a meaningful failure occurs:
- capture the input and relevant context safely;
- label the expected behavior;
- add the case to the regression set;
- reproduce the failure;
- fix the root cause;
- verify that the new version passes the case without breaking others.
This creates an improvement flywheel in which production incidents become durable tests.
15. Evaluation anti-patterns
| Anti-pattern | Why it fails |
|---|---|
| "It seems better" | Vibe-based evaluation is not reproducible. |
| Only academic benchmark scores | Benchmarks may not represent the application's real task distribution. |
| One overall quality score | Root causes across retrieval, tools, policy, and generation remain hidden. |
| Only happy-path cases | Real production edge cases and adversarial behavior stay untested. |
| LLM judge for exact requirements | Introduces unnecessary nondeterminism where code assertions would be stronger. |
| Automated judge never calibrated by humans | The evaluator may optimize for the wrong quality standard. |
| Final-answer grading for agents | Unsafe or wasteful execution paths can pass unnoticed. |
| Eval dataset never updated | Production failures repeat because they never become regression tests. |
16. A production evaluation operating model
A mature evaluation loop is straightforward:
- Define risk-based success criteria.
- Build representative datasets.
- Choose deterministic, model-based, and human graders appropriately.
- Evaluate components and end-to-end behavior.
- Version the full system configuration.
- Run regression suites before release.
- Gate critical failures.
- Monitor production traces and feedback.
- Turn new failures into permanent tests.
This is the practical meaning of eval-driven development for AI systems.
Operational companion: Read MLOps vs LLMOps: What Changes for Production Generative AI for how evals become part of CI/CD, release manifests, monitoring, rollback, and production feedback loops.
Sources and further reading
- OpenAI — Evaluation best practices
- OpenAI — Evaluate agent workflows
- OpenAI — Trace grading
- Microsoft Foundry — Evaluate generative AI applications
- Microsoft Foundry — RAG evaluators
For implementation, review OSYSTIC AI Reliability & LLMOps or send a technical brief.