AI evals beyond the demo: 15 practices for reliable systems

Calling a language model proves you can get a response. It does not establish that your product handles the situations your customers depend on. Evals turn those expectations into repeatable tests, then help you decide whether a change deserves to ship.[1][2]

This follow-up moves beyond the first spreadsheet exercise. It maps fifteen practices for builders who already work with prompts and application programming interfaces (APIs). You do not need every technique immediately: start with your failures, then add the measurements your decisions require.

1. Study errors before choosing metrics. Read failed examples before building a dashboard. Separate incorrect answers from missing information, broken tools and policy violations. For a refund assistant, confusing eligibility with a completed payment is a different failure from retrieving an outdated policy. Build checks around those distinctions.[1] Before launch, use realistic test scenarios rather than waiting for customers to encounter preventable failures.

2. Treat your golden dataset as a maintained reference. A golden dataset is a reviewed collection of test inputs and expected behavior, not an infallible answer key. Version the cases and rubrics alongside the system configuration. Include normal requests, boundary cases and harder tasks; accept valid alternative answers. Reserve cases you do not use while tuning, and regrade affected results consistently when a rubric changes.[1][2]

3. Generate synthetic cases with a coverage plan. Synthetic data means invented examples, including examples generated by a model. Vary intent, language, ambiguity and difficulty rather than producing paraphrases of one question. Review both the inputs and proposed answers before treating them as references. Keep synthetic stress tests distinct from representative traffic samples: an invented mix cannot establish how frequently failures occur in production.[2]

4. Use code for requirements code can check. Check structured outputs against their JSON schema, required fields and allowed values before asking a model to judge prose. Use text patterns for format requirements, not as evidence of meaning. Test generated SQL and code in restricted environments against expected results: successful execution does not prove correctness. Deterministic checks complement human or model judgment; they do not replace it.[1][2]

5. Separate retrieval from answer quality. Retrieval-augmented generation (RAG) searches for information before generating an answer. With labelled relevant documents, Recall@K measures the fraction retrieved within the first K results; mean reciprocal rank (MRR) measures how early the first relevant result appears. Normalized discounted cumulative gain (nDCG) rewards useful ordering and supports graded relevance.[3]

Choose metrics that fit the information needed. MRR alone cannot show that a question requiring several sources has enough evidence. Separately check faithfulness, whether the answer’s claims follow from the retrieved material.[4] A faithful answer can still rely on outdated or incorrect material, so assess source quality and answer correctness too.

6. Inspect agent actions without prescribing one path. An agent’s trajectory is its sequence of observable actions. Record tool calls, arguments, returned results and state changes, with sensitive data protected. Check mandatory safeguards, such as authorization before payment, but allow different valid routes to completion. Anthropic warns that rigid tool-sequence grading can penalize legitimate solutions.[1] Do not assume access to every internal reasoning step or treat a generated explanation as proof of what happened.

7. Distinguish occasional success from consistency. Pass@k asks whether at least one of k attempts succeeds. Pass^k asks whether all k attempts succeed.[1] A task with both passing and failing attempts satisfies the first condition but not the second. Reset the environment between attempts and report the retry budget. At-least-one success is useful only when your workflow can recognize and select the successful result; it does not mean every customer receives one.

8. Test conversations, not only isolated questions. Give a simulated user goals and constraints that the assistant must uncover through conversation. Test corrections, changed requirements and information carried between turns. A hypothetical customer might change the delivery address, then ask the assistant to confirm it: checking only the final sentence can miss an unchanged order record. Inspect state and review simulations for unrealistic behavior; simulated users are not evidence of actual user satisfaction.[1]

9. Evaluate the judge itself. A large language model (LLM) judge is another language model applying your marking guide. Compare its decisions with reviewed human labels before relying on it at scale.[1][2] Define what counts as a positive label: if positive means “failure detected,” a false negative is a missed failure. Track both missed failures and false alarms by failure category, not only overall agreement. Recheck calibration when the judge or rubric changes.

10. Control bias in pairwise comparisons. When a judge chooses between two answers, presentation can affect the verdict. Randomize which answer appears first and repeat a sample with positions swapped. Make the rubric reward correctness and relevant completeness rather than length. Position and verbosity bias are documented limitations of model judging.[2] Hide model identities where practical, and investigate inconsistent preferences instead of treating every vote as trustworthy.

11. Quantify uncertainty before declaring a winner. Compare versions on the same cases and analyze paired differences, not only their separate averages. Anthropic recommends reporting uncertainty and accounting for related questions rather than treating them as independent evidence.[5] Learn when confidence intervals and bootstrapping, resampling observed data to estimate uncertainty, suit your design. Neither fixes biased sampling or a faulty grader. Statistical significance also does not establish that an improvement matters commercially.

12. Put regression checks in the release process. Continuous integration and delivery (CI/CD) automates checks around software changes. Run relevant evals when prompts, models, tools or retrieval settings change. Keep a bounded regression suite for proposed changes and schedule broader tests separately. Define release-blocking failures beforehand and retain the dataset, grader and configuration versions behind each result. Anthropic’s introductory course demonstrates a pull-request smoke eval and batch evaluation.[7]

13. Monitor live behavior without duplicating real actions. Offline tests cannot anticipate every production input. Sample permitted live traffic and grade it asynchronously, outside the response path. Shadow testing sends a copy of requests to a candidate while returning only the production system’s answer.[6] For agents, also block real writes, messages and transactions in the shadow environment. Hidden responses alone do not prevent side effects, and shadow runs cannot measure users’ reactions to answers they never saw.

14. Measure the cost of a successful outcome. Track token usage and total task costs, including retries, retrieval and tools. Divide the total cost of all attempts by successful tasks, rather than counting only the spending on successes; if none succeed, report that rather than a finite cost per success. Track end-to-end latency too: p50 is the median, while p95 is the 95th percentile and exposes slower experiences. Keep time-to-first-token separate from time-to-completion, and report failures and timeouts alongside latency.[1]

15. Turn failures into durable tests. After a meaningful incident, preserve a privacy-safe case, verify the expected behavior and add a regression check. Group duplicates so one recurring problem does not overwhelm the dataset. Keep a representative evaluation set alongside targeted failure tests, and compare versions on the same dataset version.[1][2] A changing score on a changing test set is not evidence of improvement by itself.

Continue with Anthropic’s introduction to evals. Introduction to evals at Claude Academy is a practical next step for readers ready to work with an API. Its syllabus covers building a small dataset, grading outputs, improving prompts and maintaining checks after launch. Follow the course link in source [7] below.

Check the prerequisites before enrolling. The course uses Python and a GitHub Actions workflow, assumes API familiarity or completion of Claude Platform 101, and requires a Console account and API key for your own eval. It explicitly excludes agent evaluation and latency measurement. It is a foundation for part of this roadmap, not a course covering all fifteen practices.[7]

Source note: this article synthesizes engineering guidance, metric documentation and a public course syllabus checked on 8 November 2026. It is not a product benchmark or a claim to have completed the course. Examples are hypothetical; the recommendations do not establish any particular system’s reliability.