How to Evaluate AI Output Quality Before Launch

Why Demos Mislead

Almost every AI system looks good in a demo, because the person running the demo picks questions that work. Customers do not. They send half-finished sentences, ask about things you never documented, mix two requests in one message, and occasionally try to trick the system. An evaluation is how you find out what happens on that traffic before your customers do.

Language models are also non-deterministic: the same question can produce different wording, and occasionally a different answer, on different runs. That is one more reason to judge a system on many cases rather than a handful. The NIST Generative AI Profile names confabulation, meaning confident but false output, as a core generative AI risk, and testing is the main way to measure how often it happens in your specific use.

What to Measure

Break quality into separate criteria and score each one. A single "good or bad" judgment hides where the problems are.

• Correctness: are facts, numbers, dates and names right?
• Groundedness: is every claim supported by your data or the tool results, rather than the model's general knowledge?
• Policy compliance: does it follow your rules on pricing, promises, refunds and topics to avoid?
• Handoff decisions: does it escalate when it should, and avoid escalating when it can handle the request?
• Format and completeness: does it include required fields, links or next steps?
• Tone: does it sound like your business?
• Safety and security: does it resist attempts to override its instructions or reveal data it should not?

Weight these by consequence. For a support assistant, one wrong refund promise matters more than ten slightly stiff replies. Classify failures as critical (wrong prices, unsafe advice, data exposure, unauthorized actions), major (wrong answer the customer would act on) and minor (style), and set thresholds per class.

Three Ways to Grade

Code-based checks

Anything objective should be checked by code: did the output include the order number, is the date valid, did the agent call the booking tool with the right appointment type, does the reply avoid phrases you have banned. These checks are fast, cheap and consistent, so they can run every time anything changes.

Human review

Judgment calls such as whether an answer is helpful, appropriately cautious or on-brand need people, at least at first. Give reviewers a written rubric with a short scale and an example of each score. Have two reviewers grade the same sample independently; where they disagree, the rubric is unclear and needs fixing. The best reviewers are the staff who do the work today.

AI as a grader

Both Anthropic's and OpenAI's evaluation guides describe using a model to grade outputs against a rubric, which makes it practical to score hundreds of cases. It works well when the grader gets a clear rubric and the reference answer, and is asked one specific question at a time. Validate it first: run it on cases people have already scored and use it only for criteria where it agrees closely. Recheck that agreement whenever you change the grader prompt or model.

Thresholds and the Launch Decision

Decide pass criteria before you see results, or you will talk yourself into whatever the numbers say. A reasonable pattern for a customer-facing assistant: zero critical failures on the full set, a high pass rate on correctness and policy, correct handoff on nearly all cases that require one, and tone issues tracked but not blocking.

Compare against the current human baseline where you can. If staff answer a category correctly 92 percent of the time, an AI at 95 percent with good handoffs may be an improvement, even though it is not perfect. Then launch in stages: supervised mode with humans approving outputs, then autonomy for categories that consistently pass. Our guide to running an AI pilot covers that staging.

Keeping Quality Up After Launch

Evaluation is not a one-time gate. Every change can shift behavior: a new instruction, a new document, a tool change, or a model update from the provider. Keep the test set in version control, rerun it on every change, and track scores over time. Sample live conversations weekly, and whenever you find a failure, add that case to the test set so it can never quietly come back. For ways to reduce false answers in the first place, see what AI hallucination is and how to prevent it. If you want an outside team to set up evaluation for an existing system, AI automation services include it.