Do not accept a pilot because its demonstration handled several examples convincingly. Request permitted cases reflecting the intended use, comparison with the current method, a breakdown of failure types, and evidence that reviewers can detect and correct errors. Then make a specific decision: limited release with conditions, revision and retest, or stop. Polished output alone does not answer that question.
This discussion begins after selecting a narrow task, such as suggesting a category for an incoming document before an authorized employee reviews it. It does not revisit whether the company needs AI at all, or assume that a new model must be trained. The focus is acceptance evidence: what was tested, how it was judged and where the findings cannot be generalized.
Define acceptable output before inspecting results
Specify what the system must provide, when it should abstain and what it must never do. For document classification, the output might be a proposed category, a brief rationale and an indication that source text is incomplete. Successful classification does not authorize sending the document or approving a financial transaction. Acceptance of one task does not automatically expand authority.
NIST's measurement guidance emphasizes measures appropriate to the use and documentation of test sets and measurement limitations. That helps frame evidence requirements; it is not NIST approval or a universal accuracy target. NIST Measure guidance.
Building and evaluating the feature can fit within AI & Machine Learning Engineering. Unresolved data permissions or task boundaries belong back in AI Readiness Assessment, rather than evaluating material the project is not authorized to use.
Build a test set with an explanation and an owner
Do not select only easy examples or export customer files into a new environment without permission. Document sources, usage rights, redaction and the languages, document types and text quality represented. A small or selectively assembled set must be described honestly. More cases do not compensate for omitting an important category of use.
Separate development examples from cases reserved for an acceptance decision. When instructions are changed after inspecting a reserved case, that case remains useful for regression testing, but no longer provides the same kind of fresh independent evidence. Record the change instead of relabeling it as a new test.
| Case category | What to examine |
|---|---|
| Clear document within scope | Correct suggestion and consistent output format |
| Several plausible categories | Visible ambiguity rather than unsupported certainty |
| Missing or unreadable text | Abstention or a request for the missing information |
| Out-of-scope material | No invented category or unauthorized action |
| Content attempting to direct the tool | Document text does not become operating authority |
Fictional example: classifying incoming documents for review
Suppose an illustrative set contains 60 permitted documents: 30 clear cases, 12 ambiguous cases, 10 with missing text and eight outside the task. This invented composition explains evaluation design. It is not necessarily a sufficient sample, an accuracy benchmark or a LAB result.
A suitable domain reviewer defines judgment rules before execution, including cases that must return to a person instead of receiving a forced category. The current human or simple-rule method is examined on the same cases under disclosed comparison conditions. Record work and review time, corrections needed and disagreements between human reviewers themselves.
If two reviewers disagree about the correct category, resolve the classification rule or label the case ambiguous. Do not force a single answer merely to improve a result table. In a fictional qualitative finding, the system suggests a plausible category for an incomplete document but fails to reveal the missing text. The issue is not only categorization: the presentation makes the reviewer believe the evidence is complete.
The owner asks for changes to the display and abstention rule, then retests affected cases and cases not used to make the correction. A single aggregate score should not conceal this failure class.
Separate failure types rather than reporting one average
Distinguish an easily noticed wrong suggestion, unsupported information, failure to abstain, exposure of an inaccessible source and action beyond authority. Some categories may block release even when other cases work well. The owner and appropriate specialist should define that treatment before seeing results, according to the intended use.
Software Quality Assurance can support documented cases, versions, outcomes and repeatable checks. User Research can examine whether users notice errors and what correction costs them, through a defined study with permitted participants and suitable facilitation. A small session is not automatically representative of all users.
Evaluate review effort inside the actual workflow
Do not compare answer-generation time with an employee's entire task time. Include source reading, verification, correction and exception handling. Can the reviewer open the evidence, distinguish a suggestion from an approved fact and reject it? Human review is not an effective control if information, time or rejection authority is missing.
The pilot should remain bounded against independent consequential action, with source and output permissions reviewed. A successful evaluation does not grant access to new sources or authorize automatic publication. Model, instruction, data-source and approval-interface changes can alter behavior. Tie acceptance to a version, operating boundary and review point.
The acceptance decision record
Request a concise record covering task, version, data, coverage, exclusions, baseline comparison, failure categories and review burden. State release conditions, allowed use, expansion conditions and the owner monitoring new cases. If evidence does not establish usefulness or appropriate boundaries, revision or stopping is a legitimate result.
The decision should also identify what would invalidate it. A new language, document class or source collection may require additional evidence rather than inheriting approval from the original set. Do not quietly change the metric after results arrive to make the pilot appear successful.
Frequently asked questions
How many cases are necessary?
There is no universal count that validates every task. Size and variety depend on risks, patterns and the decision required, with confidence and coverage limits made explicit.
Is an accuracy percentage sufficient?
Not if it hides different failure consequences or substantial review effort. Show important cases and their causes instead of replacing professional judgment with an average.
Can the developer be the only reviewer?
Task correctness needs a suitable domain reviewer, with independence from development where the risk warrants it. Do not assume that capability exists without confirming it.
What should we provide for an evaluation discussion?
Bring the chosen task, authority boundaries, permitted redacted examples, current method and acceptance owner. These support an evaluation scope within the project, not a presumed new standalone service or guaranteed accuracy.
Reference
NIST AI RMF Playbook: Measure, reviewed 26 October 2026. The example dataset is fictional, not a benchmark or success threshold.
