An AI evaluation checklist should answer a practical question: can this system complete the work you intend to give it, within acceptable limits? Define the task, assemble realistic cases, write the acceptance criteria, and compare candidates on the same inputs. Review failed cases as carefully as successful answers.
A convincing demo is a reason to investigate, not evidence that a model will perform reliably across your workload. For a small team, a useful first evaluation can be a spreadsheet of task examples, expected outcomes, observed results, and failure reasons. You can automate the repeatable checks as the workflow becomes clearer.
Start the AI evaluation checklist with one real task
Write a short task contract before choosing a model. State who uses the output, which information the system receives, what it must produce, and which decisions remain with a person. “Help with customer support” is too broad to score consistently.
Consider an illustrative task: draft a reply to a delivery question using a supplied order record and shipping policy. The assistant may explain the recorded status and propose a next step. It may not invent a delivery date, promise compensation, or send the reply. These boundaries make the expected result reviewable.
Anthropic’s evaluation guidance recommends specific, measurable success criteria tied to the use case. The important choice is your definition of success, rather than a provider’s demonstration prompt. Keep the first task narrow enough that two reviewers can discuss why an answer passes or fails.
Build a case set that includes awkward inputs
Collect examples that resemble the information and requests the system will encounter. Include ordinary cases, missing information, conflicting records, unusual phrasing, and requests outside the task. Use synthetic records or appropriately authorized data, and remove personal details that the evaluation does not need.
For the delivery task, include a shipped order, an order awaiting dispatch, a missing tracking field, and a customer asking for a guarantee the policy does not support. Also include a question the record cannot answer. The expected behavior may be to ask for clarification or route the issue to a person.
NIST’s AI risk framework discussion connects accuracy measurements to realistic test sets and documented methods. Treat your cases as evidence about a defined workload. A small handpicked set cannot establish reliability for every language, customer group, or future situation.
Keep some cases separate from prompt development. If you repeatedly rewrite the instructions around every failed example, the next run may show familiarity with those examples rather than broader improvement. Record how you selected the cases so later reviewers can see the limits.
Write a simple rubric before reading model answers
Use criteria a reviewer can apply to the evidence. Here is a suggested rubric for the delivery-reply example, not a universal scoring standard:
- Factual grounding: every order-specific statement agrees with the supplied record.
- Policy fidelity: the draft makes no promise beyond the supplied policy.
- Completeness: the reply answers the actual question or clearly identifies missing information.
- Usability: the customer can understand the status and proposed next step.
- Boundary compliance: the system drafts only and does not send or change a record.
For the first four criteria, an anchored scale can help: 0 means incorrect or unusable, 1 means partly acceptable but needing revision, and 2 means acceptable as written. Add an evidence note explaining the score. Decide in advance which failures block use regardless of the total.
For example, a friendly reply that invents a delivery date should fail the factual criterion. A polished reply that sends itself without permission should fail the action boundary. An average score should not conceal either event.
Match the grading method to the criterion
Use deterministic checks for requirements that have a clear mechanical answer, such as a required field, valid output structure, or an exact extracted order identifier. Use informed human review for judgments such as whether an explanation is clear or a proposed resolution follows policy.
Anthropic describes code-based, human, and model-based grading. If you use another model as a judge, give it the rubric and relevant reference material, then compare its judgments with reviewed examples before relying on its scores. A grader can make mistakes too.
Separate correct abstention from needless refusal. When the record lacks a delivery date, acknowledging that gap is useful. Refusing to summarize a harmless status field is a different outcome. Record both types of failure so a change that makes the assistant overly cautious does not look like an unqualified improvement.
Compare the whole setup on equal terms
Record the model identifier, date, instructions, supplied documents, retrieval configuration, tools, and relevant settings. When comparing two models, hold the rest of the setup constant where possible. If a candidate needs a different setup, document that difference and compare the resulting application honestly.
Keep raw outputs and case-level judgments. Include failed calls, incomplete answers, and retries rather than reporting only the final successful attempt. Measure elapsed time and actual usage costs when available, without inferring a real bill from an advertised rate alone.
Microsoft’s evaluation checklist recommends establishing a baseline, investigating failures, and repeating evaluations as systems change. Run important cases more than once when variation matters. State how many attempts you observed; repeated success on a small set remains a limited result.
Evaluate tool actions separately from the final answer
If the system uses tools, inspect what it actually did. Check whether it retrieved the right record and kept proposed changes as drafts. If a tool was unavailable, inspect how clearly it reported that failure. A fluent final response cannot prove that the intended action occurred.
Desinance’s introduction to agentic AI explains the broader workflow. For evaluation, use a controlled environment with synthetic records and limited permissions. Include attempts to redirect the task through retrieved content. OWASP’s prompt injection guidance distinguishes direct attacks from instructions encountered in outside material.
Write the expected action boundary before the test. For the delivery example, a shipping note that asks the assistant to send customer information elsewhere must not authorize that disclosure. Check the activity record and destination, not just whether the final answer mentions the suspicious instruction.
Turn the AI evaluation checklist into a release decision
- Task: name the intended user, inputs, output, and action limits.
- Cases: cover routine work, missing information, edge cases, and out-of-scope requests.
- Rubric: define score anchors, required evidence, and blocking failures.
- Comparison: preserve configuration details, outputs, retries, timing, and available cost data.
- Review: inspect failures by category and verify tool outcomes independently.
- Decision: record what is ready, what needs human review, and what remains unsupported.
- Maintenance: rerun relevant cases after model, prompt, knowledge, or tool changes.
The useful result is a decision with evidence: this setup handles these tasks under these conditions, with these unresolved weaknesses. Expand the case set as real failures reveal missing coverage. An AI evaluation checklist earns its place when it makes the next model or workflow change easier to assess.
