The Agent QA Checklist

Use this before an AI agent runs unattended on project work, and again whenever its brief, tools, model or data source changes. It sets up test scenarios in two gates. Gate one watches the agent’s reasoning on inputs with a planted problem. Gate two checks the finished output against source data. You design the scenarios. The agent can run most of them itself.

The frame

Agent and task: [what it does, on what schedule]
Owner: [who signs off go-live]

Gate one: in-task evaluation
Test 1, wrong problem: [brief it with one ambiguous term] Expected: it asks, or states the assumption it made. Result:
Test 2, early wrong step: [plant one wrong input near the start] Expected: it catches or flags it, and later steps don't inherit it. Result:
Test 3, dropped context: [a detail given at the start is needed at the end] Expected: the detail is used. Result:
Self-report asked for: [sources used, assumptions made, confidence in the riskiest step] Yes / No

Gate two: output verification
Test 4, edge case: [a value the workflow didn't plan for: an unmapped status, an ambiguous date] Expected: flagged, not guessed. Result:
Source check: [compare a set number of outputs line by line with the source data] Result:
Acceptance criteria: [what a usable output must contain, taken from the task's Check]

Decision: [Go / Go with conditions / Not yet]
Rerun when: [brief, tools, model or data source change]

Have an AI draft the test scenarios

Below is the brief for an AI agent that will run on my project, and a description of the data it reads.

Write test scenarios for a pre-launch check in two gates.

Gate one, the agent's reasoning: one scenario where a term in the brief is ambiguous, one where a single input near the start is wrong, and one where a detail given at the start is needed at the end. For each, say exactly what to change in the input and what a good agent would do.

Gate two, the output: one edge case in the data that the brief doesn't plan for, and a method for checking outputs against source data. Then list acceptance criteria for a usable output.

Brief: [paste the brief]
Data it reads: [describe the sources, fields and formats]

Before you paste a real brief or real project data into a third-party AI tool, check your organisation’s policy on client and commercially sensitive material. Test data with names and figures swapped out works just as well.

Worked examples

The entries below are illustrative.

Weekly status report agent

Agent and task: Drafts the weekly status report from the task board every Friday
Owner: PM

Test 1, wrong problem: Brief says "report tasks completed" with no definition of complete. Expected: asks which board columns count, or states one. Result: Counted "In review" as complete without saying so. Fail.
Test 2, early wrong step: One milestone date entered a year late. Expected: flags the date. Result: Flagged it. Pass.
Test 3, dropped context: Brief says the sponsor wants budget first. Expected: budget leads. Result: Budget appeared third. Fail.
Self-report asked for: Yes. It listed the columns it read, which exposed Test 1.

Test 4, edge case: A task with status "Blocked - vendor". Expected: flagged as unmapped. Result: Reported as in progress. Fail.
Source check: Four weeks of reports compared with the board. Result: Progress overstated in all four.
Acceptance criteria: Complete means "Done" only; every RAG rating cites the workstream update it came from; budget first.

Decision: Not yet. Fix the definition of complete and the status mapping, then rerun Tests 1, 3 and 4.

Risk monitoring agent

Agent and task: Reads the RAID log and the dependency tracker daily and posts new or worsening risks
Owner: Delivery lead

Test 1, wrong problem: Brief says "worsening risks" with no measure. Expected: states the measure it uses. Result: Used its own scoring change and said so. Pass.
Test 2, early wrong step: One dependency linked to the wrong workstream. Expected: downstream analysis notes the mismatch. Result: Built two risks on the wrong link. Fail.
Test 3, dropped context: Brief excludes the decommissioned reporting module. Expected: no risks raised against it. Result: None raised. Pass.
Self-report asked for: Yes.

Test 4, edge case: A risk with likelihood blank. Expected: flagged for the owner. Result: Flagged. Pass.
Source check: Ten posts compared with the log. Result: Two rested on the planted wrong link, eight matched.
Acceptance criteria: Every risk cites a log row or tracker entry; no risk raised against excluded scope.

Decision: Go with conditions. Owner reads every post for two weeks; rerun Test 2 after the dependency check is added to the brief.

Notes on the gates

Gate one exists because an output can pass every format check while the reasoning behind it was wrong. Planting a problem and watching what happens is the only way to see that before go-live.

The self-report makes gate one faster to read. It is not a third gate, because the agent can be wrong about itself in the same way it is wrong about the work.

Gate two is where most teams already look. The source check turns “does it read well” into “does it match the data”.

Check the decision

Every answer should be yes before Go.

• Did each gate one test change only one thing in the input?
• Did someone compare outputs with source data, not just read them?
• Is every failed test either fixed and rerun, or covered by a condition on go-live?
• Is the rerun trigger written down where the agent’s owner will see it?

When it does not apply

If a person reads every output in full before it has any consequence, the review is the gate, and a lighter pass on gate two is enough. Run the full checklist when an agent will run without that read, or when you plan to remove it.

For the tools you already use every day, such as meeting summaries and drafted updates, the lighter version is a weekly spot-check of three outputs against their sources. When AI drafts the status report, the project manager role moves to three places sets that out as job two.

What this feeds

A Go decision is what moves a task to “Agent on its own” in the AI Delegation Register. The acceptance criteria come from the Check in the task’s Outcome Brief. Conditions on go-live become controls in your agent governance charter.