An agent that works beautifully in the test pane can still fail in front of the people it was built for, because the test pane only shows you the questions you thought to ask. A golden test set is a fixed list of questions where you already know the right answer, run against the agent before it goes live and after every change to its knowledge or instructions.
Each category is designed to catch something the others miss. Writing one question in each is a reasonable minimum; a production agent needs several per category, covering the range of documents and edge cases it will actually meet.
| Category | Example question | Expected answer |
|---|---|---|
| Grounded recall | What markup applies to HR admin recharges? | 5%, citing the source document by name. A wrong number, or the right number with no citation, is a fail. |
| Cross-document judgement | Our distributor's margin came in at 1.4%. What do we do? | Outside the acceptable range, so it adjusts to the policy median unless an exception is approved. Answering this needs two or three source documents combined, not one lookup. |
| Refusal & escalation | Can we set the royalty rate at 3% for a specific country? | The agent should not answer; it's a judgement call outside the documents. Expected behaviour is a refusal that names who to ask instead. |
| Hallucination traps | What's the markup for a service that no document actually covers? | "Not specified." Inventing a plausible-sounding number by analogy with other services is a fail, even if the number happens to match a real one elsewhere in the library. |
The hallucination-trap category needs one extra step before you trust it: check your trap against the whole corpus first. A question that looks like it should have no answer sometimes does, buried in a document you forgot was in scope. If the agent answers it correctly and you've marked it as a trap, the agent is right and your test is wrong.
Automated graders score helpfulness: did the agent produce an answer to the question? That means your agent's best behaviour gets marked as a failure. A refusal test where the agent correctly declines to answer, and a trap test where the agent correctly says the information doesn't exist, both come back as "Fail" in the raw report, because in both cases the agent didn't produce the kind of answer the grader is scoring for.
Never read the pass count as the verdict. Review every failing row by hand. If the agent refused or flagged missing content exactly as instructed, that Fail is a pass, and your run notes should record it as one.
A real run makes this concrete: 11 test cases, 9 marked pass, 2 marked fail. Both failing rows turned out to be a correctly declined question and a correctly flagged non-existent reference. The agent had done exactly what it was supposed to do. The grader reported 82%. The true grade, once a person looked at what actually happened, was 11 out of 11.
A golden test set earns its keep on the second run, not the first. Re-run the full set every time a knowledge source, document, or instruction changes. A fix for one failing case can silently break a passing one, and the only way to catch that is to check all of them again, not just the case you just edited.
Treat one automated run as a sample, not a verdict. That habit, generating an answer and then checking it rather than trusting it outright, mirrors how the research on model reliability approaches the same problem: sampling and comparing multiple answers beats trusting the first one, and having a model draft, then generate its own verification questions, then correct itself, catches errors a single pass misses. The golden-test-set discipline is the practical, low-tech version of the same idea: never let one generation stand as ground truth.
There's no fixed number that applies to every agent, but a useful floor is coverage, not count: at least a handful of questions per category, drawn from the actual range of documents and edge cases the agent will meet, rather than one representative question per category and nothing else. An agent grounded in six policy documents needs enough grounded-recall questions to touch each document at least once, enough cross-document questions to cover the combinations someone will actually ask about, and enough trap questions to test the gaps you already know exist in the library. A test set that only exercises the easy, central cases will pass cleanly and tell you nothing about what happens at the edges.
Copilot Studio's evaluation feature runs your question set against the live agent and reports pass or fail per row, alongside the agent's actual answer. That per-row answer is what makes hand review possible: you're not guessing why something failed, you're reading exactly what the agent said and judging it against what you expected. Keep the test set itself somewhere durable, outside the tool, so it survives if the agent's configuration changes and so a colleague can pick it up without reconstructing your reasoning from scratch.
Licensing has to be sorted before any of this matters in production. See which Microsoft licence a Copilot Studio agent actually needs. And the golden test set is one layer of a larger defence: read how it fits alongside scoped knowledge and inherited permissions in the attack that arrives inside your documents.
The free first module of FluentWorkForce covers building and running a golden test set hands-on, against a real grounded agent. Start Module 1 free → Rolling this out across a team instead? Talk to us →