Test questions for the AI agent before knowledge goes live

Key takeaways

• A sample of live answers finds errors after customers have seen them; a test set finds them beforehand.
• Five question types belong in every set: standard case, edge case, question with no answer in the knowledge base, outdated information and question with a false premise.
• Passed means: correct AND backed by the expected source, or an honest handover to a human. A correct answer with the wrong source has not passed.
• The set grows from real queries and from every error that made it to the customer; it never shrinks.
• Shadow mode, in which the agent only produces drafts, is the measurement in operation, not the replacement for the set.

Why a sample is not enough

Anyone who runs an AI agent in support checks its answers at some point. The usual route is the sample: every week someone reads part of the conversations and judges whether the answer is correct. We described this procedure in our own post on sampling, and it remains correct. It has only one property you need to know: it finds errors after customers have seen them.

A test set turns the order around. It consists of questions whose correct answer is known, and it is asked before a change to the knowledge base is released. The principle comes from software development, where tests run on every change, and it applies to language models just the same: Hamel Husain, who describes evaluation methods for AI products, demands that such checks run fast and cheap so that they can be executed on every change ¹. The reason is the same as with software: a change in one place has side effects in others.

With a knowledge base these side effects are particularly inconspicuous. A new help article about cancellation can cause the agent, when asked a question about invoices, to suddenly cite the cancellation article, because its wording is closer to the question. Nobody touched the invoice article, and yet the answer to the invoice question has become a different one. A sample discovers this in the week afterwards. A test set discovers it before the release.

Where the questions come from

The set is not invented but collected. The best source is real queries from the inbox, and not the typical ones but the ones where support has already failed: questions that had to be asked twice, questions with an answer that was later corrected, questions that went to a human even though the answer was in the knowledge base. Each of these is a candidate.

The second source is the agent's own handovers. When the agent hands a question to the team because the knowledge base offers nothing, that is either correct, because there really is no answer, or wrong, because the answer is there and simply was not found. Both cases belong in the set, with different expectations. Anthropic's guide to building test cases expressly advises not to forget edge cases: irrelevant or non-existent inputs, overlong inputs, and in chat also poor or inappropriate user inputs ². For support questions this covers exactly the cases that rarely appear in a sample and should therefore be over-represented in the set.

The five question types.

A set consisting only of standard questions tests only whether the agent works on a good day. Five types cover what really happens in operation:

• Standard case: a frequent question whose answer clearly stands in one article. Expectation: the answer from that article, with that article as the source.
• Edge case: a question whose answer has to be assembled from two articles or stands in a subordinate clause. Expectation: the assembled answer, with both sources.
• Question with no answer in the knowledge base: a question for which there deliberately is no article. Expectation: handover to a human, no invented answer.
• Outdated information: a question about something that has changed, such as a price or a deadline. Expectation: the current state, not the old one, even if an old article still exists.
• Question with a false premise: a question that assumes something untrue, such as asking about a feature that does not exist. Expectation: the agent contradicts the premise instead of confirming it.

What counts as passed

The assessment must be fixed before the first run, otherwise it will be adapted to the answers afterwards. Three rules are enough, and they are meant strictly.

First: an answer has passed only if it is factually correct AND names the expected source. A correct answer with a wrong or missing source has not passed. The reason is practical: an answer without a reliable source cannot be checked by anyone on the team, and the agent has in that case, if in doubt, assembled it from the wording of another passage. On the next run the same question can turn out differently.

Second: an honest handover to a human is the expected result for the 'no answer' and 'false premise' types and therefore counts as passed. For the other three types it has not passed, because there the answer was in the knowledge base. A handover is not a free pass; it is only right where the knowledge is missing.

Third: an answer that is correct but claims more than the source supports has not passed. This is the case that slips through most often, because the answer sounds good. An agent that correctly states the notice period and then invents a goodwill rule on top has not passed the question.

How big the set has to be

There is no number that applies to every team, but there is a rule of thumb that follows from the structure of the knowledge base: at least one question per article the agent frequently cites, plus one question each of types three to five for every topic area. For a Help Centre with forty articles in five areas that is about 55 questions. That sounds like a lot, but it is collected in an afternoon and run through in half an hour.

Anthropic recommends for evaluations putting quantity above individual quality: more questions with a somewhat coarser automatic assessment are better than a few, laboriously hand-assessed ones ². Translated for a support team without its own test automation: better 55 questions with a yes-no assessment according to the three rules above than 15 questions on a scale of one to ten. The scale generates discussions, the yes-no generates a work list.

When the set runs.

Before every release of a change to the knowledge base that is more than a typo. New article, deleted article, changed number, changed structure, new source in the crawl: every time. This is only sustainable if the run is short, and it is short when the questions are ready in a table and only need to be asked. A set that runs only monthly is not a regression check but an inventory.

Maintaining the set

A test set goes out of date the moment the knowledge base changes, and that is intended: every change pulls a change to the set behind it. Three rules keep it usable.

• Every error that made it to the customer becomes a question. If the sample or a complaint finds a wrong answer, the question goes into the set with the correct expectation. It stays there permanently, because errors that have occurred once occur again.
• A changed piece of information changes the expectation, not the question. If a price rises, the price question stays in the set and gets a new expected answer. The old answer becomes the expectation for 'not passed', which is exactly the case the 'outdated information' type checks.
• The set never shrinks. A question is removed only when the topic area it belongs to disappears from the product. A question that has passed ten times in a row is not ballast but the proof that one spot is stable.

If you keep the set in a spreadsheet, you need six columns: the question, the type, the expected answer in one sentence, the expected source, the result of the last run and the date. No more tooling is needed, and looking for a dedicated tool is the most common reason the set never comes into being.

What shadow mode contributes

The test set checks the knowledge base before the release. What it does not check is the variety of real phrasings: customers ask the same question in a hundred ways, with typos, half sentences and context from a previous order. That is what the second measurement is for, and it takes place in operation.

At Comlayer it is called shadow mode. In this setting the agent does not answer itself but places a draft with sources into the inbox for every incoming question. The team sends the draft unchanged, edits it or discards it, and the share of drafts sent unchanged shows how well the agent copes with real phrasings. A draft for which the agent would have handed over is marked as such. In doing so, the agent answers exclusively from the workspace's knowledge, that is from Help Centre articles, uploaded documents and crawled pages of your own website, names the source for every answer and hands over to the team when the knowledge does not support an answer.

The two measurements complement each other but do not replace each other. Shadow mode shows whether the agent is ready to talk to real people. The test set shows whether the knowledge base still says what it should after the last change. Whoever has only the one notices regressions only in the pile of drafts, and whoever has only the other never learns how customers really ask.

Frequently asked questions