An AI answer is only as useful as its evidence

Start with a question the product can answer

Imagine a support assistant for a fictional software product. Its job is to explain documented behavior, not improvise policy. Start with a small collection of approved documents and a set of representative questions. Write down which questions the assistant should decline before choosing a model.

Retrieve evidence before generating an answer

Retrieval-augmented generation supplies relevant external content as context for generation. Microsoft's overview describes a classic flow in which the application retrieves results and hands them to a language model. Access control belongs in this retrieval path: a user should only receive content they are permitted to see.

question
  → retrieve authorized passages
  → assemble context + source identifiers
  → generate a supported answer
  → display answer + checked references

Design the answer contract

For this hypothetical assistant, return an answer together with identifiers for the passages used. Resolve identifiers against retrieved records before showing links. A citation should lead to evidence the reader can inspect, rather than merely making the response look credible. If the evidence does not support an answer, offer a clear next step, such as opening the documentation or contacting support.

Evaluate the handoff, not just the wording

Review retrieval and answer quality separately. A fluent response can still be irrelevant if the wrong passage entered the context. Keep a small review set with answerable, ambiguous and unsupported questions. Record the source set and configuration alongside each run so that changes can be compared.

A practical review checklist

Before expanding this example into a real feature, check its boundaries:

• Does the retrieved content match the user's permissions?
• Can each cited source be opened and checked?
• What happens when the documents disagree or contain no answer?
• Can the request be cancelled, and can a failure be retried without losing the question?
• Are latency, cost and answer quality acceptable for this particular workflow?