Start with real questions
Collect the kinds of questions a business user would ask while completing a task: finding an approval threshold, checking a product exception, or locating a current operating procedure. Preserve enough context to interpret the question without exposing unnecessary personal or confidential information in the released example.
Attach precise supporting evidence
Link each answer to document identifiers, versions, pages, and passages. Some questions require more than one source; record every necessary passage rather than only the easiest citation. Separate a retrieved passage that mentions the topic from evidence that actually supports the expected answer.
Include unanswered and conflicting cases
A retrieval system should encounter questions the corpus cannot resolve. Define no-answer labels and examples where current and retired policies disagree. Explain whether the expected behavior is to flag uncertainty, request clarification, or cite the controlling version. Avoid making annotators resolve business disputes through guesswork.
Evaluate access and recency boundaries
Specify which documents a hypothetical user may see and the point in time at which an answer should be correct. Keep source permissions and allowed uses under project review. Share the question format, evidence requirements, and planned evaluation measures with Fiund before asking for a tailored sourcing discussion. Include a question asking for a policy exception that is not present in the supplied corpus. The reference should explain why the system cannot answer from available evidence. This tests whether retrieval and answer generation can respect the evidence boundary rather than producing a plausible business rule from general background knowledge.
Review checklist
- Write task-based questions
- Record passage and version identifiers
- Include no-answer and conflict examples
- Define access and time boundaries