Enterprise RAG: how to evaluate retrieval, sources, permissions and citations before a pilot

Choose a specific workflow, its users and the kind of question it should answer. Also record what is outside scope: decisions requiring human judgment, requests without authorized documentation, or actions the pilot must not perform.

For each source, identify its owner, subject-matter authority, authorized audience, version, update status, format and retention policy. Check for duplicates, conflicting documents, tables or attachments that lose context during extraction, and files with no owner. An index may quickly return an outdated or incomplete passage; faster search does not fix the state of the original document.

Define how to identify the current version when documents conflict and how content will be removed or re-indexed. Internal policies, operating instructions and reference materials may require different rules. No single configuration is suitable for every corpus.

Heterogeneous content needs to be extracted and split into passages that preserve meaning. Headings, tables, footnotes, identifiers and hierarchy may matter for accurate retrieval and citations. Chunk size, metadata enrichment, the embedding model and the choice of text, vector or hybrid search should be tested on data representative of the task—not selected only by preference or a single demonstration.

Record ingestion and indexing settings, including exclusions, parser version, chunking strategy, metadata and search parameters. This helps reproduce an evaluation when sources or the index change. Preparation, chunking, enrichment, embeddings, retrieval and evaluation each involve their own decisions.

In a corporate environment, filtering after the model has received the context is too late. Authenticated identity and authorization rules need to reach the document retrieval layer. Restrict results before sending context to the model, on every search and every retrieval step that may occur during a conversation or agent run.

Test with representative identities and groups, including negative cases: someone without access must not receive the content, a citation or a summary derived from a restricted document. Check unit and sensitivity filters when applicable. Record what was accessed and under which service identity, according to the organization’s audit policies.

The architecture may use storage isolation, platform controls, filters or an API layer that enforces the rules. The choice depends on the tenancy model and environment requirements. Authentication, authorization, filtering at retrieval time and isolation tests are distinct controls.

Build a test set with real or representative questions, reference documents and, where possible, human judgments of the expected passages’ relevance. Include direct questions, alternative terms, ambiguous queries, information spread across multiple documents and questions with no answer in the corpus. The set should reflect intended use, not only examples where search already works.

Evaluate retrieval first: do the right documents appear in a useful position? Are there irrelevant passages, gaps or old sources? When relevance labels exist, ranking and coverage measures can help compare configurations. Without them, careful qualitative review can still find failures, but it should not be described as a universal benchmark.

Then assess the final answer across distinct dimensions: relevance to the question, support in retrieved context, completeness against the expected answer and citation fidelity. An answer can be relevant yet contain an unsupported claim; it can also be well-grounded but omit an important condition. Automated metrics and model-based evaluators help organize comparisons, but should be interpreted alongside examples and human review.

Document the questions, corpus version, retrieval configuration and acceptance criteria before comparing changes. Product, data and use-case risk owners should define thresholds. Vendor examples are not automatically suitable targets for an organization.

A citation is useful when it leads to an authorized, identifiable source—such as a document, version and section—and the passage actually supports the associated claim. Test whether the interface preserves the reference to the original document, whether the recipient can open that source and whether the claim remains accurate when quoted out of context.

Check for missing citations, broken links, duplicate sources, changed documents and claims supported only by a related but inconclusive passage. A link does not prove that an answer is correct. When the evidence does not support a claim, the system should make that limitation clear.

Include cases where the answer is not in the sources, documents conflict, permissions prevent access or retrieval provides insufficient context. Assess whether the system signals the limitation, asks for clarification or routes the question to an owner instead of filling the gap with an inference presented as fact.

Test the abstention policy alongside answer usefulness. A system that never takes a risk may fail the pilot’s purpose; one that always answers may hide missing evidence. The balance depends on task criticality and should be approved by process owners.

RAG does not eliminate hallucinations or replace quality, security or review controls. Even with retrieved context, a model may answer incompletely or make unsupported claims.

Retrieval adds components, processing, latency and maintenance to the generation flow. Ingestion, index updates, embeddings, context volume and search strategy also affect behavior and cost. Record these signals during evaluation and compare options against the use case’s objectives and constraints rather than setting generic targets.

A pilot should make clear how source changes will be detected, test questions and examples reviewed, regressions identified and ownership assigned for access or content incidents. Approval of a prototype does not replace the operational planning needed before broader use.

Use these questions to structure the evaluation. They do not guarantee that a solution is ready for production.