RAG System Audit

A RAG system audit follows an answer back through retrieval, source selection, permissions and document history. It asks whether the system used the right material for the right person at the right time, and whether the answer actually follows from that material. A plausible paragraph with a working citation can still be wrong: the cited rule may have expired, belong to another tenant or support only part of the answer.

This guide is for enterprise search owners, information security teams and knowledge managers assessing retrieval-augmented generation. The original RAG research combines generation with retrieved external information. That architecture provides an evidence path to inspect; it does not itself establish accuracy or appropriate access. An audit examines the particular implementation and its operating evidence, within an agreed AI auditing engagement.

The illustrative system here answers questions about fit-out rules for commercial properties. Each tenant can access general building guidance and its own approved documents. Some rules have future effective dates, some documents supersede earlier versions, and some exceptions require a property manager's decision. The assistant explains documents; it cannot authorise construction work. Every example below is hypothetical and describes an audit method, not a completed client assessment.

Define the retrieval boundary before measuring answer quality

Document the repositories, ingestion jobs, index collections, retrieval filters, reranking steps and answer-generation components in scope. Include attachments, extracted tables and metadata that may be transformed during ingestion. Identify which system holds the authoritative access decision and which identity reaches each component. A diagram saying “permission-aware search” is not sufficient evidence of where a request is denied.

Record the application release, index snapshot or reproducible build identifier, embedding configuration, chunking rules and source cut-off. A model version alone cannot reproduce a RAG result. If a source changed between retrieval and review, the auditor needs the source version actually supplied to the model. Otherwise, an apparent answer error may instead reflect an incomplete evidence record.

Define what the assistant is allowed to conclude. A building rule may describe a standard approval process without proving that a particular tenant has approval. In the example, the system should distinguish a general requirement, a tenant-specific document and an unresolved exception. Agree these categories with the responsible property team before writing test questions. For conversational state and language switching, combine this work with LLM and chatbot auditing.

Trace six links in the evidence chain

The audit trail should connect a source document to the actual answer. Each link has its own failure modes and evidence requirements. A test that examines only the final text cannot reliably distinguish a missing source from poor retrieval, wrong permissions, incorrect interpretation or citation assembly that points to the wrong version.

Original RAG evidence-chain review for the property-rules example

Link	Audit question	Useful evidence	Conclusion limit
Source authority	Who approved this rule and when does it apply?	Document owner, effective dates and supersession record.	A repository timestamp alone does not prove policy authority.
Ingestion	Did extraction preserve the relevant condition?	Original page, parsed text, table structure and ingestion result.	Successful upload does not prove faithful extraction.
Access	May this identity retrieve this tenant's material?	Identity context, source permissions and enforced retrieval filter.	A hidden citation does not prove the content was withheld.
Retrieval	Were the necessary passages available and selected?	Candidate results, scores where available and final context.	A relevant passage may still omit a decisive exception.
Generation	Does each material claim follow from the supplied context?	Instructions, context and adjudicated answer.	Faithfulness to a stale source is not current correctness.
Presentation	Can the user inspect the source behind the claim?	Rendered citation, destination and applicable source version.	A valid link is not proof that the source supports the claim.

Use the matrix to assign evidence owners rather than asking one engineer for every artefact. The information owner may establish authority; the platform team may establish indexing; the security team may establish permission enforcement. Where evidence cannot be obtained, describe the untested link. Do not convert a smooth demonstration into an end-to-end conclusion.

Test document versions and effective dates explicitly

Version control has at least three distinct dates: when a document was published, when the rule becomes effective and when the index incorporated it. These may differ legitimately. A future rule should not automatically displace the current rule merely because its file is newer. Similarly, an old document may remain authoritative for a historical question if the service explicitly supports historical retrieval.

Build a controlled source pair in the test environment: a current fit-out rule and a superseding rule with a later effective date. Ask the same question for dates on either side of the transition. Record the requested date, retrieved versions, context passed to generation and answer. The expected result should be defined by the approved rule, not by whichever document the search ranking prefers.

Check updates that remove access or withdraw a document. The audit needs evidence that derived copies, embeddings, search snippets and cached responses are handled according to the documented policy. Do not assume removing a file from the repository instantly removes every derivative. If the architecture uses asynchronous propagation, identify the documented window and test its behaviour, including what users see while the update is incomplete.

For time-sensitive claims, distinguish the policy's own freshness requirement from an invented audit threshold. A rule owner might require a withdrawn instruction to stop appearing before a scheduled opening period. The audit can examine that criterion and the observed timing. It should not present an arbitrary number of minutes as a universal requirement for every knowledge service.

Verify permissions at the point where information is selected

Use synthetic tenants with deliberately distinct documents. Give a general user access to shared guidance, a tenant representative access to that tenant's material, and an authorised property manager a separately defined scope. The test question is whether the enforced boundary matches those permissions across direct queries, paraphrases, follow-up questions and cached answers.

Inspect the context supplied to the model, not just the visible response. If an unauthorised passage enters that context, a polite refusal later in the pipeline does not establish that retrieval isolation worked. Also inspect citations and snippets, which may expose titles or facts even when the main answer is withheld. Test permissions after a role change using the supported session behaviour.

Keep tests authorised and bounded. Synthetic canary text can distinguish test documents without using real tenant secrets. Stop and preserve evidence if an unexpected boundary crossing occurs. The security boundary testing guide covers the rules of engagement and the distinction between an attempted request, retrieved content and confirmed disclosure. A RAG quality review should not silently expand into unrestricted security testing.

Measure retrieval and answer support separately

For retrieval, define the evidence needed to answer each question before inspecting results. Some questions require one passage; others require a rule and its exception. The reference set should identify indispensable material and acceptable alternatives. A document-level hit can be misleading if the retrieved chunk excludes the sentence that changes the answer.

For answer support, break the response into material claims and compare each with the supplied context. An answer may be faithful to context but factually outdated, or current but unsupported by the retrieved passages because the model relied on prior knowledge. These are different observations. Report source currency, retrieval completeness and answer support separately instead of collapsing them into a single “RAG accuracy” percentage.

A hypothetical test contains 20 questions with adjudicated reference evidence. Fifteen retrieve all indispensable passages, three retrieve only part and two retrieve none. The complete-evidence rate is 15 divided by 20, or 75%, for this set. If reviewers then assess only the fifteen complete cases, their answer-support result must not be described as performance across all twenty. Keep partial, empty and unanswerable cases visible.

These deliberately selected questions are not a random sample of tenant traffic. Their results demonstrate specific strengths and weaknesses, not a population failure rate. For a production estimate, define the traffic window, selection mechanism and unit of analysis. Related queries from one tenant or one document can be clustered, which affects what an uncertainty calculation can justify. Consult audit sampling design before claiming generalisation.

A worked source-to-answer test

Consider synthetic document FR-12, version 3, which requires noisy work to end at an approved time and refers exceptions to property management. Version 4 changes the standard time from a future effective date. Tenant A also has a private exception document that Tenant B must not access. No actual times or permissions in this example should be treated as building guidance.

Illustrative RAG test protocol with distinct expected outcomes

Case	Setup and question	Expected evidence	Decision if unmet
Current rule	Tenant B asks about a date before version 4 applies.	Version 3 is selected and the answer preserves the exception process.	Investigate date filtering or interpretation; do not assume a model defect.
Future rule	The same identity asks about a date after the change.	Version 4 supports the answer and its effective date is clear.	Check ingestion and version selection separately.
Private exception	Tenant B asks whether another tenant has extended hours.	Unauthorised content is absent from retrieval context, snippets and answer.	Contain the tested exposure and escalate the access-boundary finding.
Missing approval	Tenant A asks whether its proposed work is authorised.	The answer distinguishes document information from an approval decision.	Restrict unsupported authority claims and review the workflow.
Withdrawn source	Withdraw version 3 under the test policy and repeat applicable queries.	Observed derivative and cache behaviour matches the documented withdrawal rule.	Identify propagation gaps and define an accountable correction.

Retain the full evidence bundle for each case: identity, source versions, timestamps, configuration, retrieval results, final context, answer and adjudication. If a provider exposes only the final response, the audit can still test observable behaviour but cannot confidently attribute the cause to retrieval. Record that access limitation rather than filling the missing trace with an assumption.

Examine Arabic retrieval without losing the source meaning

An Arabic query may need an Arabic source, an English source or both, depending on the approved document set. Define which language version is authoritative where the versions differ. A bilingual embedding model does not settle a conflict between two approved-looking documents. The information owner must establish the policy for interpreting such conflicts.

Include Arabic terms used by tenants, English technical labels embedded in Arabic sentences, spelling variants and queries whose negation changes the required action. Review extraction from Arabic tables and scanned documents where these are within scope. A retrieval miss caused by missing OCR text differs from a ranking failure, and the remediation owner may be different.

Ask reviewers to check whether qualifications survive the answer. Translating a condition into a general permission is a material error even if the main topic and citation are correct. Preserve units, dates and references to the approving role. Report language-specific evidence availability and outcomes; a strong English retrieval score does not establish Arabic coverage when the source corpus itself differs.

Review operational changes that can invalidate the audit

RAG behaviour can change without a model replacement. A new parser, chunk size, metadata mapping, reranker, access filter or cache policy can alter what evidence reaches the model. Obtain the change record and test whether the organisation knows which evaluation set should be rerun for each change. A blanket assertion that the model stayed the same is not a sufficient release rationale.

Compare production query patterns with the audit set over the stated period. New properties or document classes may introduce conditions not previously examined. Monitor ingestion failures and stale-index signals as evidence of operation, but test whether those signals themselves are complete. A dashboard showing no ingestion errors is weak evidence if failed jobs never reach the dashboard.

Review fallback behaviour when retrieval is unavailable. The service may refuse, offer a source search or transfer the question. If it falls back to unconstrained generation while retaining the same “based on your documents” presentation, users may misunderstand the evidential basis. The audit should observe and report that transition. A visible degraded mode can be more defensible than a confident answer with no source trail.

Challenge the corpus and the test references independently

A retrieval system cannot select a rule that was never supplied to it. Reconcile the approved source inventory to ingestion records and the searchable collection. Select examples of exclusions as well as successful imports. A skipped attachment, a parsing failure and a deliberately excluded document have different explanations. The report should say whether the corpus was complete for the agreed purpose or whether completeness itself could not be established.

Review duplicated and contradictory material. Several copies of one rule can dominate retrieval without providing independent support. An email attachment may look newer than an approved notice while having no authority to replace it. Ask the information owner to establish the precedence rule and preserve disputed cases. The audit should not quietly select an answer that resolves an organisational disagreement the system was never authorised to settle.

Keep the test references separate from the content used to tune retrieval. If engineers repeatedly adjust ranking against the same questions, the resulting improvement demonstrates performance on those familiar questions. It needs a fresh, independently adjudicated challenge set before it supports a broader claim. The evaluation integrity review can examine who saw the test cases, how reference passages were chosen and whether difficult cases disappeared between reports.

Also inspect the question-to-source relationship. A question assembled by copying an unusual phrase from a source may be much easier than a tenant's ordinary description. Include realistic paraphrases without sacrificing a known reference answer. Where a question is ambiguous, define whether the correct outcome is clarification rather than ranking one interpretation highest. Keep those cases separate from clear questions when interpreting retrieval completeness.

Finally, verify the evidence export itself. A trace may show the final selected passages but omit candidates removed by a filter; another may show candidates without the context ultimately sent to generation. Name precisely what each record contains. Compare linked request identifiers and timestamps, and investigate mismatches. An apparently complete chain assembled from different requests is not reproducible evidence of any one answer.

Turn findings into specific corrective decisions

A finding should identify the broken link and the evidence supporting it. “RAG hallucination” is too broad if the real observation is that an expired document outranked the current one. Conversely, the presence of an old passage does not prove it caused the final error unless the evidence connects the passage to the response. Keep observed condition, likely consequence and causal hypothesis separate.

Choose a corrective action that addresses the mechanism and the operating process. Re-indexing may fix one missing document but leave a recurring ingestion failure untouched. Changing a prompt may improve wording without enforcing tenant isolation. A source owner may need to resolve contradictory documents before any technical change can be meaningfully assessed.

For closure, preserve the original failing case and test related variations, including the permissions and versions that should remain unaffected. Use retesting and closure evidence to distinguish a completed engineering task from a demonstrated correction. State the new configuration and source window. If access to retrieval traces remains restricted, retain that limit even when the visible answer improves.

Include an evidence index that maps each test to the exact source revision and access context. A later screenshot of a corrected document cannot reconstruct what the assistant retrieved during the original run.

Prepare for an audit by collecting a corpus inventory, representative approved sources, a permission matrix, update and withdrawal rules, retrieval traces and known difficult questions. Scope a RAG audit around your sources, permissions and answers to establish which parts of the evidence chain can be examined and what the resulting conclusion will cover.