RAG Readiness Assessment

A RAG readiness assessment establishes whether a defined body of information can support reliable retrieval and appropriately bounded answers for particular users. It examines source authority, extraction, permissions, retrieval evidence, answerability and ongoing change. Successfully placing files in a vector database demonstrates a technical step. It does not show that the right source can be found, interpreted and used at the time a business question is asked.

For a Saudi organisation preparing an Arabic or bilingual knowledge assistant, the practical decision is usually whether the proposed corpus is ready for a controlled pilot, which questions that pilot may handle and what remains excluded. This guide focuses on that corpus-and-retrieval decision within an enterprise data and AI readiness assessment. Its worked example concerns a hypothetical commercial caterer’s menu specification library. It supports staff finding approved information; it does not authorise dietary or allergen-safety decisions.

Define the answer boundary before inspecting the files

Write the user task as a question with an audience, an effective date and an expected source. “Help with menus” is too broad. “Find the approved ingredient specification for a named menu item and service date, showing its source version for an authorised planning colleague” is assessable. It makes clear that a draft recipe, a supplier marketing sheet and an approved specification play different roles even when their text is similar.

List the answers the assistant must not infer. In the catering example, absence of an allergen statement is not evidence that an ingredient is absent, and a historical specification does not approve the current batch. A question requiring information outside the approved corpus should produce an explicit limitation and the appropriate human route. This response is part of successful operation. It is not automatically a retrieval failure.

The original RAG research combines a language model with retrieved external information. It does not establish that a particular enterprise’s documents are authoritative or fit for its intended task. The assessment therefore begins with the information contract: which claims the system may support, which evidence must accompany them and which unresolved conditions prevent an answer.

Build a corpus admission record

Inventory meaningful source families rather than counting every file as equally useful. For each family, identify the accountable owner, the permitted use, the approved version mechanism, its update route and the users allowed to retrieve it. Record whether source attachments and tables are integral to the meaning. A document count can grow while the amount of usable information stays unchanged because most additions are duplicate exports or superseded copies.

Illustrative admission record for a menu specification corpus

Source family	Potential use	Evidence required before admission	Unresolved condition
Approved menu specifications	Identify stated ingredients and approved descriptive wording	Owner, item identifier, version and effective interval	Two active versions for the same item need adjudication
Supplier specification sheets	Provide attributable supporting information	Supplier item match and applicability to the supplied product	A supplier sheet cannot silently override the caterer’s approval
Draft recipe notes	Potential internal research in a separate workflow	Explicit draft status and a separately authorised audience	Excluded from approved-answer retrieval
Retired menu packs	Answer explicitly historical questions	Retirement marker and retained effective dates	Must not be treated as a current recommendation

Resolve authority disputes with the information owner before experimenting with retrieval ranking. Boosting the newest file is not a substitute for knowing which file governs. A recently uploaded scan may contain an older version. The source authority and meaning assessment develops the rules for identifying authoritative records; here the readiness evidence is that those rules can be applied consistently to the proposed corpus.

Inspect what the ingestion process actually preserves

Compare source pages with extracted text and metadata. In a specification table, a row label, ingredient description, quantity unit and qualifying footnote may need to remain connected. An apparently clean paragraph can be misleading if extraction has dropped “not,” moved a table cell into the next row or detached an exception from the statement it qualifies. Review difficult formats deliberately, while keeping them distinct from a representative quality sample.

Ask how chunks inherit source identifiers, version, effective interval and access labels. A passage without its document identity may still be semantically similar to a query, but the application cannot reliably establish its authority or provide an inspectable citation. If a table is split, preserve sufficient surrounding information to identify the row and its header. Measure the effect of alternative chunk boundaries on actual questions instead of selecting a universal chunk size.

Retain a reversible route back to the original source. Reviewers should be able to inspect the cited page or approved record without searching manually for a similarly named file. Extraction failures belong in a visible exception queue, with a disposition such as rescanning, manual transcription, exclusion or restricted use. For scanned Arabic material, the Arabic OCR and document quality assessment supplies a separate examination of recognition and layout errors before they become retrieval defects.

Separate coverage, retrieval and answer construction

A failed answer can arise because the corpus contains no approved answer, because the answer was lost during ingestion, because retrieval did not return it, or because generation misused the retrieved evidence. Treating these as one accuracy number hides the action required. Save the approved source, indexed representation, ranked passages and generated answer for a controlled test so the reviewer can locate the first unsupported transition.

Use a query register with expected evidence rather than a collection of preferred model responses. Each entry should state the user question, permitted identity, date context, relevant source passages and the minimum facts required. Where several documents are necessary, identify the evidence set. A single relevant paragraph may be insufficient when the answer also depends on a version exception or a separate qualification.

Microsoft’s RAG evaluation guidance distinguishes retrieval assessment from answer qualities such as grounding and completeness. For this assessment, the organisation should also define what makes the source authoritative and permitted. A response can accurately repeat retrieved text while relying on an outdated document. That is a business failure even when a context-consistency measure looks strong.

Worked diagnostic: a catering library with 60 test questions

Consider an illustrative challenge set of 60 questions selected to expose known issues: 36 English and 24 Arabic. The set deliberately includes historical versions, ambiguous item names and missing specifications. It is not a random sample of staff demand and cannot estimate the percentage of all future questions that will succeed. The following counts are designed to show how to make an assessment decision, not to report a client result.

Illustrative question disposition before and after retrieval inspection

Observation	Questions	Meaning for the next step
Approved answer exists and required evidence appears in the first five retrieved passages	39	Eligible for separate answer-construction evaluation
Approved answer exists but required evidence is missing from those passages	9	Inspect ingestion, filters, query interpretation and ranking
No approved answer exists in the admitted corpus	8	Test limitation handling; do not count as ordinary retrieval misses
Source owners have not resolved conflicting authority	4	Keep outside the answerable set until adjudicated

The answerable set currently contains 48 questions: 39 plus nine. Evidence coverage within the specified retrieval window is therefore 39 divided by 48, or 81.25%. It is not 39 divided by 60 as a general retrieval-success estimate, because 12 questions lack a settled approved answer. Report all four categories so exclusions cannot make the system appear stronger than it is. The chosen first-five window is an evaluation design, not an industry requirement.

Suppose 33 of the 39 evidence-covered questions then produce acceptable answers, while six omit a necessary qualification. That identifies an answer-construction problem on those six cases. It does not erase the nine retrieval misses. Record Arabic and English outcomes separately before deciding whether either language is ready; an aggregate count can conceal a weak Arabic workflow. The Arabic retrieval testing guide explains how to design those language-specific judgments.

Require evidence for no-answer and conflict cases

A good limitation response distinguishes missing information from conflicting information and restricted information. “I could not find an approved current specification for this item” communicates a different issue from “two approved-looking records disagree and the owner has not resolved them.” Neither should be converted into a plausible recommendation. Design the response so the user knows what evidence or human decision would be needed next.

Test whether asking the same question more forcefully changes the boundary. An assistant should not become more certain simply because the user requests a short answer or says that a deadline is urgent. Include paraphrases that omit an item identifier or confuse two similarly named menus. The appropriate behaviour may be to request the missing identifier, rather than choosing the most frequently retrieved item.

Do not reveal the existence or contents of restricted records when explaining a limitation to an unauthorised user. The safe wording depends on the application’s permitted disclosure rules. Evaluate with authorised test identities and synthetic documents carrying known access boundaries. Keep retrieval permission tests separate from questions of whether an employee ought to have the underlying access. That broader entitlement decision belongs with the information owner.

Test updates, corrections and removals

Readiness needs a change test, not only a static snapshot. Introduce an approved synthetic specification revision, change its effective interval, retire its predecessor and observe when the retrieval system reflects each event. Check both stored passages and any answer cache. A connector that synchronises new files correctly may still mishandle deletion, metadata changes or changes in user entitlement.

Choose a business freshness requirement before choosing a technical schedule. If planners need the latest approved specification at the start of each service window, record when approvals occur and when the system must stop presenting the superseded version. Where a refresh is delayed, define whether the assistant may answer with a dated limitation or must pause that source family. The acceptable choice depends on the consequence of stale information.

Preserve a versioned evaluation pack so the reviewer knows which corpus snapshot was tested. A result from yesterday’s index does not automatically apply after a major source replacement. Re-run the cases affected by the change and a small regression set covering adjacent questions. Explain the selection of that set; repeatedly testing only the question that failed can miss damage to another item or language.

Use experiments to isolate the limiting factor

When evidence is missing, change one meaningful factor at a time. Compare retrieval over verified extracted text with retrieval over the original ingestion output. Compare a query containing an exact item identifier with the user’s ordinary wording. Compare permitted metadata filters with an intentionally broader diagnostic search in an authorised test environment. These comparisons help locate whether the defect concerns representation, terminology, filtering or ranking.

Keep the corpus snapshot and evaluation questions stable during a comparison. If the source set, embedding model and relevance labels all change at once, a higher score cannot show which change helped. Record the candidate configuration and use a held-back group of questions for confirmation where feasible. A small challenge set can support debugging, but repeated tuning against it creates a risk of improving only those familiar examples.

Avoid prescribing fine-tuning, a larger model or a new vector store before identifying the failure stage. None supplies an approved source that does not exist. If staff cannot find a document even with direct navigation, the first action may be catalogue or ownership work. If direct source lookup succeeds but the ranked passages fail, use the enterprise search retrieval diagnostic to narrow the retrieval problem before making a platform commitment.

Protect the evaluation from selection bias and answer leakage

Keep two clearly named collections of questions. An operational sample is intended to describe the actual work population and therefore needs an explicit selection mechanism, such as selecting eligible requests across defined teams and periods. A challenge set deliberately overrepresents difficult cases. Both are useful, but combining their results into a single percentage without a weighting rationale obscures what the percentage means. Record repeated questions and linked questions so several paraphrases of one specification do not appear to be independent business cases.

Have a source owner establish the expected evidence before reviewing the model’s answer where practicable. Otherwise a persuasive response can influence the relevance judgment. If reviewers disagree, preserve their reasons and resolve whether the disagreement concerns source authority, interpretation or the acceptance rule. Do not silently change the expected answer to match a convenient output. Keep evaluation-only questions and annotations out of the assistant’s source corpus, because accidental ingestion would make the test easier without demonstrating general retrieval capability.

Treat retrieved instructions as source content

A document can contain instructions directed at its human reader, accidental text copied from a prompt, or malicious content attempting to redirect the assistant. Corpus admission should therefore record who may contribute material and how unusual content is handled. In the catering example, a supplier attachment telling an assistant to ignore approval dates is not a valid instruction to change the organisation’s source policy. It remains untrusted content from that attachment, even if it ranks highly.

Test this boundary with synthetic material in an authorised environment. The expected outcome is that source text cannot grant access, redefine the task or cause an external action. Application permissions and tool restrictions must enforce the relevant limits; a paragraph in the model instructions is not a universal security control. Record suspicious-source handling and the reviewer’s route for excluding or investigating a document. This keeps the readiness conclusion specific to the tested corpus contribution and retrieval paths rather than assuming that every indexed passage is benign.

Make acceptance conditions usable by source owners

Separate mandatory conditions from graded quality measures. An unresolved authority conflict for a permitted question family may block that family even when most answers are good. A missing citation could be tolerable for low-consequence navigation but unacceptable where staff must inspect the approved specification. State the rule, the consequence and the decision owner instead of presenting one global threshold as universally appropriate.

A practical decision record names the admitted source families, excluded content, tested user roles, languages, corpus snapshot, evaluation population and remaining gaps. It says which work is allowed during the pilot and who reviews uncertain answers. Include the effort needed for source corrections and recurring evaluation. Otherwise the pilot may depend on an analyst quietly repairing documents after every demonstration, an operating model that has never been approved.

Convert each unresolved issue into a task with an owner and observable completion evidence. “Improve knowledge quality” is too vague. “Resolve which of two item specifications is effective for the stated period, update metadata and pass the affected retrieval cases” can be assigned and checked. The readiness remediation backlog provides a useful place to manage such tasks without confusing the assessment with completed implementation.

Prepare a bounded RAG readiness discussion

Bring a short description of the intended user questions, a masked source inventory, examples of difficult documents, the current approval process and known language or access constraints. If a prototype exists, provide its retrieved passages as well as its answers. A polished transcript alone gives the reviewer little ability to separate retrieval quality from model phrasing or hidden manual curation.

Discuss RAG corpus and retrieval readiness with Open for AI to identify what the available evidence supports and what must be resolved before a defined next step. The useful outcome is a defensible boundary for investigation or pilot work, with clear source ownership and targeted follow-up evidence. It is not a promise that every document question will be answerable or that a chosen architecture will eliminate human judgment.