Arabic Document Retrieval Testing

Arabic document retrieval testing establishes whether a search or RAG system returns the right permitted evidence for the questions Arabic-speaking users actually ask. It evaluates the passages and records before judging the final answer. A fluent Arabic response cannot compensate for a missing exception, an obsolete source or a result that belongs to the wrong audience.

For Saudi organisations, the practical task is to create a local relevance benchmark with clear source authority, natural questions and inspectable judgments. This guide uses a hypothetical exhibition venue’s visitor-information library. Its rules are invented solely to explain the method and must not be treated as guidance for any real venue. The resulting evidence supports a bounded decision within an enterprise data and AI readiness assessment, not a claim that one model is universally best at Arabic.

Define the retrieval task before selecting a metric

Specify who asks, what they need and what evidence would satisfy the request. A visitor seeking a photography rule needs the applicable permission and exception. An employee searching for the owner of that rule needs a different record. A similarity score cannot establish which of these tasks the system has served. State the intended audience, source collection, language, date context and result window.

Decide whether the test evaluates document discovery, passage retrieval or complete evidence assembly. Finding the correct policy file is useful, but a RAG application may still need the exact clause and its qualifying paragraph. Conversely, a passage can look relevant while its parent document is a retired draft. Record the unit being judged and preserve the parent source identity so the evaluation does not reward a fragment stripped of its authority.

Keep answer generation out of the first diagnostic pass. Save the query and ranked results, then ask a reviewer whether the evidence needed for the task is present. Only after that should the generated answer be assessed. The RAG readiness assessment connects these stages; this page develops the Arabic query and relevance judgments that make retrieval evidence trustworthy.

Collect Arabic questions from the intended work

Use authorised and appropriately minimised sources such as staff interviews, approved enquiry examples or controlled task workshops. Preserve how people describe the need before rewriting it into formal terminology. A visitor may ask هل أقدر أصور كل المعروضات؟ while the policy uses formal wording about photography restrictions. The benchmark should test that genuine language difference rather than making every query repeat the document title.

Include Modern Standard Arabic, relevant local wording and mixed-language terms where they occur in the actual workflow. Do not assume a single Saudi dialect or invent colloquialisms to make the set appear local. Ask native users to validate the wording and intent. Record whether an example came from observed work, a controlled paraphrase or a deliberately constructed challenge.

The original MIRACL research uses native-speaker relevance judgments for multilingual retrieval over Wikipedia. Its methodology supports the importance of language-informed judgment, but its scores do not establish enterprise readiness. A venue library, with effective dates and access boundaries, needs its own queries and evidence labels.

Build a query register with expected evidence

Illustrative Arabic query register for a fictional visitor-information library

Query	Intent to preserve	Evidence requirement	Trap to test
هل أقدر أصور كل المعروضات؟	Whether photography is allowed everywhere	General permission plus the excluded gallery clause	Returning permission without its exception
هل يسمح بالفلاش في القاعة العامة؟	Whether flash is permitted in the named hall	The applicable no-flash wording	Matching the word photography while losing negation
ما شروط زيارة المجموعة المدرسية؟	Requirements for a school group	The approved group-visit rule for the intended date	Substituting rules for private corporate events
أحتاج سياسة الزيارة التي كانت نافذة في سبتمبر	Historical policy lookup	The version effective during the specified September	Always returning the newest upload
هل يشمل الحجز ورشة الأطفال؟	Whether a particular booking includes a workshop	Clarification of booking type before evidence selection	Guessing an entitlement from a general brochure
أين أجد لائحة الزوار رقم V-17؟	Exact document identifier lookup	The authorised document with the exact identifier	Replacing an exact match with a similar document

Each test entry should also carry a stable identifier, expected source identifiers, an allowed user role, a corpus version and an answerability label. Avoid storing unnecessary personal details from real enquiries. If the question depends on a date or booking class, include that context explicitly or label the missing information as requiring clarification. An underspecified question should not be scored against a hidden assumption known only to the evaluator.

Do not equate an acceptable answer with one preferred wording. The expected evidence can support several valid phrasings. Relevance labels should identify the facts and qualifications needed, not reward a model for copying the evaluator’s sentence. Keep the reference explanation short enough to inspect and specific enough that another reviewer can understand why a passage is sufficient or insufficient.

Use native judgments with a visible rubric

A practical rubric can distinguish sufficient evidence, useful but incomplete context and irrelevant material. For this illustrative assessment, a grade of two means the passage provides the required fact under the correct scope; one means it concerns the topic but does not complete the evidence; zero means it does not support the question. These are proposed internal labels, not a universal Arabic retrieval standard.

Apply authority, time and permission conditions explicitly. A superseded rule should not receive full credit for a current-policy question merely because it is semantically close. A restricted source should not become acceptable evidence for an unauthorised identity. Where several passages are jointly necessary, grade the evidence set as well as individual passages. This prevents a partial result from appearing complete simply because one relevant paragraph ranked highly.

Have a second reviewer inspect consequential or disputed judgments. Record whether disagreement concerns Arabic meaning, source precedence, missing context or the rubric. Resolve the underlying issue and retain a decision note. Do not resolve every disagreement by averaging scores: if one reviewer identifies an omitted prohibition, that may be a substantive evidence defect requiring adjudication by the information owner.

Test negation and exceptions with paired questions

In the fictional library, suppose the current rule permits photography without flash in the general hall and prohibits photography in the loaned-collection gallery. The broad question “Can I photograph every exhibit?” requires both clauses. Retrieving only the general permission is incomplete, even if the passage uses nearly every word in the question. The final answer should not generalise a limited permission to the whole venue.

Create controlled query pairs that differ in one meaningful element: with flash versus without flash, general hall versus loaned gallery, individual visitor versus school group. Preserve natural Arabic wording. The aim is to see whether the system responds to the distinction, not merely whether it returns familiar passages. Keep the source set fixed while comparing the pair.

Also create source pairs that share topic vocabulary but differ in scope. A private-event guide may permit arrangements unavailable during public visits. Such a document is a useful difficult negative when it is otherwise allowed in the diagnostic corpus. Explain why it is not sufficient for the visitor’s question. Similarity is a retrieval signal; source applicability remains an evaluated condition.

Keep Arabic normalization observable

Document how the search pipeline handles diacritics, elongation, letter variants, attached particles and mixed numerals. Evaluate transformations against actual queries and identifiers rather than assuming that more normalization always improves relevance. Overly aggressive processing can collapse distinctions the organisation needs, while insufficient processing can miss ordinary variants. Preserve the original query and the form actually searched for diagnostic review.

In Azure AI Search, the official language-analyzer documentation lists Arabic analyzers and an analysis interface for inspecting tokens. Such inspection can help explain lexical matching. It does not by itself establish vector-retrieval quality or prove that a hybrid configuration handles Arabic correctly. Test the configured search stages and their outputs rather than treating an analyzer name as a readiness badge.

Keep exact identifiers and numeric conditions under suitable field rules. The document identifier V-17 should not disappear into a broad semantic interpretation of “visitor rules.” If a query contains a date or age-related condition in a fictional policy, verify its representation and matching explicitly. The test should expose whether the system preserves the constraint or retrieves a generally relevant but inapplicable passage.

Evaluate Arabic-to-English retrieval as a separate task

A bilingual enterprise may store the authoritative answer in English while users ask in Arabic. That is a cross-language retrieval task, distinct from Arabic queries over Arabic documents. Label the source language and query language for each test. Otherwise a strong result in one direction can conceal a weak result in the other, particularly where translations differ in coverage or publication date.

Do not assume that the two language versions are interchangeable. If an Arabic visitor guide and an English operational policy disagree, first establish which source governs the intended user question. The bilingual document authority assessment addresses that decision. Retrieval tuning should not decide authority by selecting whichever version produces a higher similarity score.

When translation is part of the query pipeline, preserve the original and translated query in controlled diagnostic evidence. Have a qualified reviewer check whether a prohibition, condition or entity reference changed. Compare retrieval with and without that translation step on a fixed set. This isolates whether the problem lies in translation, search representation or source availability rather than attributing every miss to Arabic capability.

Worked benchmark: distinguish evidence recall from complete coverage

Consider an illustrative 40-question Arabic challenge set. Thirty questions have settled approved answers, six have no answer in the admitted library and four require clarification before the correct source can be selected. Of the 30 answerable questions, 18 require one source passage and 12 require two. The expected evidence count is therefore 18 plus 24, or 42 required passages.

Illustrative first-five retrieval results on the answerable subset

Question group	Questions	Required passages	Required passages found	Questions with complete evidence
One-passage questions	18	18	14	14
Two-passage questions	12	24	22	11
Total answerable	30	42	36	25

Under this constructed result, required-passage coverage is 36 divided by 42, approximately 85.71%. Complete-evidence question coverage is 25 divided by 30, approximately 83.33%. The two-passage row assumes that one question misses both required passages. These measures answer different questions and should be labelled accordingly. The first-five window is a chosen test setting, not a universal acceptance rule.

Keep the six unanswered and four ambiguous questions in the overall report. Evaluate whether the system returns an appropriate limitation or requests the missing context. Do not remove them silently to advertise a better result. Because the 40 cases were deliberately constructed, these percentages describe the benchmark; they do not estimate the share of all real visitor enquiries that the system will satisfy.

Use ranking metrics without hiding the business condition

Ranking measures can distinguish a relevant passage at position one from the same passage buried lower in the result list. If using a graded metric such as normalized discounted cumulative gain, declare the relevance scale, result depth and treatment of unjudged results. If using recall, establish the expected relevant set and acknowledge where judgments are incomplete. A metric name alone does not make two experiments comparable.

Report a simple operational measure alongside the ranking metric: can a reviewer obtain all required permitted evidence within the result window used by the application? This keeps a ranking improvement connected to the task. A system may move a general overview upward while still missing the exception that the answer needs. That may improve some aggregate measures without resolving the readiness issue.

Inspect query-level differences between configurations. List gains, regressions and unchanged cases, with an explanation for consequential changes. Do not select a winner solely by a small average increase on a limited set. If the apparent gain depends on one query family or several near-duplicate paraphrases, state that concentration before recommending broader use.

Do not confuse an unjudged passage with an irrelevant one

A new retrieval configuration may find a valid source that the original reviewers never considered. Preserve an unjudged state until the passage is assessed under the same rubric. Automatically assigning it zero can favour the older configuration whose results formed the initial reference pool. Conversely, accepting every new candidate as relevant would reward broad retrieval without evidence. Expand the judgment pool deliberately and version the resulting labels.

For a small enterprise corpus, ask the owner whether all authoritative sources for a question can be enumerated. If they can, document that evidence set. If they cannot, state the limits of the known judgments and avoid claiming exhaustive recall. Inspect a balanced sample of new candidates from competing configurations, with the system identity hidden from reviewers where practical, so judgments focus on source applicability rather than a preferred vendor or model.

Protect the benchmark from memorised questions

Separate development questions from a held-back confirmation set where the available material permits. Group paraphrases of the same underlying question together so a nearly identical wording does not appear on both sides unnoticed. Keep source versions and task boundaries consistent across the comparison, while recording any intentionally different temporal or audience conditions.

Do not manufacture a large benchmark by generating hundreds of superficial paraphrases of a few source clauses. That inflates the count without widening the evidence. Cover distinct intents, source families, constraints and failure mechanisms. Native reviewers should examine whether generated candidates sound natural and preserve the intended meaning before they become test cases.

Document test-set exposure. If developers repeatedly inspect and tune against every question, call the set a development benchmark rather than independent confirmation. It remains useful for regression checks, but the readiness decision should acknowledge its limits. New questions from authorised real work can later provide additional evidence under the same source and judgment rules.

Trace failures to the earliest missing evidence

For a failed query, first verify that an approved answer exists for the intended date and user. Next inspect whether the source was extracted correctly and indexed with its necessary metadata. Then inspect filtering, query transformation, candidate retrieval and ranking. Only after those stages should the answer generator be blamed for not using evidence. Preserve the intermediate results needed for that sequence.

If the Arabic text is missing a negation or table relationship before indexing, return to the Arabic OCR quality assessment. If the text is correct but the wrong passages rank, use the enterprise search diagnostic to isolate the search configuration. If two sources conflict, resolve authority. Assigning the right owner is more useful than labelling every issue “poor Arabic AI.”

Test a proposed repair against the affected family and relevant neighbouring cases. A synonym that improves one colloquial query may broaden another query incorrectly. Record the tradeoff and preserve the previous configuration for comparison. The assessment needs evidence that a repair solves the identified problem without creating an unacceptable new one within the proposed pilot scope.

Publish a decision pack that a native reviewer can inspect

The pack should contain the corpus snapshot, query register, source-language directions, relevance rubric, adjudications, configuration record and per-query results. Show unresolved authority and permission questions separately from retrieval misses. Include examples of correct limitation handling and clarification. These are important parts of the permitted experience, especially when users ask broad questions with missing context.

State which Arabic query families can enter a controlled pilot and which remain excluded or require human review. Proposed acceptance thresholds should reflect the task’s consequences and available evidence. They are internal design decisions, not a claim that Arabic retrieval has one universal passing score. Use the readiness decision pack to connect the technical findings with the sponsor’s actual next-step decision.

Discuss Arabic document retrieval testing with Open for AI using masked source examples, naturally written user questions and the current retrieval outputs if available. The assessment can build a reviewable benchmark and identify the specific barriers to useful Arabic evidence retrieval. It should produce a bounded, testable path forward rather than a ranking claim based on a fluent demonstration.