Arabic AI Quality Evaluation

Evaluate Arabic AI by checking whether it performs the intended task correctly, preserves the source meaning and handles uncertainty appropriately. Fluency is one dimension of quality. A polished response that reverses a negation, changes a reference number or turns a possibility into a promise can fail the business task even when its grammar is excellent.

This guide is for Saudi product owners, Arabic reviewers and business teams accepting generated or classified Arabic outputs. It focuses on meaning and task performance rather than optical character recognition or search retrieval. The examples and scoring exercise are original, fictional evaluation designs; they are not measured results for any model or a claim about national language accuracy.

Define the output and the consequence of being wrong

Start with the action the output supports. A service-message classifier may identify a request for a human queue. A drafting assistant may prepare a reply from approved facts. A summariser may distinguish an agreed action from a proposal. Each task needs different acceptance evidence. Do not use a general Arabic writing score as a substitute for the actual result.

Specify what the system is allowed to infer. If a customer names two order identifiers, the evaluator should know which one belongs to the requested change. If the message lacks a date, the expected handling may be clarification rather than choosing a plausible date. Define the permitted uncertainty response before looking at the model output so confident guessing is not rewarded after the fact.

Record the consequence of an error. A stylistic preference might require a small edit; an incorrect cancellation intent can misroute work or lead a human reviewer toward the wrong decision. The task in this guide classifies or drafts for review and does not execute cancellations. Keep that boundary explicit so the acceptance method matches the actual authority of the system.

Separate meaning, task quality and language presentation

Dimensions for a practical Arabic evaluation rubric

Dimension	Question	Example failure
Meaning	Are intent, negation and qualifications preserved?	A request not to cancel becomes a cancellation request
Facts and identifiers	Are supplied values retained without invention?	AB-109 becomes AB-190
Task completion	Does the output contain the required fields or answer?	A summary omits the requested next step
Uncertainty	Does it distinguish known, conditional and missing information?	A possible date change becomes a confirmed change
Register and clarity	Is the wording appropriate and understandable?	An internal note is rewritten as an unsupported formal commitment
Operational usability	Can the reviewer use the output in its destination?	Mixed-direction text obscures the relevant reference

Score dimensions separately before deciding whether a case passes. A high style score must not cancel a critical meaning error. Also avoid penalising every alternative wording: several Arabic expressions can preserve the same approved meaning. The reference should identify required content and prohibited changes, not force exact string matching for an open-ended writing task.

For structured classification, define the permitted labels and field meanings precisely. For generated prose, define material facts, acceptable omissions and required uncertainty. Use Arabic natural language processing task design for structured language-analysis requirements and bilingual workflow design when the Arabic output must stay aligned with an English source.

Use contrast pairs to expose meaning errors

A contrast pair changes a small part of the input while preserving most of its surface form. The expected interpretation must change in a specific way. This makes it harder for an evaluator to accept a system that reacts to a familiar word without understanding its role in the sentence. The pairs below are invented test material and should be adapted to the actual service vocabulary.

Original Arabic challenge cases

Input	Required interpretation
ألغوا الطلب 842، ولا تلغوا الطلب 843.	Cancellation intent applies to 842; preserve the instruction concerning 843
لا تلغوا الطلب 842؛ أريد تعديل موعده.	Rescheduling intent for 842, not cancellation
ما أبي ألغي، أبي أغيّر الوقت.	Preserve the informal request to change time without cancellation
قد يتغير الموعد بعد تأكيد المورد.	The change is conditional and not yet confirmed
أصبح تغيير الموعد مؤكداً بعد رد المورد.	The source states that the change has been confirmed

Keep the expected output tied to the task. A classifier may return a request category and relevant identifier. A reply draft should acknowledge the requested change without claiming it has already been performed. A summary should preserve the distinction between request, approval and execution. These are different outputs even when they share the same input sentence.

Use the contrast pair to diagnose, not merely decorate, the test set. If the model fails both negated and positive versions, the task instructions may be unclear. If it handles the positive version but reverses the negative one, the failure is more specific. Preserve the output and settings so the next change can be evaluated against the same evidence.

Do not confuse emotional tone with urgency

Consider two fictional messages. The first says: لو سمحتم، الطلب المتعلق بفعالية اليوم لم يصل. The second says: تعبت من تكرار الرسائل، لكن الرد الأسبوع القادم مناسب. The first is polite and time-sensitive because it refers to an event today. The second expresses frustration but explicitly accepts a response next week. A useful rubric separates sentiment from the stated timing requirement.

Do not infer urgency solely from exclamation marks, politeness or a negative adjective. Define the evidence that the task should use: a stated deadline, an operational dependency or an explicit service condition. If the message lacks enough information, the correct classification may be needs clarification. The evaluator should not invent an emergency simply to avoid a neutral output.

Likewise, do not infer personal characteristics from a dialect, name or writing style. The evaluation concerns the meaning needed for the task. A model may need to understand informal wording while returning formal Arabic, but that does not justify conclusions about the speaker's nationality, competence or reliability.

A 2024 Association for Computational Linguistics (ACL) study of cultural bias in language models evaluated cultural alignment separately from broad multilingual capability and found problems in the models and tasks it tested. That supports investigating contextual fit; it does not establish the performance of a current product on your workload. Use locally relevant, authorised cases and avoid turning cultural familiarity into stereotypes.

Preserve identifiers, units and qualifications

Use cases where a small factual change matters. For example: الشحنة AB-109 للفرع 07، وليست AB-190. The output should preserve the relevant shipment and branch while retaining the exclusion. Inspect the values after export as well as in the original response. A reviewer should not need to guess whether a reordered mixed-language string names the same record.

Define acceptable normalisation. Converting a displayed digit style may be harmless if the system preserves the underlying value and the destination accepts it. Removing a leading zero from an identifier may not be harmless. The business owner must define which fields are identifiers, numbers, dates or free text so the evaluator does not treat every visual change identically.

Test qualifications with a fictional service statement: الرد المتوقع خلال يومي عمل بعد اكتمال المستندات. A draft that promises a response within two days regardless of completeness changes the condition. A draft that converts expected into guaranteed changes certainty. The evaluation should identify both, even if the final paragraph sounds more persuasive.

Include missing information deliberately. If the source says a date will be confirmed later, a blank or explicit pending status can be correct. An output should not receive a higher completeness score merely because it fills every field with plausible content. Completeness must mean all supported required information, together with appropriate handling of what is unknown.

Construct a case mix that represents use and challenges it

Maintain two complementary views: representative cases for the workload and challenge cases for known failure mechanisms. Representative cases help estimate how the system behaves on ordinary work. Challenge cases test whether it can handle a particular difficult distinction. Do not combine a deliberately difficult set with ordinary traffic and report the result as a natural production error rate without explaining the sampling.

As an illustrative design, an initial eighty-case evaluation could contain forty ordinary cases, twenty meaning-contrast cases, twelve incomplete-information cases and eight unfamiliar-terminology cases. These counts are a planning choice, not a recommended universal sample size. The right evidence depends on the variation and consequences of the intended use. Preserve the category of every case so results remain interpretable.

Use sources that the organisation may legitimately evaluate. Remove or protect unnecessary personal information and preserve enough context to understand the task. Have knowledgeable Arabic reviewers check the reference interpretation before model testing. A mistaken reference can make a correct output appear wrong and encourage an unnecessary change to a working system.

Reserve new cases for assessing a revised configuration. If every failure is added to the instructions and then tested only on the same examples, the team may be measuring familiarity with its test set. Keep a documented distinction between development cases, acceptance cases and later operational observations.

Calibrate reviewers before trusting the score

Give two qualified reviewers a small common set and ask them to score independently using the rubric. Discuss disagreements at the level of meaning and task requirements. Was a different phrasing acceptable? Did one reviewer assume a business rule not present in the source? Is the definition of critical error clear? Amend the rubric when disagreement reveals ambiguity.

Keep both initial judgements and the adjudicated result. Agreement reached after discussion is useful, but it should not be reported as if the reviewers independently agreed from the beginning. For consequential cases, record who resolved the disagreement and the evidence used. A third business owner may be needed when the issue concerns the intended process rather than Arabic wording.

Do not rely on an automated language-model judge alone to settle subtle Arabic meaning or business consequences. It can assist a defined review process, but its decisions require validation against trusted human references for the task. If the judge shares the same blind spot as the system under test, a polished automatic score can conceal the problem.

Report acceptance, correction and rejection separately

For a fictional reporting exercise, suppose the eighty cases produce 58 outputs accepted directly, fourteen requiring a permitted human correction and eight held because they cannot be accepted as produced. The direct acceptance rate is 58 ÷ 80 = 72.5%. Correctable cases are 17.5%, and held cases are 10%. These are invented figures used to explain reporting.

If the fourteen corrections are successfully completed, 72 cases may become usable in the assisted workflow, or 90% of the set. That does not mean the model produced 90% correct outputs initially. Preserve the correction effort and the nature of the changes. A customer-facing draft and a classification system may have different permitted correction arrangements.

Break down the held cases by failure mechanism and consequence. For this exercise, three might reverse negation, two corrupt identifiers and three overstate certainty. Those counts sum to eight, but their implications depend on the task. Do not let a larger number of harmless style passes dilute the visibility of a small set of consequential errors.

Report results by relevant slice as well as overall: task, language variety, message length, identifier presence and uncertainty type. Keep small denominators visible. A perfect result on two examples is not equivalent to a well-supported quality claim across an entire dialect or sector.

Turn failure analysis into a controlled change

Classify the cause before choosing a remedy. The source may be ambiguous, the task may lack a required definition, the prompt may obscure the priority, the model may fail the language distinction or the destination may corrupt the output. Changing the model is only one possible response. Fixing the wrong layer can add cost while leaving the actual failure unchanged.

Use the Arabic tool-selection guide if a product-category limitation is the issue. Use employee training if reviewers cannot identify the relevant error. Use the automation operating model when correction effort makes the process uneconomical. Keep the quality finding linked to its operating consequence so the evaluation informs a decision.

After a change, rerun the affected cases and a broader regression set. Confirm that solving one distinction has not damaged another. A prompt that overemphasises caution could avoid unsupported promises while failing to complete ordinary requests. Record the configuration, evidence and accepted scope for the new release rather than describing it simply as improved Arabic.

Distinguish genuine ambiguity from a model mistake

Some inputs do not support one definite answer. Consider the invented message خلّوه لبكرة، وإذا تأخروا ألغوه without the preceding conversation. The referent, exact timing and condition may be unclear. An evaluator should not invent the missing thread and then mark every different answer wrong. The reference can instead require a clarification or a conditional interpretation that explicitly identifies what is missing.

When context is available, include the authorised context in both the test input and the reference. If tomorrow must become a calendar date, specify the relevant message timestamp and time-zone rule. Otherwise, preserving the relative expression may be more faithful than choosing a date from the evaluator's current clock. The task specification should settle this before scoring.

Allow a set of acceptable answers where appropriate. A concise clarification and a structured missing-information flag may both satisfy the workflow if the downstream reviewer understands them. Define the essential condition they share. This prevents the evaluation from confusing one preferred sentence with the only correct interpretation.

Also distinguish an unclear source from an unfamiliar language variety. A reviewer should not label an informal but unambiguous request defective merely because it is not written in formal Arabic. Ask a qualified reviewer familiar with the actual language use to assess the meaning. Document disagreement instead of silently rewriting the source into an easier test that no longer represents the workload.

The final decision should say whether the problem belongs to the input, the reference, the task definition or the generated output. This separation improves remediation: better intake may resolve missing context, while better model or instruction behaviour may be needed for a clear negation. Preserve both the original case and the corrected reference when an evaluation defect is found.

Approve a bounded language capability

A release record should state the tasks, source conditions, language varieties, output requirements, evaluation set, unresolved limitations and retained review. Define what happens outside the accepted scope. The responsible owner should understand the difference between a draft for a skilled reviewer and an output that can trigger an action; their evidence requirements are different.

Reassess when the model, instructions, product workflow, terminology or input population changes materially. Preserve difficult examples discovered in operation and investigate them without treating every complaint as proof of a general model defect. The Saudi AI authority guides connect language quality to broader adoption decisions. Open for AI solutions can help define the task; discuss an Arabic acceptance set built around the meaning your organisation must preserve.