AgentOps for Arabic and English Workflows

Running an AI agent in Arabic and English requires more than translating its interface. The agent must preserve the meaning of a request, retrieve the appropriate evidence, retain identifiers and apply the same authorised action boundary in both languages. Operational quality is established by what happens to the task, including when the correct outcome is to ask for clarification or wait.

For Saudi enterprise teams, the useful question is not whether a model supports Arabic in general. It is whether the specific workflow works for the language varieties, mixed-language inputs and source documents that the organisation actually receives. This guide provides a bilingual evaluation and release protocol for that decision.

Define semantic equivalence at the business boundary

Two responses can use different wording and still produce the same valid business outcome. Conversely, a literal-looking translation can change a condition and lead to a different action. Define equivalence through the task: the same entity, requested change, authority, unresolved conditions and permitted next step.

For an illustrative contact-preference workflow, an English request says to update a preferred channel only after the account owner approves. The Arabic counterpart must preserve that condition. If the agent correctly identifies the new channel but updates it before approval, the language task has failed operationally even if the generated explanation is fluent.

Use the original request as controlled evidence. Any translation used internally should remain linked to it and be identified as a derived representation. Do not let an intermediate English paraphrase silently become the authoritative user instruction for the rest of the workflow.

Build paired cases with deliberate failure conditions

Create paired cases around business meaning rather than mechanically translating a generic benchmark. Have a competent Arabic reviewer and the workflow owner agree the expected action and evidence. Include ordinary cases, ambiguous cases and cases where an action must be withheld. Keep the expected outcome independent of the model being tested.

Illustrative bilingual agent test cases

Risk	Case design	Acceptance rule
Negation and conditions	Pair “Do not send before approval” with “لا ترسل قبل الموافقة”	No send action occurs before the required approval
Pending status	Use “لم يُعتمد الطلب بعد” and “The request has not yet been approved”	The task remains unapproved; wording does not promote the state
Identifiers and digits	Include A-017 and a separate numeric input ٠١٧	Apply the field contract; preserve identifier meaning and leading zeros
Ambiguous dates	Present 03/04 without sufficient calendar or locale context	Clarify the intended date instead of silently choosing an interpretation
Mixed-language text	Embed an English product or field name in an Arabic request	Resolve the intended field without translating a controlled identifier
Conflicting source versions	Provide Arabic and English documents with different revision status	Use the authorised source relationship or escalate the conflict

Keep raw input, the approved interpretation, tool arguments and observed outcome together in the test record. A correct final sentence does not compensate for an incorrect tool call made earlier. Equally, a cautious refusal is not automatically a success if the task was clearly eligible and should have been completed.

Test the full path, including retrieval and tools

Separate language understanding, retrieval, response generation and action execution in diagnosis. An Arabic answer may be wrong because the retriever missed the applicable source, because the source itself is outdated, or because the model changed a condition. Changing the model alone may not address the failure.

Evaluate retrieval with the organisation’s actual document structure. Include headings, tables, scanned or extracted text where relevant, abbreviations and bilingual titles. Record which source passages were available to the agent and whether their authority and version were correctly represented. A relevant passage can still be the wrong one for the user’s role or case.

Use task tracing to preserve the chain from original request to retrieved evidence, proposed arguments, approval and confirmed action. Label any translation or normalisation step so an investigator can locate where meaning changed. Avoid collecting unrestricted content when controlled references and selected evidence are sufficient.

Normalise text without rewriting identity

The Unicode Normalization Forms specification distinguishes canonical and compatibility equivalence and cautions that compatibility transformations can remove meaningful distinctions in some contexts. Text normalisation is therefore a deliberate processing choice, not a universal rule that every similar-looking string represents the same business entity.

Define handling separately for free text, search fields, names, dates, amounts and identifiers. A search index may benefit from approved alternate forms while the displayed source and controlled key remain unchanged. A name-similarity match should not independently authorise merging two people or organisations.

For numeric fields, define accepted digit sets, separators, units and leading-zero rules, then parse into a typed value with an explicit validation result. For identifiers, preserve the documented representation or use a controlled mapping. Do not remove punctuation or translate digits across every field merely because doing so improves a retrieval score.

Dates need a calendar, format and relevant timezone where time matters. Ask for clarification when the source is ambiguous. Keep the original expression for review and show the resolved value before a consequential action where the workflow requires confirmation. Silent calendar or date-order assumptions can survive perfectly fluent translation.

Separate language cohorts in evaluation

Define cohorts that reflect observed use: formal Arabic, the dialect expressions actually encountered, English, mixed-language requests and source-language combinations. Do not claim coverage of every Saudi dialect from a small translated set. Record how the cases were obtained and which varieties remain untested.

Track acceptance, harmful errors, deferrals, unnecessary interventions and unresolved outcomes by cohort. Include sufficient detail about sample sizes and consequence categories to interpret the results. An aggregate can look strong while one language has a much higher rate of unresolved requests.

The original ArabicMMLU research evaluates Arabic multitask language understanding using multiple-choice questions. That is useful benchmark context, but it does not establish performance on an organisation’s permissions, tool execution or current source records. Select production agents using workflow evidence alongside any public benchmark.

Use model judges with bilingual human calibration

A judge model can help classify or prioritise review, but its agreement with an answer is not independent proof of correctness. A judge may prefer a fluent paraphrase that lost a condition, or apply different standards across languages. Establish a rubric that names the business distinctions and allows uncertainty.

Calibrate the judge against bilingual human decisions on representative and adversarial cases. Inspect disagreements by failure category rather than using one agreement percentage as permission to remove review. Include answers with correct style but wrong authority, identifiers or status, and answers that are valid despite different wording.

The LLM-as-judge guide develops calibration and abstention. Use the judge’s output as evidence within that process, with defined escalation for ambiguity. It should not be the only mechanism deciding whether an externally consequential action was authorised.

Make language expansion a controlled release

Adding Arabic to an English workflow changes the operating population. So can adding speech transcription, a new dialect-heavy channel or English documents to an Arabic knowledge base. Treat each material expansion as a release with defined evidence, owners and limits.

Use release control to record which language and source combinations were accepted. Start with the supported population and a working human route for the remainder. Do not let a language-detection label automatically grant access to tools or decide which policy applies to a user.

Keep regression cases for previously observed semantic failures. When a prompt, model, retriever or normalisation rule changes, rerun the relevant paired cases. Preserve the original failure and the reason the expected result was accepted; otherwise, a test update can accidentally redefine a defect as correct behaviour.

Design bilingual intervention and incident handling

A review queue needs people who can assess the original request and the relevant evidence. Routing every Arabic exception through an English paraphrase can hide the very issue the reviewer needs to see. The oversight model should state the required language competence and how uncertain translations are handled.

When an incident occurs, identify the affected language cohort, source version and transformation step. A fault in digit parsing may affect both languages; a missed Arabic qualifier may affect only certain request patterns. Contain the relevant path based on evidence rather than disabling or trusting a whole language indiscriminately.

Corrected messages should preserve the actual business state. Rephrasing an apology does not undo an action already executed. Reconcile the authoritative record and let the appropriate owner decide any corrective action. Keep the original and corrected versions available under approved access and retention.

Use a release evidence sheet

For each language expansion, record the workflow, cohorts, source combinations, evaluation cases, human reviewers, unresolved failures and permitted actions. Add the acceptance decision and the route for unsupported inputs. This sheet makes the scope understandable to procurement, operations and governance without reducing quality to a generic Arabic support claim.

A scorecard can show ongoing differences in accepted outcomes and intervention effort, but it should retain the cohort definitions behind the numbers. Revisit the evidence when real use introduces new vocabulary, channels or document types. Bilingual operation is maintained through controlled changes and review, not completed once at translation.

For AgentOps services, bring masked examples in both languages, the intended actions and known ambiguous cases. Discuss Arabic and English agent operations around the semantic distinctions your workflow must preserve and the evidence needed to release them confidently.