Choose against the real workload
LLM Selection and Integration
Select an LLM using a controlled bilingual workload, deployment constraints and cost per accepted task, then integrate it behind an explicit capability contract.
The best LLM for an integration is the model that satisfies the actual task and operating constraints with an acceptable total burden. A general benchmark ranking cannot establish whether it preserves your identifiers, handles your Arabic documents or fits the receiving application contract. Selection should produce a reproducible decision and an integration plan, not just a preferred model name.
This guide is for application owners and engineering teams choosing a model for a bounded enterprise workflow. It connects evaluation, provider constraints and adapter behaviour within AI integration and delivery. Model availability and commercial terms change, so verify the exact model, access route and terms at the time of selection.
Screen non-negotiable constraints before scoring quality
Establish permitted data handling, deployment options, required languages, input types, response deadline and necessary output capabilities. Identify whether the workflow needs text only, image interpretation, tool calls or constrained structured output. Treat these as actual product and configuration checks; a provider family name does not establish that every model supports the same features.
Check the access route too. A model accessed through a provider API, a cloud platform or a self-hosted deployment can have different operating responsibilities, available options and commercial conditions. Record the exact route evaluated. Do not assume a model name alone determines logging, retention, support access or location of every data flow.
Use data-residency planning to document those flows where they matter. A model that cannot satisfy a binding deployment constraint is not made eligible by a high language score. Conversely, a suitable hosting arrangement does not prove the model can perform the task.
Build a task contract before a model shortlist
The illustrative workload is bilingual product-datasheet intake. The model proposes a product category, preserves the exact model code, extracts stated values with units and identifies missing evidence. It does not recommend electrical installation or approve technical suitability. The receiving catalogue application validates and reviews the proposed record.
Define correctness at field and record level. A correct category with a corrupted model code can still make the record unusable. Preserve the stated unit rather than silently converting it without an agreed rule. Require an unresolved value when the sheet does not contain the requested information, and retain the source location needed for review.
Anthropic's evaluation guidance recommends measurable, task-specific success criteria across quality and operating dimensions. Apply that principle with your own acceptance rules. Do not import a provider example threshold as if it were suitable for your catalogue workflow.
Create a bilingual evaluation population that exposes real errors
As an illustrative design, use 200 cases: 80 Arabic, 60 English, 40 mixed-language or identifier-heavy inputs and 20 unsupported or insufficient-evidence inputs. Those counts define a test plan, not measured market demand. Adjust the composition to permitted production evidence and important rare failures.
Include similar model codes, Arabic and Western digits, conflicting unit labels, multi-column layouts, scanned text and a source with no relevant value. Keep exact-match checks for identifiers and numerical fields, and use a documented rubric for category interpretation and evidence sufficiency. Review ambiguous labels before using them as a scoring standard.
Separate development, validation and final evaluation cases. Keep related variants of the same underlying datasheet together so near-duplicates do not leak into the final set. Give each candidate a comparable configuration-tuning allowance, then freeze prompts and settings before the final comparison. Otherwise one model may appear better simply because it received more iteration.
Score the errors that affect the integration
| Dimension | Measure | Why it matters |
|---|---|---|
| Identifier preservation | Exact model-code correctness under agreed normalisation | Prevents linking a valid extraction to the wrong product |
| Evidence fidelity | Fields supported by the cited source location | Separates plausible completion from documented fact |
| Unresolved handling | Correct disposition for missing or conflicting evidence | Prevents fabricated values entering review |
| Language coverage | Results by Arabic, English and mixed-language cases | Prevents an aggregate score hiding local weaknesses |
| Adapter compatibility | Valid completion, refusal, truncation and tool-output handling | Shows whether the application can interpret real responses |
| Operational fit | Latency distribution, failures, review effort and total task cost | Connects model performance to usable throughput |
Set critical failure rules before testing. An invented safety-related product attribute may warrant a blocking decision even when average extraction quality is good. Distinguish an evaluation failure from a harmless formatting difference. Keep the review rubric close to the business consequence so assessors do not reward eloquence over fidelity.
Use automated grading where the answer is deterministic and qualified review where meaning matters. If a model-based grader assists, calibrate it against human decisions on representative cases and investigate disagreements. Do not let a candidate model effectively approve its own output without an independent check.
Preserve disagreements rather than collapsing them into an early winner. A candidate may perform better on narrative classification while another preserves identifiers more reliably. Record those differences against the minimum acceptable behaviour for each field. If routing different task classes to different models is proposed, evaluate the router and its failure cases as another component. The extra path can be justified, but it introduces configuration, support and release obligations that a single-model score does not show. The final decision should identify the approved workload for each model, not merely announce a winner across incomparable tasks.
Measure latency and cost for an accepted task
Measure the complete path: preprocessing, retrieval if used, inference, validation, retries and review. Report a latency distribution rather than only the fastest response or an average that hides slow cases. Separate time to the first visible token from time to a complete valid result when the application cannot act on partial output.
Use the actual token accounting and current commercial terms for each access route. Arabic and English inputs should be measured rather than costed with an assumed universal words-to-tokens ratio. Record context size, output length, caching conditions and concurrency so the comparison can be reproduced.
Consider this purely illustrative cost calculation, with invented assumptions rather than market prices. At 10,000 records per month, candidate A costs SAR 0.12 in model usage per record and sends 8% to additional review. Candidate B costs SAR 0.07 and sends 14% to additional review. Assume that review costs SAR 4.50 per affected record and both candidates meet the mandatory quality gates.
Candidate A totals SAR 1,200 in usage plus 800 reviews multiplied by SAR 4.50, or SAR 4,800. Candidate B totals SAR 700 plus 1,400 reviews multiplied by SAR 4.50, or SAR 7,000. The cheaper model call is more expensive under these assumptions. This comparison excludes shared infrastructure, ordinary review, implementation and support; include them when they differ between options.
Put a capability adapter between the application and provider
An adapter should translate the application operation into the verified provider request and translate responses into explicit application states. Record capabilities rather than assuming complete interchangeability: supported inputs, output constraints, tool-call structure, refusal representation, stop reasons, streaming behaviour and usage reporting.
Test incomplete and unusual responses. If generation stops because of an output limit, the application should not treat a partial record as complete. If a stream contains fragments of tool arguments, do not execute them before the full validated request is available. If the provider returns a refusal, preserve that state rather than converting it into an empty successful result.
The API-first contract defines the stable business operation. The adapter implements provider-specific mechanics behind it. Keep provider credentials outside prompts and use explicit request limits. Evaluate any abstraction layer against the features you actually need; a common method name does not guarantee identical semantics across providers.
Design fallback as another evaluated path
A backup model may differ in language performance, output shape, refusal behaviour or data handling. It must pass the relevant acceptance conditions before production routing. Do not enable fallback merely because the second endpoint responds successfully.
Define the trigger and consequence: a temporary provider outage may send a read-only draft task to an approved alternative, while a sensitive task may enter a queue for later processing. Record which model produced each proposal. Avoid retrying an entire action workflow when only the interpretation step failed.
Use a controlled release with representative traffic and explicit rollback conditions. Maintain the previous evaluated configuration until the new path has sufficient evidence. The testing strategy and LLMOps guide connect model selection to repeatable release decisions.
Know when model choice is not the main problem
If the required source is never retrieved, a different generator may not solve the failure. If the prompt omits the allowed category definitions, improve the task contract first. If the pattern is stable but repeatedly misunderstood despite good context, investigate the trade-offs in fine-tuning versus RAG using a controlled experiment.
Keep production prompt engineering separate from unsupported claims that a model is universally “best for Arabic”. A selection decision should state the workload, evidence, constraints and review date. It remains valid within that scope rather than becoming a permanent brand preference.
For related commercial context, Trusted Data's Saudi GenAI implementation guide discusses enterprise language and delivery considerations. Discuss LLM selection for an integration with representative inputs, mandatory constraints and the accepted-task outcome that the model must support.
FAQ
Your questions, answered.
Which LLM is best for Arabic enterprise integration?
There is no defensible universal answer without the workload and constraints. Evaluate the exact Arabic, English and mixed-language inputs, identifiers, evidence requirements and deployment route you intend to use. Compare results by case type and operating cost. Record the model version and configuration so the decision can be revisited when the application or available models change.
How many cases should we use to compare LLMs?
Choose enough representative and consequential cases to support the decision, rather than treating one fixed number as a universal minimum. Include common work, important rare errors and cases requiring an unresolved response. Keep a held-out set and document the population. A small curated demo cannot establish broad performance, while a large poorly labelled set can produce misleading precision.
Should we choose the model with the cheapest token price?
Compare the cost of an accepted task, including actual usage, retries, validation, review and any differing infrastructure or support. A lower model price can be offset by more correction work. Use current terms and measured workload characteristics. Keep hypothetical budgeting assumptions clearly separate from vendor quotations and observed production results.
Can one adapter make every model interchangeable?
An adapter can stabilise the business interface, but providers may differ in input support, structured outputs, tool calls, refusals, streaming and operational terms. Maintain a verified capability map and tests for the features used. Do not force unsupported behaviours into a generic success response. A switch is a release that needs acceptance evidence.
Is a backup LLM automatically safe to use during an outage?
No. The alternative must satisfy the relevant task, data-handling and adapter requirements. Define the routing trigger and whether the task should instead wait or use a manual path. Record which model produced the result. A fallback that returns text but mishandles evidence or permissions does not preserve the original application contract.
How do we know whether to change the model or improve retrieval?
Inspect the failure stage. If required evidence never reached the prompt, investigate ingestion and retrieval. If it was present but misinterpreted, compare prompt changes and model candidates on controlled cases. If a stable specialised behaviour remains weak, evaluate tuning as a separate experiment. Change one major factor at a time so the result identifies the cause.