Inspect the system behind the demo
AI Agent vs LLM Wrapper
A practical way to assess model-backed applications and agents by evidence, identity, state, tool contracts, evaluation and operating ownership.
“LLM wrapper” is an informal term for an application built around a language model service. It is not a reliable quality category. A small application around a model can solve a valuable problem, while a complicated agent can fail its users. The useful distinction is whether the system provides the task behaviour, controls and operating support that the buyer actually needs.
An AI agent typically adds model-directed selection of steps or tools within a bounded task. That capability does not automatically provide accurate evidence, secure access, durable execution or business value. This guide helps a technical buyer inspect the system behind the demonstration without treating complexity as proof of substance.
Ask what the application contributes
Start with the job. An illustrative product documentation service answers employees' questions about approved product specifications. A simple interface that sends a question and a selected document to a model may be enough for a narrow, supervised task. Its value could be convenience, accessible presentation or integration into an existing work surface.
If the requirement expands to selecting the correct document version, respecting role-specific access, comparing several sources and preparing a traceable answer, the surrounding application needs more explicit responsibilities. Those additions may still form a fixed workflow rather than an agent.
Agency becomes relevant when the model must choose how to investigate: which permitted source to inspect next, whether to clarify an ambiguous product identity or when the available evidence is insufficient. The automation comparison explains why multiple model calls alone do not establish useful autonomy.
Do not use the number of components as a quality score
A vector database, orchestration framework and several named agents can appear in a diagram without solving the hard task. Conversely, a concise application can enforce permissions and produce a well-supported answer. Evaluate the contracts and evidence at each boundary instead of rewarding the length of the technology list.
The same applies to model choice. A powerful model cannot supply a missing authoritative product record or determine which employee is authorised to see it. Those facts belong to the application and source systems. The buyer needs to know where each responsibility is implemented.
Anthropic's tool design guidance treats tools as contracts between deterministic software and variable agent behaviour. That is a useful reason to inspect the tool boundary, rather than assuming a model call makes an integration dependable. See the primary tool design discussion.
A capability ladder, without assuming higher is better
| Design | What it can provide | What still needs evidence |
|---|---|---|
| Selected-document answer | Interpret a supplied document for a supervised user | Source support, limitations and acceptable use |
| Grounded retrieval workflow | Find permitted evidence through a defined path | Correct source selection, freshness and access |
| Integrated business application | Manage identity, task state, review and outputs | End-to-end correctness and operating ownership |
| Bounded agent | Choose evidence-gathering or action steps dynamically | Useful choices, stopping behaviour and controlled execution |
These designs can overlap. They are not mandatory stages every organisation must climb. If a selected-document answer satisfies the supported need, adding an autonomous planner may create work without adding value. If the task requires controlled transactions, an attractive answer interface is insufficient.
Worked inspection: a specification question with missing evidence
Suppose an employee asks whether a product meets a requirement stated in a customer's specification. The approved product document describes some relevant properties but does not establish the requested certification. The correct output must preserve that gap. Similar wording or a related product's document is not proof of compliance.
A simple application might answer well when the user provides the correct document and understands the limits. A retrieval workflow must additionally locate the applicable version. A bounded agent may inspect permitted supplementary sources or ask which product variant is intended. Every design should stop short of asserting an unsupported certification.
Ask the supplier to show the evidence path and the unresolved field, then change the source to a superseded version. Next, remove the supporting passage entirely. The proposed inspection tests whether the system's answer depends on actual evidence or merely continues the expected narrative.
Do not mistake this demonstration for a complete evaluation. It is a targeted buying test that exposes a critical responsibility. A broader assessment needs representative cases and failure categories, as described in agent testing and evaluation.
Inspect identity and source selection
Ask how the application knows which records the caller may access. If every user shares one broad repository connection, request evidence that the application still enforces the intended permissions. A login screen alone does not establish that retrieval respects document-level restrictions.
Inspect source metadata. Can the answer identify the relevant product, variant, document version and effective context? What happens when two documents disagree? A source link is useful only if it supports the associated claim and is accessible to the authorised reviewer. A decorative citation can create confidence without traceability.
For Arabic questions over English specifications, examine identifiers, units, negation and exceptions. A fluent translation can obscure a material change in meaning. Language support should be judged on the actual task, with the same source and permission obligations in both languages.
Inspect state and actions
If the product only answers a question, its state requirements may be limited. If it manages a case across sessions or approvals, ask what persists and how it is corrected or deleted. A conversation transcript is not automatically a reliable case record. The memory guide separates these responsibilities.
If the system can act, ask for the exact command contract. Which fields may it change, who authorises the change, what proves completion and what happens after a timeout? A model-generated tool request is a proposal until the application validates and executes it. The final response should reflect the destination's result.
Also inspect what the application cannot do. A narrow product can be a good purchase when its limitations match the need. Problems arise when the demonstration suggests broad capabilities but the contract excludes the data, integrations or controls that would make those capabilities usable.
Inspect evaluation and change management
Ask which cases were used during development and which were reserved for assessment. Request results by task category, including missing evidence, restricted records and unsuccessful runs. An aggregate success claim without a defined population gives little basis for deciding suitability.
Ask what changes trigger a re-evaluation. Model configuration, instructions, tool schemas and source-processing rules can all affect behaviour. A provider should explain how it identifies the deployed configuration and how it detects regressions. You do not need every implementation detail to require a clear change process.
Evaluate correction as well as initial output. If a reviewer identifies an unsupported claim, can the team trace its source, fix the underlying issue and prevent recurrence? Rewriting the answer manually may solve one case while leaving the same fault available to the next user.
Inspect the commercial and operating boundary
Clarify what is included: application access, configuration, integration work, evaluation assets, source maintenance, support and export. Do not assume that subscribing to a model-backed interface includes a managed business workflow. Equally, do not reject a simple service if it transparently covers a narrow need at an acceptable cost.
Ask what remains your responsibility. Someone must own approved source material, user access and the business interpretation of ambiguous cases. A supplier can support those processes, but the agreement should make the division explicit. The build versus buy guide compares different ownership arrangements.
For a formal selection, convert the inspection into specific evidence requests rather than a demand that the supplier prove it is “not a wrapper”. The pre-purchase evaluation guide helps structure that assessment.
Judge the smallest sufficient system
A well-designed solution can be simple. The buying standard should be that it performs the supported task, makes limits visible and provides the controls needed for its authority. More autonomous behaviour is useful only when it solves a real coordination or interpretation problem.
Open for AI scopes custom AI agent development around those requirements. To review the substance of a proposed agent design, bring the expected task, a source-sensitive failure case and the capabilities a supplier has promised. Those details make the evaluation more useful than debating a label.