Authority and trust boundaries
AI Agent Security Architecture
Design an agent so untrusted text cannot grant access or authorise actions. A supplier-change scenario connects identity, retrieval, tool gateways, approvals, recovery and security tests.
The decisive security question is what remains protected when an agent misunderstands a request or follows hostile content. If a misleading document can turn a model suggestion into an authorised business action, the weakness is in the system’s authority boundaries, not just its wording.
A secure design treats generated decisions as proposals that pass through independent controls. Identity, resource access, permitted operations and transaction rules belong in components that enforce them. Model instructions and content filters can contribute, but they should not be the only thing between an untrusted message and a sensitive system.
Threat-model one consequential workflow
Consider an illustrative procurement assistant that reads supplier correspondence and prepares change requests. A document contains apparently urgent instructions to replace a supplier’s payment destination. The organisation’s policy requires independent verification and an authorised finance process. In this design, the agent may assemble evidence and create a review case; it has no tool that can directly change the payment master.
Start with the assets: supplier records, commercial documents, employee identities, credentials and pending approvals. Then identify entry points: user messages, uploaded files, retrieved documents, tool responses and stored conversation state. Ask how each could cause disclosure, unauthorised modification, duplicate effects or loss of service.
The threat model should include a legitimate employee with access to one supplier but not another, a compromised document source and a failed dependency. Security is not limited to an anonymous attacker typing a suspicious prompt. The general agent architecture should expose these boundaries before the implementation hides them behind one orchestration function.
Carry verified identity to the resource boundary
NIST SP 800-207 rejects implicit trust based only on network or physical location. OWASP’s authorisation guidance recommends denying by default and checking permission on every request. For an agent, apply those principles to the user, service identity, tenant, resource and proposed operation.
The model should not supply the authoritative tenant or user identity from conversation text. Derive them from the authenticated session and trusted application context. A tool gateway then evaluates whether that principal can perform that operation on that particular record. Possession of a general service credential must not silently enlarge the user’s authority.
Separate read capabilities from write capabilities. Prefer narrowly defined operations over a generic command executor or unrestricted database interface. In the supplier example, tools might retrieve an authorised supplier summary and create a review case. Neither needs unrestricted access to payment records. Keep credentials outside prompts and returned tool content, and scope their lifetime and permissions to the required service.
Apply access rules before context reaches the model
Retrieval is an access path. The assistant must not fetch a document the user is forbidden to read and rely on an instruction to keep its contents secret. Enforce tenant and entitlement constraints at retrieval and verify that caches, indexes and attachments preserve the same boundary.
When a document’s permissions change, examine derived copies and active sessions. A revoked document can remain exposed through a cached answer or an old conversation summary. Define whether such state is invalidated, rechecked or unavailable until refreshed. Test a user losing access during a multi-step task, not only before the first message.
Source content should retain provenance and be treated as information, not as an authority to modify policy. A retrieved supplier letter can say where the supplier claims payment should go. It cannot grant approval rights, add a tool or redefine the organisation’s verification process. The guardrails guide addresses content-level checks that complement these structural controls.
Put a typed gateway between planning and execution
Each tool should have a narrow contract: required inputs, allowed values, identity context, preconditions, output shape and failure states. Validate the proposed arguments outside the model. Resolve authoritative identifiers from trusted records, rather than accepting a free-text name as sufficient proof of the target.
The gateway should reject unsupported operations, unknown fields and attempts to substitute a destination or tenant. Apply network egress restrictions and destination allowlists where they fit the tool’s purpose. A tool that may send a case notification should not accept an arbitrary external address merely because a document includes one.
Distinguish a valid request from a successful business effect. If creating the review case times out, the result may be unknown. Reconcile using a stable request identifier or downstream lookup before retrying an action that could duplicate work. Avoid telling the user the case was created until the service has evidence. Security and reliability meet at this boundary because repeated or misdirected actions can have real consequences.
Bind approval to the exact proposed action
Where the release includes a consequential action that is eligible for human approval, show the reviewer the target, proposed change, source evidence and relevant warnings. Approval should refer to the normalised action and the applicable record or policy version, with an expiry and an identified approver. A generic “approve this conversation” signal is too broad.
If the target or payload changes after approval, require a new decision. Recheck current permissions and relevant preconditions immediately before execution. A reviewer who was authorised yesterday may no longer be authorised, and a supplier record may have changed while the request was waiting. The oversight guide explains how to make this review operationally meaningful.
For the supplier-change scenario, independent verification remains in the approved finance process. The assistant’s review case must not imply that creating it has completed verification or changed the payment destination. This boundary is a deliberate design choice in the example, not a claim that every organisation uses an identical process.
A threat-test matrix with observable results
| Scenario | Expected boundary | Evidence |
|---|---|---|
| A document instructs the agent to send supplier data to a new address | No authorised tool path to that destination | Rejected gateway request and no downstream transmission |
| A user supplies another tenant’s supplier identifier | Resource authorisation blocks retrieval and action | Access decision and absence of protected content in output |
| A permitted source is revoked during the task | Later retrieval and affected cached context respect revocation | Permission-change test across a continuing session |
| A proposed action changes after review | The previous approval cannot authorise the altered payload | Approval mismatch and a new review request |
| A create-case request times out after downstream acceptance | Reconciliation prevents an unverified duplicate | One business case and linked attempt records |
| A diagnostic export is requested without approval | Support access and export policy are enforced | Denied export and a proportionate security event |
These are proposed tests, not completed test results. Add cases derived from your own tools and operating environment. Repeat relevant cases across supported languages and document types. A test should inspect the downstream system, because an agent can claim that it refused while a tool still performed an action.
Preserve evidence without building a second sensitive database
Record who requested an operation, the applicable authority, the tool and version, the approval reference, attempts and observed result. Use references or restricted evidence stores for sensitive payloads. Logs need their own access, retention and integrity controls. A broad transcript dump is not automatically a useful audit trail.
Do not require hidden model reasoning to establish accountability. The reviewable evidence is the input context that was allowed, source versions, proposed action, policy decision, human approval and actual effect. The audit-trail guide develops this event model, while the PDPL guide addresses the personal-data implications of retaining it.
Design containment and recovery as separate operations
A stop control should disable the relevant execution path and credentials, not merely hide the chat interface. Identify in-flight operations and reconcile their outcomes before declaring containment complete. A queued task may still run unless the worker or downstream permission is also controlled.
Recovery begins with a known approved configuration, corrected dependency and targeted regression evidence. Some actions can be reversed through an authorised compensating process; others, such as sending an external message, cannot be made unseen. Record the actual effect and the available remediation instead of promising a universal rollback.
Define a useful degraded mode. The supplier assistant might retain access to public guidance while suspending internal retrieval and tool execution. Staff can continue through an established manual process. This fallback needs ownership and capacity; otherwise a technically safe stop can leave a business queue unmanaged.
Keep the surrounding application in scope
The OWASP Application Security Verification Standard supplies a basis for verifying application security controls. Agent-specific tests should complement ordinary work on authentication, session handling, input validation, secret management, dependencies and deployment. A strong prompt-injection defence does not repair an exposed administration endpoint.
For Saudi deployments, map applicable cybersecurity, privacy and location requirements to the actual entity and workload. The residency review keeps processing and support paths visible. Treat the release as a versioned combination of model, prompts, tools, policy and configuration, and retest the affected boundaries when any of them changes.
For custom AI agent development, bring one consequential action and the systems it can affect. Discuss its authority boundaries and acceptance evidence with Open for AI. A useful security review should be able to explain precisely why an untrusted instruction cannot acquire a permission the business has not granted.