Agentic AI Audit

An agentic AI audit examines the relationship between a requested goal, the authority granted to an agent and the effects it produces in other systems. The decisive evidence is not a convincing plan or a message saying “completed”. It is the trace connecting the authorised request, the exact proposed action, the approval where required, the attempted operation and the confirmed destination state.

This guide is for technology owners, security leaders and internal audit teams assessing agents that can call tools or change business records. The running example is a hypothetical software-licence seat assignment agent. It may propose assigning an existing seat to a named employee under a defined entitlement policy. It may not purchase licences, expand permissions or substitute another employee without the relevant authority. This bounded example keeps the audit focused on action integrity rather than conversational quality alone.

The OWASP agentic applications guidance provides a security starting point for systems that plan and act. The method below is an original evidence-led approach, not an OWASP certification or a universal compliance checklist. A wider review should define its criteria, independence and access within an AI auditing scope.

Start with the authority model

List the actions the agent can propose and the actions it can execute. Distinguish reading an employee directory, checking a licence inventory, creating an assignment proposal, submitting an approved request and changing the destination entitlement. These are different permissions even when the interface presents them as one helpful task. Identify the person or service responsible for authorising each transition.

Record the identities used at every boundary. An agent may receive a request from an employee but execute through a service account with broader permissions. The audit should establish whether the application constrains that account to the approved purpose. A prompt telling the model to respect policy is not equivalent to an enforced permission check in the destination or a trusted execution layer.

Define explicit prohibited effects. In the licence example, an assignment agent might be allowed to use an available standard seat but not create a paid subscription or grant an administrator role. Test the distinction using controlled fixtures. Avoid a vague rule such as “only safe actions”, which leaves the reviewer unable to determine whether an observed operation complied with the intended authority.

Use an action record that preserves distinct states

A useful action record separates what was requested, proposed, approved, attempted, executed and confirmed. These states should not collapse into one success flag. An approved proposal can fail to execute; an operation can execute while its acknowledgement is lost; and a confirmation can refer to a different object from the one originally approved.

Original action-evidence record for a hypothetical licence assignment

State	Evidence to retain	Question the auditor answers
Requested	Request identifier, requester identity, employee and intended product.	Was the goal attributable to an authorised requester?
Proposed	Exact employee identifier, product, seat type, role and relevant conditions.	Was the proposed effect clear enough to approve?
Approved	Approver identity, authority, proposal version, expiry and constraints.	Did approval bind to this specific action?
Attempted	Tool operation identifier, authenticated executor and submitted parameters.	Did the execution request match the approved proposal?
Executed	Destination transaction evidence, or before-and-after state linked to the operation.	Did the entitlement actually change?
Confirmed	Reconciled outcome, timestamp and user-facing status.	Was the reported outcome supported, including any uncertainty?

The record does not require storing unrestricted prompts or personal data. Retain the minimum evidence needed to connect the action and decision, under the agreed handling rules. Where a field is unavailable, say what cannot be established. The working-paper guide explains how linked evidence supports a reproducible test without turning every internal log into an indiscriminate audit export.

Test whether approval remains attached to the approved action

In the example, a manager approves a standard product seat for employee E-41. The auditor should test whether changing the employee, product, role or other material condition invalidates that approval. The relevant evidence includes the version shown to the approver and the parameters actually sent to the destination. An approval button associated only with a conversation identifier may be too imprecise to establish what was authorised.

Inspect the user's decision surface. If the interface hides the role, destination account or cost-bearing effect, a recorded approval may not demonstrate informed approval of those elements. The audit should distinguish a deficient approval design from an operational failure to follow an adequate design. Both can matter, but they require different evidence and corrective owners.

Test expiry, revocation and changes in the approver's authority. A proposal approved yesterday may no longer be valid after the employee changes department or the product becomes restricted. Define the relevant rule with the policy owner rather than inventing a universal approval lifetime. Where a new check is required, establish where it is enforced and whether the agent can bypass it through another tool path.

Human oversight must also be operationally credible. A manager who receives an unintelligible summary or cannot reject the proposal is not providing the same control as a reviewer with clear information and effective authority. The human oversight audit examines that decision process in more detail. This page concentrates on whether the approval actually binds to the effect executed.

Challenge concurrent changes, not just the happy path

A read immediately before a write does not eliminate a race. The destination state can change between the availability check and the assignment. MITRE's time-of-check/time-of-use weakness description captures this general problem. An audit should examine the actual concurrency control, not accept the phrase “we recheck before execution” as sufficient proof.

For a controlled test, begin with one available test seat and two otherwise authorised assignment requests. Coordinate their execution in a sandbox so both encounter the relevant transition. Inspect whether the destination enforces the required condition atomically, or whether another tested mechanism serialises or rejects conflicting operations. A separate pre-read and write may leave a gap even if both calls succeed individually.

The expected outcome depends on the destination's semantics. One request might succeed and the other receive a conflict; a reservation system might hold the seat before final assignment. The auditor should verify the implemented mechanism and its failure behaviour. Do not claim that a lock in one application protects another system unless the evidence establishes that relationship across the relevant boundary.

If the destination cannot enforce the necessary condition, management may need a narrower automation boundary or an additional reconciliation process. That is a risk decision, not proof that the original design works. Record residual exposure and the person accepting it. A compensating review after an unintended entitlement is created differs from prevention of that entitlement.

Test retries and uncertain outcomes without assuming exactly-once effects

A timeout can occur after the destination has performed an operation. Retrying immediately may therefore produce a duplicate or another unintended change. In the licence example, the user-facing system should not equate a missing acknowledgement with a proven absence of assignment. The audit needs evidence of how the service resolves an uncertain result before taking further action.

Use an authorised test harness or vendor-supported fault simulation to interrupt acknowledgement after a controlled operation. Record the operation identifier and inspect the destination state through an independent supported route. Examine whether a retry refers to the same logical request and whether the destination's duplicate handling actually covers the operation. An idempotency field in a request is not proof of end-to-end duplicate protection.

Review the scope and lifetime of any deduplication mechanism. It may apply only to one endpoint, one tenant or a limited window. A different tool path may bypass it. The report should describe what was tested, including the outcome after the documented window where relevant. Avoid promising exactly-once effects across distributed systems on the basis of a single retry demonstration.

Check status language. “Assignment pending verification” may be appropriate while the result is uncertain. “Failed” can encourage another request; “completed” can cause the requester to rely on an entitlement that does not exist. The audit should connect user messages to recorded states and verify that later reconciliation updates both the operational record and the information shown to the requester.

Examine delegation and tool selection

An agent may delegate research or execution to another component. Identify whether the delegated component receives the same constraints, a narrower authority or an unintended broader credential. Follow at least one action through the full chain. A parent agent's valid approval does not automatically authorise every action a delegated agent may choose.

Review how tool descriptions and external content affect selection. A licence request attachment should be treated as task data, not as authority to add a new tool or change the permitted employee. Controlled tests can include benign synthetic instructions that conflict with the established workflow, without using real secrets or unauthorised targets. The expected result is enforcement of the approved boundary.

Distinguish the model's proposed call from the call accepted by the execution layer. If an invalid proposal is rejected before reaching the destination, that is evidence of a functioning boundary in the tested case, even though the model attempted it. If the executor silently rewrites material parameters, examine whether the revised action needs renewed approval. A syntactically valid tool call is not necessarily an authorised one.

Run a compact but consequential test programme

The following original protocol uses synthetic employees and test licences. It is designed to challenge mechanisms, not estimate incident frequency. Agree environments, identities, permitted operations, rollback arrangements and stop conditions before execution. Broader adversarial work should be separately governed through security boundary testing.

Illustrative agent action audit tests and decision evidence

Test	Controlled change	Evidence needed	Interpretation
Approval substitution	Change E-41 to another synthetic employee after approval.	Approved proposal, attempted parameters and denial or renewed approval.	Acceptance without valid authority demonstrates a binding failure in this path.
Role expansion	Propose an administrator role outside the permitted seat assignment.	Execution-layer decision and destination state.	A model refusal alone does not establish enforcement.
Seat contention	Two authorised requests target one available test seat.	Ordered requests, destination conditions and final assignments.	Assess atomicity or the actual conflict control; do not infer it from prechecks.
Lost acknowledgement	Suppress the response after a supported test operation.	Destination outcome, retry identity and reconciliation trail.	Uncertainty must not become an unsupported success or failure claim.
Delegated execution	Follow a proposal through a secondary execution component.	Identity, constraints and operation at each boundary.	Confirm that delegated authority does not exceed the approved scope.
Revoked request	Withdraw approval before the execution boundary.	Revocation timing and the rule applied by the executor.	State whether the system prevented execution or only detected it later.

Record the configuration and repeat conditions necessary to understand each result. A single successful run may show that a control can work, while repeated or timed variations may reveal a boundary condition. Explain why cases were selected and preserve unsuccessful runs. The coverage guide helps show which paths, tools and states remain untested.

Inspect operating evidence and stop mechanisms

Controlled tests should be complemented by a defined period of operating records where authorised access permits. Reconcile action counts across the agent, execution layer and destination. Investigate actions with no corresponding request, approvals with no outcome and outcomes with conflicting statuses. Missing linkage may be an evidence limitation, a logging defect or an actual unauthorised action; do not assume which before investigating. A post-operation state may confirm that an entitlement exists without establishing which request created it; reconcile the relevant state transition and transaction evidence before attributing the effect.

Review the mechanism for suspending the agent's ability to act. A disabled chat interface may leave background workers or delegated credentials active. A stop control should be tested against its stated scope, including in-flight operations and already committed changes. The audit should distinguish stopping new work from cancelling a pending operation and from reversing a completed one.

Reversal also needs authority. Removing an assigned licence may interrupt legitimate work, and some effects cannot be undone cleanly. Examine the documented recovery decision and its owner rather than assuming every automation failure has an automatic rollback. Where a manual process is the fallback, inspect whether the team can identify unresolved actions and avoid repeating them.

Configuration changes can alter authority without changing the model. New tools, expanded service-account permissions, changed approval screens or a different destination endpoint may invalidate earlier evidence. Maintain a relationship between the approved action catalogue and the tested release. Changes to that catalogue should trigger a reasoned decision about focused testing.

Distinguish action coverage from a persuasive demonstration

An inventory of tools is not an inventory of all consequential actions. One tool may support several roles, destinations or parameter combinations with different authority requirements. Group the permitted effects into meaningful classes, then explain which classes the audit examines. Include read-only paths where they can expose sensitive information, and write paths where they can change entitlements. Count the logical business action separately from the number of internal tool calls it generates.

Operating records may overrepresent successful actions because failed proposals never reached the destination. Reconcile those records with proposal and rejection events before describing a success rate. Likewise, a series of retries for one assignment should not become several independent completed tasks. Define how cancelled, pending, rejected and unresolved actions appear in the population. Their treatment can materially change the apparent reliability of the service.

Targeted tests such as approval substitution intentionally select difficult conditions. They can establish a control weakness but cannot estimate how often ordinary users encounter it. If management needs an operating failure estimate, use a separately justified sampling design and retain the distinction in reporting. Rare but consequential authority failures should not disappear inside a high aggregate completion rate.

Illustrative interpretation of an approval-binding failure

Suppose a controlled test shows that a standard seat approved for E-41 is assigned to E-42 after a parameter change. The observed condition is an execution that differs from the approved target. The destination record and linked approval establish that condition for the tested path. They do not, by themselves, establish that every assignment is vulnerable or that the model caused the substitution. The change may have occurred in orchestration, user-interface state or an execution adapter.

The immediate decision may be to suspend that write path while retaining a read-only proposal service. This is an illustrative containment choice, not a mandatory response for every organisation. The owner should consider whether existing assignments require review, based on the exposed period and available evidence. A repair that binds the target identifier must then be tested against product and role substitutions too, if those share the same mechanism.

After correction, compare the approved representation with the destination effect in fresh controlled cases, and verify the rejection evidence for changed proposals. Check that the legitimate original request can still complete. If the audit cannot obtain destination records, it should not claim verified correction of the external effect merely because the application now displays a rejection message. That evidence gap remains a separate limitation for the release decision.

Report the effect, the evidence and the remaining authority

A useful finding states the intended boundary, observed action, destination consequence and evidence limitations. “The agent ignored instructions” is less actionable than a finding that an approved employee identifier could be substituted before execution. Keep the cause hypothesis separate until configuration, code or traces support it. A model behaviour change may expose a missing deterministic control rather than being the sole cause.

Assign remediation to the boundary owner and define closure evidence. Repeating the original failing case is necessary but may be insufficient if another endpoint performs the same action. Test relevant alternatives and confirm that permitted actions still work. The retesting guide distinguishes a closed engineering ticket from verified correction of the finding.

Before an audit scoping discussion, gather the action catalogue, permission map, one masked action trace, approval interface and destination evidence available for review. Discuss an audit of your agent's approval and execution boundaries to agree a conclusion that matches what the system is actually allowed to do.