AI Audit vs AI Red Teaming

An AI audit and an AI red-team exercise serve related but different decisions. An audit examines agreed claims and controls using evidence, then reports findings and limitations. Red teaming deliberately searches for ways an AI system can behave undesirably under authorised test conditions. A red-team result can become important audit evidence, but it does not automatically answer the audit's wider questions about governance, operating records or the completeness of control coverage.

The choice should follow the uncertainty you need to resolve. If the question is whether a new attack path can defeat an existing boundary, a focused adversarial exercise may be appropriate. If the question is whether management has support for a release decision across performance, access, oversight and change control, a broader audit may be needed. Many engagements benefit from both, provided their different populations, methods and conclusions stay visible.

Compare the question before comparing the provider

NIST describes AI red teaming as structured work intended to discover flaws or vulnerabilities, often in a controlled setting. The important word for a buyer is discovery. A capable team adapts its attempts as it learns about the system, because the objective is to find credible weaknesses rather than preserve a representative sample of everyday use.

An audit can include exploratory testing, but its report must also account for the criteria and boundaries agreed with the commissioning party. A negative result from one exercise cannot fill an untested audit area. Conversely, finding one serious weakness can be enough to challenge a control claim even if the exercise cannot estimate how often the weakness would occur in production. These differences explain why a single “pass rate” is often a poor way to compare the services.

Choosing work by the decision it must support

Decision question	Suitable starting point	Evidence still needed
Can a selected boundary be defeated?	Authorised red-team exercise with explicit threat assumptions	Reproduction, affected configuration and practical consequence
Did approval controls operate during the review period?	Audit of design, implementation and selected operating records	Complete enough records and a justified selection approach
Should a release proceed despite known weaknesses?	Audit synthesis informed by red-team results	Management's risk decision, restrictions and closure conditions
Did a repair close the original exploit path?	Controlled retest with relevant regression cases	Changed version, destination effects and residual limits

This is an original procurement aid, not a claim that one discipline owns every procedure. Skilled audit and red-team practitioners can overlap. The contract should describe their work and competence instead of relying on a service label. Audit scoping helps turn an ambiguous request for “AI safety testing” into questions a provider can actually answer.

A hypothetical label-drafting assistant

Consider a household-product company using an AI assistant to draft descriptive text for packaging. Approved source records contain product features and controlled wording. Staff review every draft before artwork is prepared. The assistant is not authorised to establish chemical safety, approve mandatory labelling or replace specialist regulatory review. This hypothetical example concerns the integrity of the drafting workflow, not advice on product formulation or a certification of labels.

The audit asks whether the assistant uses the approved records, preserves required review status and retains evidence of changes. The red team asks whether selected adversarial inputs can make the application disregard its source restrictions or present an unapproved draft as approved. Both examine the same application, but their test designs differ. The audit may trace ordinary completed drafts; the adversarial team intentionally constructs unusual combinations that a normal operating sample may never contain.

A useful starting boundary identifies the staging environment, synthetic product records, allowed user roles and disconnected publishing endpoint. The team may test whether a draft can move through an unauthorised state transition, but it should not send real artwork to a printer or expose customer records. A convincing simulated consequence is enough to demonstrate a control weakness when the test environment reproduces the relevant enforcement mechanism.

What the exercise must be authorised to do

Written rules of engagement should name the target, ownership authority, dates, test accounts, permitted techniques and excluded actions. They should also identify stop conditions, an operational contact and the process for handling unexpected sensitive data. Permission to inspect a chatbot does not imply permission to attack its supplier, test another tenant or access unrelated employee information.

Separate the ability to submit an input from authority to trigger its consequences. An application can expose a tool call whose destination is outside the agreed exercise. Testing that boundary may require a simulated tool or a controlled destination. Agree whether the team can inspect logs, change configuration, create accounts or attempt workflow transitions. Restrictions are legitimate, but the report must explain how they limit the findings.

Illustrative exercise charter for the label-drafting assistant

Charter field	Example agreement	Why it matters
Target	One staging release with recorded application, prompt and source-set versions	Findings can be connected to a reproducible configuration
Data	Synthetic product records and masked examples authorised for testing	The exercise does not create an unnecessary sensitive-data collection
Permitted consequence	Write only to a simulated artwork queue	Workflow evidence is obtained without publishing a real label
Stop condition	Unexpected access to real records or an uncontrolled external destination	The team pauses and informs the agreed contact
Evidence	Inputs, relevant context, resulting state, logs and destination receipt	A persuasive response alone is not mistaken for an executed action

Coverage is a set of hypotheses, not a pile of prompts

A thousand variations of one prompt may explore one narrow weakness. A smaller, carefully organised exercise may examine several distinct trust boundaries. Ask the team to group attempts by the mechanism being challenged: source authority, role permissions, review-state enforcement, tool use or record integrity. The categories should follow the application, not a copied catalogue of fashionable attacks.

For each hypothesis, record the starting privileges, the information visible to the tester and the expected boundary. A test using an administrator account cannot establish what an ordinary user can do. A test that gives the adversary internal configuration is different from one limited to public documentation. Both can be useful when their assumptions are explicit; combining their outcomes into a single unexplained success rate obscures the threat model.

Coverage also includes paths that could not be exercised. If the supplier prevents access to an export component, the team can report observations at the interface but should not claim the component was examined internally. Tested versus not-tested coverage provides a way to make these limits visible. An untested high-consequence path may be more important to the release decision than dozens of completed low-consequence tests.

What a successful attack actually establishes

Suppose a controlled test makes the hypothetical assistant state that a draft is approved. That is evidence of an incorrect statement, but it does not by itself establish that the publishing workflow accepted the draft. The investigator must inspect the application state and simulated destination. If the server rejected the action, the reporting should distinguish misleading user communication from an approval-control bypass.

If the simulated artwork queue accepted the unapproved draft, preserve the exact version, user role, state sequence and receipt. The confirmed observation challenges the claim that only approved drafts can enter that queue. It does not establish that the same path exists in every deployment, that it has been exploited historically or that every product label is affected. Those are separate questions requiring separate evidence.

Reproduction should use the smallest controlled procedure sufficient to verify the issue. Avoid circulating unnecessary sensitive transcripts or executable attack material beyond the authorised audience. A technical appendix can provide the repair team with the detail it needs, while the management finding explains the violated boundary, consequence and uncertainty. The level of disclosure should serve remediation without making the evidence package itself a new risk.

Why attack success rates need careful interpretation

Red-team attempts are often adaptive. Later attempts build on earlier discoveries, and many attempts share a mechanism. They are therefore not automatically independent samples. A ratio of successful attempts to total attempts describes that exercise under its counting rules. It cannot be presented as the probability that an ordinary customer, or any attacker, will succeed in production.

Consider an illustrative exercise containing 60 attempts across 6 hypotheses. Four successful attempts may all exploit one missing permission check. Reporting 4 of 60 is less useful than explaining that one boundary failed, the access required, the affected path and whether other hypotheses were adequately explored. Counting each paraphrase as a separate vulnerability inflates the picture; collapsing distinct failed boundaries into one generic issue hides it.

Define what counts as success before comparing results. A harmful-looking answer, a proposed tool call, an attempted action and a confirmed destination effect are different events. Pending or unobservable outcomes should remain separate. A zero-success result means no success was observed under the performed tests; it does not prove the absence of exploitable weaknesses. This is particularly important when the exercise had short time limits or incomplete access.

How red-team evidence becomes an audit finding

The auditor first checks relevance: does the tested version, role and environment correspond to the claim under review? Next comes credibility: are the records sufficient to understand what happened, and can material observations be corroborated? Then comes interpretation: which criterion was not met, and what conclusion is justified? The red team's narrative is a starting point for this work, not an automatic substitute for it.

In the label example, an audit finding might state that the approval-state control did not prevent an unapproved draft entering the simulated queue in one tested configuration. It can link the red-team evidence to the approved workflow requirement and identify the release restriction proposed by management. If the environment differs materially from production, that difference belongs in the finding, not in an obscure footnote.

The audit may also discover why the organisation had not detected the issue: no owner for source restrictions, missing review records or a release process that omitted the affected path. These are different findings with different evidence. Avoid assuming that a technical weakness proves every surrounding process failed. The working-paper trail should allow a reviewer to follow each conclusion back to its own support.

Sequencing discovery, repair and independent review

A practical programme may begin with a design and scope review, then run an authorised adversarial exercise, repair material findings and conduct retesting before a broader release conclusion. Another programme may audit an existing service and commission targeted red teaming only where evidence reveals a significant unresolved boundary. The right sequence depends on the decision and available evidence, not a universal maturity ladder.

Preserve a baseline before repair. If testers and developers repeatedly change the application without recording versions, a final clean demonstration cannot explain which earlier issues were fixed or which configuration was examined. Keep discovery evidence, remediation evidence and closure evidence distinct. Retesting should include the original failure condition and relevant neighbouring cases, because blocking one wording may leave the underlying authority failure untouched.

The same provider may help discover and explain weaknesses, but independence claims require care if it also implements the repairs or designed the original system. Agree who evaluates closure and who owns the final judgement. A collaborative exercise can be valuable while still having a disclosed self-review limitation. Buyers should prefer a precise statement of roles over an unsupported promise of total independence.

Compare exercises without rewarding theatrical results

When comparing red-team proposals, ask how the team will document learning during the exercise. A useful record shows the initial hypothesis, the reason for a new attempt and the evidence that changed the team's understanding. This helps distinguish systematic exploration from a large transcript of loosely related prompts. It also gives defenders a clearer account of the mechanism they need to repair.

In the label-drafting example, several attempts may produce unauthorised wording while only one reaches the simulated queue. Those observations should not be collapsed into one attack score. The wording failures may concern communication or source adherence; the queue result concerns an enforced workflow boundary. The report should connect each to its own consequence and show whether the same underlying defect explains them. A dramatic generated sentence is not necessarily the most important result.

Ask what the team will do when it finds nothing in a selected area. It should describe the hypotheses explored, conditions and available time, then state the limit of the negative result. A provider should not be rewarded for inventing a weak finding to justify the exercise. Equally, a no-finding report should not omit that an important role or tool was inaccessible. The buyer needs an honest account of search effort and coverage, not a predetermined number of vulnerabilities.

Separate discovery yield from defensive improvement

An exercise can be productive because it reveals one material authority flaw, even if its count of successful attempts is small. Another can generate many superficially different failures without changing the organisation's understanding. Evaluate discovery yield by the distinct mechanisms, consequences and new evidence produced. This is a qualitative programme judgement, not a universal numerical benchmark or a claim that one finding always represents good value.

Defensive improvement needs additional evidence. After repair, test whether the original boundary is enforced and whether related routes remain vulnerable. If a fix merely blocks a phrase, a changed wording may still reach the same unauthorised transition. If a server-side authority check is added, the evaluator should inspect the relevant destination effect and failure handling rather than relying only on the model's refusal message. The method should follow the actual repair.

Keep a controlled record of lessons that can be reused without circulating sensitive exploit detail unnecessarily. For example, the organisation can record that approval must be bound to the exact draft and enforced outside the generated text, while restricting detailed reproduction material to authorised repair and review teams. A later audit can examine whether that lesson changed the control and whether evidence of operation exists. In this way, discovery contributes to a broader assurance programme without being mistaken for the whole programme.

The buyer should also decide which results require independent challenge. A team that helped implement the repair can explain it, but another competent reviewer may be needed for the intended closure claim. This is particularly relevant when a release decision depends on a small number of consequential boundaries. The report should state who discovered, who repaired and who evaluated closure so the reader can assess the chain of responsibility.

A comparison should therefore preserve the conditions that made each result interpretable, including the access actually used and the evidence retained for review.

Questions to put into the procurement brief

Ask what decision the deliverable supports, which threat assumptions apply, which environments and languages are included, and how success is confirmed. Request an example of a redacted finding that distinguishes an observed response from an executed consequence. Ask how the team handles denied access, unresolved results and changes to the target. The answer should describe actual evidence and reporting, not just a number of prompts or testing days.

For an audit-plus-red-team engagement, require a coverage map linking adversarial hypotheses to audit criteria. Make clear which areas depend on other procedures, such as operating-record review or human-oversight testing. Include rules for urgent notification and evidence retention, then agree the retest boundary. These provisions make two complementary disciplines work together without pretending that either provides universal assurance.

Open for AI can scope AI auditing around the evidence needed for a defined decision. If your immediate concern is an enforceable trust boundary, review AI security boundary testing and bring a masked architecture and permitted-action map to an audit scoping discussion about adversarial testing. The resulting brief should say what will be challenged, what will be audited and what will remain outside the conclusion.