AI Auditing

Do not ask whether your AI looks convincing. Ask what the evidence proves.

Your supplier says the system is accurate. Your team says a human approves important actions. Your policy says restricted information stays restricted. Before those claims support a business decision, somebody needs to establish what was tested, which configuration was tested, and whether the controls work in the conditions that matter.

Open for AI provides AI auditing and technical assurance assessments in Saudi Arabia. We examine selected systems, evidence and operating controls against an agreed assessment basis, then report findings your business and engineering teams can act on. The result is a defined account of what is supported, what failed, what could not be verified, and what needs to happen next.

This is an evidence-led system assessment. It is not a blanket declaration that an AI product is safe, unbiased or legally compliant.

Begin with the decision the audit must inform

An audit before launch has a different purpose from a review after an incident. A procurement assessment may need to test a supplier's claims; a periodic review may need to establish whether approved controls continued to operate; a board discussion may need a concise explanation of unresolved exposure.

We agree the decision first. Then we define the system boundary, intended use, version, environment, data sources, tools, users and relevant period. A hosted model is only one component: retrieval, application rules, permissions, human review and downstream execution can change the outcome.

Our engagement letter also identifies the assessment criteria, available access, sampling approach, testing authority and report audience. A missing integration or unavailable production record becomes a visible limitation. It does not quietly become a pass.

The questions we can test

Audit questions and the evidence they require

Question	Evidence examined	What the conclusion must distinguish
Does it perform the intended task?	Representative cases, reference answers, reviewer criteria and observed outcomes	Model fluency versus correct completion of the business task
Can it see only permitted information?	User identities, source permissions, retrieval results and negative-access tests	A documented access rule versus an enforced boundary
Are consequential actions approved?	Approval records, tool requests, execution controls and destination records	A confirmation screen versus approval bound to the exact action
Can the team investigate a failure?	Version references, logs, trace coverage, retention and access controls	An available dashboard versus sufficient evidence for an investigation
Do fixes address the finding?	Changed controls, repeated tests and relevant regression cases	An acknowledged recommendation versus a verified remediation

The audit can cover generative AI applications, retrieval-based assistants, tool-using agents and predictive systems. The test methods differ. For a predictive model, data separation, thresholds and relevant population performance may be central. For an agent, tool authority, execution outcomes and recovery may be decisive. We do not force every system through an identical chatbot checklist.

A finding should connect a claim to an observed condition

Consider this illustrative finding, not a result from a customer engagement. The system owner claims that a user in one business unit cannot retrieve another unit's documents. In an authorised test environment, a restricted test document appears in the first user's answer.

A useful finding records the expected boundary, test identity, configuration, source-document permissions, observed retrieval, sanitised evidence and conditions needed to reproduce it. It explains the plausible consequence without exposing the restricted content in a broadly distributed report.

The recommended correction addresses the authorisation boundary and the affected retrieval paths. Retesting includes the original case, a permitted-access case and adjacent cases such as a recently revoked permission. A stronger prompt alone would not establish that the retrieval boundary is enforced.

Severity is assigned using agreed impact and exposure criteria. We distinguish an observed failure, an inferred wider risk and an untested possibility. Management can then decide on containment and remediation with a clear understanding of the evidence.

Four examination layers, chosen around your system

1. Performance and evaluation integrity

We examine how the team defined success, assembled test cases and chose acceptance thresholds. We look for overlap between development examples and evaluation cases, ambiguous reference answers, unreported exclusions and metrics that hide materially different outcomes.

For systems with variable outputs, selected cases may need repeated runs and an explanation of observed variation. For an automated evaluator, we examine its criteria and calibration against human review. Its score is evidence from another measurement process, not an unquestionable ground truth.

When Arabic and English are in scope, the assessment addresses both. We can examine relevant dialects, mixed-language identifiers, translation of commercial conditions and language-specific escalation. Coverage is reported by the tested task and language rather than concealed inside one overall number.

2. Security and data boundaries

We review the relevant attack surface and perform authorised tests for such concerns as untrusted instructions, information exposure, excessive permissions and misuse of tools. The OWASP Top 10 for Agentic Applications is one reference for organising agent-specific security questions; it is not the complete scope of an AI audit.

Testing is performed under agreed rules of engagement, with explicit limits on environments, data, actions and third parties. We favour synthetic or approved test material, bounded actions and recoverable environments. There is no assumption of permission to probe a supplier's infrastructure or execute consequential production actions.

3. Human oversight and operational controls

We follow the path from a proposed action to an approved action and an observed result. Does the reviewer see the relevant evidence? Is approval tied to the actual recipient, amount, content or record? Can a later change bypass that approval? What happens if the reviewer is unavailable?

We also examine monitoring, incident ownership, safe stopping, rollback and repeated-action handling where relevant. A control's design is assessed separately from evidence that it operated during the stated period. One successful demonstration does not establish uninterrupted operation.

4. Governance and supplier evidence

We assess selected responsibilities, approvals, risk decisions, change records and supplier claims against the agreed criteria. Documents are examined for relevance to the actual deployed system, not merely their existence.

The NIST AI RMF Measure guidance supports context-specific measurement and documentation of risks that cannot be measured. We follow that discipline when reporting coverage: tested, not tested, inconclusive and unsupported are different outcomes.

Access determines how strong the conclusion can be

A documentation-only review can identify gaps in stated controls and supplied evidence. It cannot establish that inaccessible runtime checks work. Interface testing can expose behavioural failures, but may not reveal the underlying cause. Source, configuration and operating-record access can support deeper investigation, subject to the scope and completeness of those materials.

We agree the access model before quoting the work. For a vendor-controlled application, the engagement may combine observable behaviour, available configuration and supplier evidence. The report names the controls that remained inaccessible and the additional assurance that would require supplier participation.

Evidence handling is agreed too: collection authority, redaction, storage, access, retention and disposal. The audit should not create a new uncontrolled repository of sensitive conversations, credentials or business records.

The report must support a decision and a fix

The executive report explains the assessment purpose, tested boundary, principal findings, limitations and decisions needed. It avoids presenting a single “AI score” as a substitute for the actual exposure.

The technical findings register contains the criterion, observation, evidence reference, affected component, consequence, severity rationale, recommended action and retest method. An associated remediation plan assigns owners and priorities. Where a potential issue requires further access or investigation, it is recorded as such.

For quantitative results, we disclose the workload, sample, denominator, exclusions and evaluation method. Statistical claims require an appropriate design; a convenient set of test prompts does not automatically support a population-wide accuracy claim. For rare or high-consequence events, absence from a limited sample is not proof of absence.

At handover, the responsible teams should be able to locate the evidence, understand the disputed or uncertain points and reproduce authorised tests where feasible. A finding should not depend on accepting the assessor's unexplained opinion.

Retesting is part of closure, not a change of wording

An accepted finding, a planned correction, a deployed change and a verified fix are separate states. We can scope a follow-up assessment to check the original condition and agreed related cases. Evidence from that retest is appended with its own configuration and date.

Where a fix changes retrieval, permissions or an approval flow, the retest also checks that legitimate work still functions. Disabling the feature may be a valid containment measure, but it should not be reported as restoration of the original capability.

A point-in-time audit does not replace ongoing AgentOps. Subsequent model, data, tool or policy changes can create new behaviour outside the tested configuration.

Independence, certification and Saudi applicability

Before accepting an engagement, we disclose relevant prior work on the system and agree the reviewer relationship. If Open for AI designed or implemented the assessed controls, the work is described as an implementation review or internal readiness assessment unless a suitable separate assurance arrangement is established. We do not label our own work independently certified.

Technical assessment, legal advice and accredited certification are different services. This engagement does not issue regulatory approval or an ISO certificate. ISO itself explains that certification is performed by external certification bodies. Any formal attestation requirement must be identified before the assessment is commissioned.

For Saudi organisations, applicable privacy, sector and contractual requirements are agreed with the relevant responsible specialists. SDAIA's controller and processor guidance is a reference for personal-data questions. We can evaluate selected technical and organisational evidence against an agreed requirement; we do not infer universal PDPL compliance from a test suite.

Scope the assessment before buying reassurance

Bring the decision you need to make, the system and environment to assess, available architecture and test material, supplier restrictions and your material concerns. Pricing depends on the boundary, access, number of workflows and languages, evaluation design, authorised security testing and retest requirements. An attractive fixed price without an agreed scope can leave the most important questions untouched.

If your immediate need is to create the controls, use AI governance setup. If the information foundation cannot yet support an evaluation, enterprise data and AI readiness may be the appropriate first engagement.

Request an AI audit scoping discussion. We will define the claims to examine, the evidence needed and the limits of the conclusion before testing begins.