LLM and Chatbot Audit

An LLM and chatbot audit examines whether a particular conversational application gives supportable answers, respects its permitted role and handles uncertainty consistently. The unit under review is the deployed service: its instructions, knowledge inputs, conversation state, identity controls, escalation process and recorded outcomes. A model benchmark alone cannot establish how that service behaves when a user corrects a date, changes language or asks it to make a decision it has no authority to make.

This guide is for product owners, digital service managers and assurance teams responsible for Arabic and English assistants. Its running example is a hypothetical professional examination information chatbot. The assistant explains published registration requirements and dates; it cannot determine an applicant's eligibility, reserve a place or grant an exception. The audit question is whether its conversations maintain that boundary while helping people find the next appropriate step. A broader engagement can be scoped through AI auditing services.

Define the claims that the chatbot must support

Start with the intended service promise. “Answers examination questions” is too broad to test. A useful specification says which examination programmes, registration periods, languages and information sources are supported, what personal context may be used, and when the assistant must transfer the enquiry. Identify the owner who can settle disagreements about the underlying rules. Without that authority, a reviewer can measure agreement with a reference answer but cannot know whether the reference is correct.

Separate informational claims from individual determinations. “The published closing date is 18 November” and “You can still apply” have different evidential requirements. The second may depend on programme, time zone, document status and exceptions. A chatbot that cites the right date can still make an unsupported personal promise. Record these claim types before collecting test conversations so that convenient examples do not redefine success after the results are visible.

Identify the exact deployed model identifier where available, application release, instruction version, retrieval configuration, session rules and enabled tools. If the provider cannot disclose an underlying model version, record the observable deployment identifier and the resulting reproducibility limitation. Freeze the sources used to judge time-sensitive answers. A later change to a registration notice should not silently turn a previously correct answer into an apparent model error.

Audit the whole conversation, not isolated answers

Single-turn testing asks whether one response is satisfactory. Conversation testing asks whether the service preserves relevant context, discards superseded information and maintains its role over a sequence. Both are useful, but they answer different questions. A correct first response may be followed by an incorrect summary after the user changes programme. A sensible refusal may become an unsupported commitment when the same request is phrased as a hypothetical.

Define a conversation as one test case with a documented starting state, ordered turns and end condition. Retain turn-level observations so a failure can be located. Do not count eight turns from one conversation as eight independent successes: they share context and may share one underlying mistake. Separate fresh sessions, resumed sessions and transfers to a human. The sampling guide explains why these populations should not be merged without a reason.

For the examination assistant, useful sequences include a user changing from a foundation examination to an advanced examination, correcting the intended year, asking in Arabic after an English response, and retracting an earlier statement about qualifications. The expected behaviour is not always a final answer. Asking a precise clarification or directing the user to an authorised official may be the appropriate outcome.

Conversation evidence matrix

Original audit matrix for the illustrative examination information assistant

Claim or boundary	Controlled test	Evidence and interpretation
Current registration date	Ask about a named programme and period using the approved notice.	Save notice version, response and displayed citation. Matching one date establishes that case, not the accuracy of every date.
Corrected user context	Change programme midway through the conversation.	Inspect subsequent answers and final summary. Continued use of the old programme is a state-handling failure.
No eligibility determination	Provide incomplete qualifications and request a definitive eligibility answer.	Record qualification of the answer and referral. A generic disclaimer does not cancel an explicit unsupported promise.
Language consistency	Repeat the same meaning in Arabic and English, including a negation.	Qualified bilingual reviewers compare material claims rather than literal wording.
Escalation continuity	Request authorised review after correcting an earlier detail.	Compare conversation, transfer summary and receiving queue acknowledgement.
Session isolation	Use separate synthetic applicants and fresh identities.	Check that one applicant's context does not appear in another session. These cases do not prove universal isolation.

The matrix is an original planning tool, not a regulatory checklist. Each row needs a named criterion, owner and permitted evidence location. Where the assistant uses a document index, use a separate RAG system audit to examine retrieval and access decisions in more detail. Otherwise, a correct final answer may conceal a retrieval failure that the model happened to compensate for.

Build reference answers that can withstand challenge

A reference answer should distinguish mandatory facts from acceptable variations in wording. For a registration deadline, mandatory facts might include the programme, period, closing date and applicable time convention. A helpful link and concise phrasing may be desirable but should not carry the same consequence as the wrong deadline. Define what makes an omission material to the user's next action.

Have a domain owner approve reference facts independently of the chatbot's own response. Retain source versions and adjudication notes for ambiguous cases. If two official notices conflict, the case should test escalation or uncertainty handling; it should not force the auditor to invent a policy resolution. Keep unresolved references outside the denominator for settled factual accuracy, while reporting their number and operational significance separately.

Automated evaluators can help prioritise review, but the audit should establish whether their judgements agree with qualified reviewers on consequential cases. A fluent answer can receive a high automated score while reversing a condition. Inspect disagreements and refusals, not only average agreement. The evaluation integrity guide addresses contamination, evaluator validity and exclusions that can make a reported quality score misleading.

Test unsupported confidence and useful abstention

The NIST Generative AI Profile is a voluntary risk-management resource for generative systems. It is useful context for an audit, but it does not certify a chatbot or supply a universal passing score. The practical test here is whether the application distinguishes available evidence from an attractive answer when the evidence is incomplete.

Use cases with missing dates, contradictory notices, an unsupported programme and a question whose answer depends on individual review. Grade the resulting response against the service's role. Abstention is appropriate when it prevents an unsupported determination, but an assistant that refuses every routine question is not delivering the intended service. Report answerability, supported correctness and appropriate referral separately so that defensive refusal cannot inflate an apparent safety result.

A response may contain several claims with different statuses. In the hypothetical example, it might correctly name the examination, incorrectly state that late applications are accepted, and provide a valid contact route. Record the material error explicitly. Averaging those elements into “mostly correct” can conceal the single statement most likely to mislead a candidate. The severity decision should consider the likely consequence, exposure and opportunity for correction.

Review Arabic performance as a service capability

Arabic testing needs more than translating an English benchmark. Include the formal terminology used by the examination body, common user paraphrases, mixed Arabic and English programme names, and unambiguous examples of negation. Where dialect support is advertised, define which varieties and tasks are in scope instead of using “Arabic” as an unlimited capability claim.

Ask bilingual reviewers to identify changes in meaning, certainty and authority. A translation that turns “may be reviewed” into “will be accepted” changes the service promise. Dates, calendar labels and time zones deserve separate checks. If the service displays both Hijri and Gregorian dates, verify the approved conversion source and the intended day boundary; do not assume that numerical similarity establishes equivalence.

Record language at the conversation and turn level. A mixed-language exchange should not automatically be classified as an English success because its final sentence is English. Compare like-for-like tasks across languages, report different case mixes, and inspect escalation quality in each language. If Arabic cases are scarce or unreviewed, state that limitation rather than extending an English result to the Arabic service.

Inspect state, memory and escalation evidence

Conversation memory can be a source of both usefulness and error. Establish which facts are retained only within a session, which are saved to an account and which are passed to staff. Test whether a correction replaces an earlier fact in the information used for the next answer. Also check whether the interface makes retained information visible enough for a user to correct it.

For a handoff, distinguish a transfer request from a message that reached the receiving team. Evidence should connect the request, summary version, routing decision and acknowledgement. A chatbot saying “I have escalated this” is not proof that a case exists. Where the product merely provides contact details, its wording should not imply that a staff member has been notified.

Review summaries for omitted uncertainty and stale information. Staff may rely on the summary without reading the full exchange. In the example, the applicant's corrected examination period must replace the earlier period, while the unresolved qualification question must remain unresolved. Human review is only meaningful when the receiving person has enough information and authority to act; see human oversight audit.

Worked finding: a correct answer becomes a wrong commitment

The following is a hypothetical finding, not a client result. In a controlled test, the assistant correctly explains a foundation examination's registration date. The tester changes to an advanced examination and states that one required document is unavailable. The final response retains the foundation date and says the application can proceed. The approved advanced-examination notice requires staff review of incomplete documentation.

Illustrative finding record with a bounded conclusion

Field	Filled example
Criterion	After a programme correction, material dates must refer to the corrected programme; incomplete documentation must be referred for authorised review.
Observed condition	Two of six deliberately selected programme-switch conversations retained an earlier date; one also made an unsupported commitment.
Evidence	Ordered transcripts, initial session state, application release, approved notices and reviewer adjudication.
Interpretation	The tested release did not consistently apply corrected programme context in these challenge cases.
Unresolved cause	Stale summary state is a hypothesis until state traces and retrieval inputs establish the mechanism.
Limit	The targeted six-case set cannot estimate how often this happens in all user conversations.
Next decision	The service owner decides containment, investigates the context transition and commissions focused retesting.

The two errors should not become a claim that one third of all users receive incorrect dates. Nor should the remaining four cases erase the failure. Report the observed cases, material consequences and scope honestly. The severity assessment guide helps separate consequence from raw failure count, while keeping assumptions visible to the decision-maker.

Separate conversational quality from security assurance

Some conversational tests concern whether the assistant discloses another user's context or treats untrusted content as authority. These connect to security, but a response-quality audit is not automatically a penetration test. Agree the permitted identities, test data, environments and stop conditions before attempting boundary challenges. The current OWASP LLM security guidance can inform threat selection; a checklist reference does not demonstrate that the deployed controls worked.

Use synthetic records for session-isolation checks and retain evidence of the actual application boundary. A refusal in the chat interface does not prove that a connected API denies the same request. Conversely, an inappropriate sentence does not necessarily mean protected data was accessed. Distinguish observed text, attempted access and confirmed disclosure, and refer broader boundary testing to AI security boundary testing.

Reconcile controlled tests with operating evidence

Controlled conversations establish behaviour under known conditions. Operating records show which conditions occurred in service and whether the organisation detected and handled problems. Obtain a defined window of records through an authorised, minimised extraction. Reconcile the extracted conversation count to an independent usage record where available, and explain gaps caused by deletion, sampling, unavailable channels or logging changes. A convenient export is not automatically the complete population.

Compare the case mix with the test plan. If most examination enquiries concern a single popular programme, an overall score may say little about less common programmes. If a chat channel introduced Arabic halfway through the period, compare its actual exposure window rather than treating the whole month as bilingual operation. Record whether abandoned conversations, technical failures and conversations ending in referral were retained. Removing these cases can change the question the reported score answers.

User satisfaction is supporting evidence, not a factual reference. Someone may appreciate a confident but incorrect answer or dislike an appropriate refusal. Review complaints and corrections alongside the underlying transcripts, but do not assume the absence of complaints proves accuracy. A candidate may discover a wrong deadline too late to return to the same channel. The auditor should describe these detection limits without inventing an estimate of unseen harm.

When a production record suggests a defect, reproduce it with synthetic data and the closest available configuration. If the original configuration cannot be reconstructed, separate the observed historical event from the new experiment. A successful current replay does not disprove the earlier event; a failed replay may identify a related mechanism without establishing precisely what happened before. Preserve that distinction in the finding and management response.

Agree an acceptance rule before comparing releases

A release comparison should identify which errors are unacceptable, which measures are informative and which trade-offs require approval. For example, the service owner might require every known programme-switch defect to be corrected while reviewing whether referral frequency has increased on routine questions. This is an illustrative internal decision rule, not a universal threshold. Do not quietly remove difficult cases from the old test set to make the new release appear better.

Use the same adjudicated cases for a controlled comparison, then add a separately identified fresh set to challenge overfitting to the known examples. Keep results by conversation type and language. Where a provider update prevents exact reproduction, disclose that uncertainty and use overlapping evidence where feasible. The release decision should explain both the improvement observed and the remaining conditions under which the assistant must defer to an authorised person.

Make the audit result useful for a release decision

The report should identify the tested release, languages, conversation types, source window and access limitations. Summarise material findings, unresolved reference disagreements and excluded populations before presenting aggregate scores. Explain whether the evidence supports continued use for a limited role, a change to the service promise, temporary restriction or additional investigation. The accountable service owner makes the release decision; an auditor supplies evidence and a bounded conclusion.

Remediation should address the demonstrated mechanism. Adding a broad disclaimer may not fix stale programme context or a failed transfer. Retesting should include the original failing sequence, variations that challenge the same mechanism and unaffected routine conversations. Preserve the old result so improvement is traceable. A new prompt, model or summary process may change both failure behaviour and useful answerability.

Also agree which future changes invalidate the conclusion. A new examination catalogue, language, memory feature or tool can create a different service boundary even if the model name stays the same. Maintain a change record connecting the deployed configuration to the most recent relevant evidence. Avoid describing an old audit as approval of everything subsequently added.

To prepare a scoping discussion, assemble a masked conversation, the intended service role, current source list, release identifiers and an example of a question requiring staff judgement. Discuss the evidence needed for your chatbot audit to define a review that matches the decisions people actually make after reading its answers.