AI Audit for Energy

An AI audit for an energy organisation should establish what a model predicts, which operational decision uses that prediction and what happens when the prediction is late, incomplete or wrong. Good average accuracy does not establish suitability for every operating condition. The evidence must connect the evaluated model to the deployed data, decision rules and authorised human response.

This guide is for Saudi energy and utilities teams responsible for forecasting, operations, data, cybersecurity, risk and internal audit. Its original hypothetical example is a short-term electricity-load forecast used as an advisory input to planning. It does not directly issue dispatch instructions or change equipment settings. The audit examines the forecast and its use without treating a statistical assessment as an engineering safety approval.

Define the forecast and the decision it supports

Specify the forecast target, unit, aggregation boundary, horizon and issue frequency. An hourly average load in megawatts is different from energy consumed over a period, a local asset constraint or a prediction of an exceptional peak. A model can be accurate for its stated target while unsuitable for a downstream decision that assumes another target.

Document when the forecast becomes available and how long it remains useful. A more accurate forecast arriving after the planning decision may offer less operational value than a timely alternative. Trace issue time, delivery time, review time and the decision made. These are separate events, and a successful batch job does not prove that a planner received a usable result.

Identify the decision authority. In this example, a planner considers the forecast alongside other approved information and procedures. Record the model's advisory status in the interface, integration and working instructions. If an automated process consumes the same output, include that process explicitly; a dashboard label alone cannot constrain a machine-to-machine action.

Determine which operational and cybersecurity boundaries apply

NCA's Operational Technology Cybersecurity Controls, OTCC-1:2022, define an applicability scope concerning industrial control systems in specified critical facilities. Their scope is not equivalent to every AI tool used by an energy company. The organisation's responsible functions should map the actual facility, systems and connections to applicable requirements and document the rationale.

A forecasting application outside an operational technology environment may still exchange information with it. Examine that boundary, including permitted data directions, service identities, administrative access and dependencies. The audit should identify whether it is reviewing a statistical service, an operational integration or both. Use NCA-related AI audit mapping where the engagement includes relevant cybersecurity criteria.

Agree testing conditions with operations and the appropriate technical owners. Historical replay, isolated fixtures and read-only evidence can answer many audit questions. Live changes to operational settings require a separately authorised objective and operating arrangements. Do not expand a forecast audit into an uncontrolled experiment on physical equipment merely to demonstrate that an integration is connected.

Build an evidence map from measurement to planning use

The evidence chain starts before model inference. Obtain definitions for metered quantities, quality flags, estimated readings, corrections and aggregation rules. Determine whether missing measurements are omitted, filled or carried forward. Each choice changes the meaning of the training and evaluation data, and the audit should examine whether the deployed process follows the documented treatment.

Original evidence map for an advisory load forecast

Boundary	Evidence to inspect	Question answered
Measurements	Source definitions, timestamps, quality flags and corrections	What quantity was known, and when?
Input preparation	Aggregation, missing-data rules and feature versions	Does the input preserve the intended target and timing?
Forecast issue	Model identifier, input snapshot, horizon and issue record	Which prediction was actually available for the decision?
Planner use	Delivered view, acknowledgement and decision record	Was the output considered within its authorised purpose?
Fallback	Activation rule, alternate procedure and recovery evidence	What happens when the forecast cannot be used?

Reconcile the map against a real scoped release and an authorised sample of decisions. A diagram may describe the intended architecture while an export or manual spreadsheet creates another route. Identify these routes before selecting tests. Otherwise, a careful model review can miss the version that planners actually use.

Reconstruct information available at forecast issue time

A historical evaluation can accidentally give the model information that would not have existed when the forecast was issued. Revised measurements, updated weather forecasts and later operational records can all create this problem. Distinguish the time an event occurred from the time the forecasting service could access its record.

For a selected forecast, reconstruct the input snapshot available at its issue time. Check publication, ingestion and correction times where relevant. If the organisation retains only the latest corrected dataset, the auditor may be unable to reproduce the historical information boundary. Report that limitation instead of presenting a retrospective calculation as an exact replay of live performance.

Use the same discipline for the comparison method. A baseline evaluated with different information or a different target is not a fair comparator. State whether the exercise tests the model's mathematical behaviour, the full deployed pipeline or a reconstructed approximation. Evaluation-integrity auditing addresses the broader evidence needed to support those distinctions.

Design chronological and operating-condition comparisons

Choose evaluation periods that match the intended use and known operating variation. Explain the separation between training, tuning and final evaluation periods. Repeatedly adjusting the model against a supposedly final test period weakens its independence, even when the records were originally held out. Keep a record of which evidence informed each change.

Define relevant operating conditions before interpreting results. These may include high-load periods, unusual weather, planned maintenance, changes in demand composition or incomplete measurements. The appropriate categories depend on the system. Avoid selecting only the conditions that make the model look favourable, and disclose where the sample is too limited for a dependable comparison.

Do not treat adjacent hourly errors as automatically independent observations. Shared weather, operating conditions and source defects can affect many hours together. Statistical uncertainty and sampling plans should account for the relevant dependence. A large row count can still represent a narrow range of underlying conditions.

Worked example: better average error can hide worse peak behaviour

Assume an illustrative evaluation of 20 hourly intervals, with load expressed in megawatts. The AI forecast has a total absolute error of 80 MW across those intervals, while an approved comparison method totals 100 MW. Their mean absolute errors are therefore 4 MW and 5 MW. On this descriptive measure, the AI error is 20% lower.

Four intervals meet a peak-condition definition established before the comparison. The AI absolute errors in those intervals are 12, 16, 14 and 18 MW, totalling 60 MW and averaging 15 MW. The comparison errors are 8, 9, 8 and 7 MW, totalling 32 MW and averaging 8 MW. The model is worse on this selected operating condition despite its better overall average.

The arithmetic is consistent: the remaining 16 intervals contribute 20 MW of absolute error for the AI and 68 MW for the comparison. The example shows why the decision matters. If peak conditions drive an important planning requirement, management should not approve broader use solely because the overall mean improves. It should examine the condition-specific evidence and the authorised response.

These 20 hypothetical intervals do not establish production performance or a statistical confidence level. Absolute error also hides direction. Underprediction and overprediction may have different consequences in the actual workflow, so analyse signed errors and decision effects where appropriate. Any acceptance tolerance should be agreed by the responsible owners, not invented as a universal energy-sector threshold.

Check whether aggregation conceals a different planning problem

A forecast can match a combined load while misrepresenting its components. In a separate hypothetical illustration, two planning areas each have an actual load of 100 MW. The model predicts 110 MW for one and 90 MW for the other. The combined prediction and actual load both equal 200 MW, but each area's absolute error is 10 MW. The correct total does not establish accurate allocation between areas.

Ask which level the planning process uses. If the approved purpose is only the combined total, component error may be outside that performance claim, although its relevance should be considered. If planners allocate resources using the area forecasts, reporting only total accuracy omits evidence they need. State the target level in the evaluation and report rather than allowing the reader to infer broader coverage.

Inspect how the organisation reconciles forecasts across levels. A reconciliation rule can change the component values after the model produces them. The audit should therefore identify whether reported performance describes the raw model outputs or the values delivered to planners. Trace a controlled case through both stages and retain the transformation rule. A good score for the raw model does not automatically describe the final planning input.

The same issue arises across time horizons. An updated forecast close to the operating period may be more accurate than the earlier issue that informed a planning commitment. Keep issue identifiers and horizons distinct when matching predictions to actuals. Otherwise, replacing earlier values with the latest forecast can rewrite historical performance without changing a single actual measurement. Report the forecast actually available for the relevant decision. This is an evidence test, not a conclusion about equipment capacity or the engineering adequacy of a particular operating plan.

Examine prediction intervals and uncertainty claims

If the system displays an uncertainty range, ask what it represents and how it was evaluated. A nominal coverage label does not establish realised coverage for every condition or horizon. Compare the stated interpretation with observed results over a justified dataset and inspect whether the range becomes uninformatively wide or systematically misses relevant conditions.

Check how planners see uncertainty. A central estimate copied into a spreadsheet may lose the range, issue time and limitations displayed in the original interface. Audit the information that reaches the decision point. If the system does not provide a meaningful uncertainty estimate, the report should say so without manufacturing one from a confidence-looking interface field.

Distinguish uncertainty about demand from uncertainty caused by missing or unreliable input data. A range around a forecast may not account for a failed measurement feed. The response to input failure may therefore require a separate status and fallback decision rather than a wider interval that appears statistically sophisticated but has no validated interpretation.

Test stale forecasts and fallback operation

Use an isolated, authorised scenario in which a scheduled input is delayed and the latest forecast exceeds its agreed useful age. Check whether the delivered view exposes the issue time, data condition and fallback status. A green service-health indicator should not cause an old forecast to appear current merely because the application itself is available.

1. Record the approved freshness rule and the role authorised to act on a breach.
2. Introduce a controlled delayed-input condition without affecting live operations.
3. Observe the forecast status, downstream delivery and planner-facing information.
4. Verify that the approved alternative procedure is available and its use is recorded.
5. Restore the input and inspect how a new forecast becomes eligible for use.
6. Preserve evidence of any review needed before returning to the normal process.

The existence of a fallback document does not prove that the alternative data, permissions or staff capacity are available. Examine the complete path for the scoped scenario. Equally, a successful exercise proves only what was tested; it does not establish resilience to every combination of equipment, communication and staffing disruption.

Review human challenge and override evidence

Ask how a planner can question a forecast and what evidence accompanies an override. The objective is not to maximise or minimise override frequency. A low rate may reflect excellent predictions, poor challenge conditions or routine acceptance. A high rate may reflect justified local information, confusing output or a model that does not fit the task.

Trace selected decisions to the information available at the time and the documented rationale. Later actual demand should not be used to portray every successful override as foreseeable. Separate the quality of the decision process from the eventual outcome. Where the record is incomplete, identify the evidence gap rather than assigning an unsupported cause.

Review Arabic and English operating descriptions for consistent meaning. Terms for forecast, approved plan and execution instruction should remain distinct. An advisory recommendation should not acquire the appearance of a mandatory operating order through translation, copied templates or interface wording. Human-oversight auditing examines whether challenge is workable in practice.

Trace changes in assets, boundaries and source systems

A new meter, changed aggregation boundary, asset addition or source-system correction can alter the forecast target. Compare the deployed data definition with the definition used during evaluation. An apparently stable model identifier does not establish an unchanged system if its inputs now represent a different set of loads.

Inspect change records for the components that influence the result: input preparation, model, calendar features, external data, interface and downstream consumer. Decide which changes require targeted evaluation or a broader review. The audit should assess that decision against the organisation's criteria and the actual consequence, rather than demand identical testing for every cosmetic change.

Where a supplier controls an input or model service, obtain evidence of relevant version changes and limitations. Inability to inspect proprietary internals does not prevent every useful test, but it limits some conclusions. Specify the available evidence and unresolved dependencies, using vendor-controlled-system audit methods where access is restricted.

Handle operational and customer information proportionately

Energy datasets can include commercially sensitive operational information and, depending on granularity and linkage, information about identifiable people. Determine the actual data categories and authorised processing conditions. Aggregation is not automatically proof that identification is impossible, particularly when a small group or a single customer dominates a time series.

For the audit, prefer authorised synthetic or appropriately masked fixtures when they answer the question. Restrict exports, define evidence recipients and preserve necessary context without copying an entire operational dataset by default. Keep the audit's evidence-access arrangements separate from the production system's normal access; both require appropriate handling.

Turn a forecast finding into a bounded management decision

An illustrative finding could state that approval relied on overall mean absolute error while the pre-defined peak subset performed worse than the comparison method and no condition-specific acceptance decision was recorded. Attach the population, calculations, model version and decision evidence. Do not claim that the model caused an operational incident unless the evidence establishes that connection.

The response might involve limiting the approved use, improving data, revising the evaluation or defining a more suitable planning procedure. Management should choose the action based on consequence and feasibility. Audit retesting should verify the agreed correction and relevant adjacent conditions, while preserving any continuing limitation in the report.

Report model performance, delivery reliability, human use and operational authority separately when each has different evidence. An effective forecasting component can coexist with a poor delivery process; a well-controlled workflow can still rely on an inadequately evaluated model. A combined verdict should explain these dependencies rather than average them into an opaque score.

Prepare a focused energy AI audit scope

Bring the target definition, horizon, issue schedule, data lineage, evaluation design, comparison method, operational-use map, fallback arrangements and recent changes. Include a masked forecast-to-decision example and the relevant facility or cybersecurity applicability assessment. State which historical input versions are available and which cannot be reconstructed.

Trusted's energy and utilities page offers commercial reading on possible sector applications. The audit should use the organisation's actual operating evidence and applicable authoritative criteria to reach its conclusions.

Request an audit scoping discussion for an energy forecasting workflow through OpenForAI's AI auditing service. A useful engagement makes the forecast's purpose, information boundary, operating limitations and decision authority explicit, so the responsible teams can act on the findings.