Predictive AI Readiness Checks

A predictive model is ready to be investigated when the organisation can describe the future event, reproduce what was known before that event, observe outcomes reliably, and identify a useful response to the prediction. A large historical table does not establish any of those conditions. Before comparing algorithms, test whether the proposed prediction is a well-defined business problem with evidence that can support an honest evaluation.

This guide is for operations sponsors, data owners and analytics teams considering a forecast, propensity estimate or prioritisation model. It concentrates on prerequisites. It does not determine a model's eventual accuracy, authorise automated decisions or replace a controlled evaluation. Enterprise data and AI readiness assessment places these checks within the users, permissions, operating ownership and boundaries of the intended use case.

Write the prediction as a decision contract

Describe who receives the prediction, at what moment, about which entity, for what future period and with what permitted response. “Predict difficult invoices” is ambiguous. “At the opening of a newly logged business invoice dispute, estimate whether it will require a defined operational escalation within the next 30 calendar days, to help a supervisor allocate review capacity” is testable. It still needs an exact definition of escalation and an explanation of why additional review would help.

The unit may be an invoice, dispute episode, customer or invoice line. These are not interchangeable. A reopened dispute should not silently become a new independent outcome if it is a continuation of the same case. Define whether multiple invoices in one dispute produce one decision or several. Record the decision time, observation window, outcome horizon, permitted users and excluded cases in the contract before anyone extracts a training dataset.

For the illustrative workflow used here, the prediction supports a human operations queue. It does not decide liability, credit terms, collection action or a customer's legal rights. That boundary changes the evidence needed and the consequences of errors. A future proposal to automate consequential actions would require a new assessment, even if it reused the same model.

Make the outcome observable and meaningful

An outcome definition should identify the event, its authoritative record and the rules for unusual cases. An escalation might mean assignment to a specialist dispute-resolution team with a recorded reason. A supervisor opening a record, a late payment or an angry message is not automatically that event. Avoid constructing a label from whatever column happens to be available without checking the operational meaning with the process owner.

Distinguish the event from its administrative recording. A case escalated on 15 May might be entered into the reporting system on 18 May. Both times matter. If escalation records are occasionally deleted or reclassified, decide whether the historical label should reflect the event as understood at a fixed maturity point or a later adjudicated account. Neither definition is inherently universal; the choice must match the question the model will answer.

“Not escalated” is an observed outcome only after the full horizon and any justified recording delay have elapsed with adequate follow-up. An open case halfway through its horizon is not a negative example. A missing case outcome is not a negative either. Maintain explicit states for positive, observed negative, pending, missing and excluded. These states prevent the data pipeline from turning uncertainty into reassuring accuracy.

Worked cohort: why the denominator changes the problem

The following figures are hypothetical and demonstrate readiness reasoning, not a measured business result. An extract contains 2,400 newly opened dispute episodes. At the assessment cutoff, 2,160 have completed the 30-day horizon and the agreed recording allowance. Another 240 are still pending. Among the mature episodes, 96 meet the documented escalation definition.

Illustrative outcome reconciliation at the assessment cutoff

Cohort component	Count	Treatment
Mature escalation	96	Observed positive outcome
Mature without escalation	2,064	Observed negative after adequate follow-up
Outcome window still open	240	Pending; excluded from this mature-outcome rate
Total extracted episodes	2,400	Reconciles to the extraction population

The observed escalation rate in the mature cohort is 96 divided by 2,160, or approximately 4.44%. Dividing 96 by all 2,400 gives 4%, but that denominator incorrectly treats pending episodes as if their outcomes were already known. The mature-cohort rate is descriptive of that cohort; it is not automatically the expected rate in future disputes or proof that the missing and pending groups behave similarly.

Check whether maturity exclusions select a different business mix. If the most recent cases come from a new subsidiary, omitting them also omits that subsidiary's recent process. If long-running cases lack reliable closure evidence, restricting the data to easily resolved cases may remove precisely the situations the model is intended to help. A readiness report should describe that coverage limit rather than hide it behind a complete-looking percentage.

Prove that candidate inputs existed at prediction time

For each proposed input, record its business event time, time it became available to the prediction workflow, source version and any later changes. The final number of follow-up calls, eventual settlement amount or escalation-team identifier may explain historical outcomes extremely well because they arise after the proposed prediction. They cannot be used merely because they appear on the same current database row.

The scikit-learn guidance on data leakage explains why information unavailable at prediction time can produce overly optimistic evaluation. It also recommends separating data before fitting preprocessing. These safeguards do not, by themselves, reconstruct historical business availability; that requires explicit source and timestamp evidence.

Create a small reconstruction exercise before attempting a large model. Select authorised, masked dispute episodes and ask an independent reviewer to recreate the opening-time view from preserved records. Compare that view with the proposed analytic row. A field marked “current status” may contain a later correction even if the record's original creation date is old. Time-based separation in predictive AI explains the temporal mechanics in more detail.

Inspect labels as a process measurement

An escalation label often measures organisational behaviour as well as underlying case difficulty. One team may escalate early because it has spare specialist capacity; another may retain comparable cases because that capacity is scarce. A change in authority limits, routing rules or recording practice can alter the label without a change in customer disputes. That does not make prediction impossible, but it changes what is being predicted.

Build a label-review sample around plausible disagreements: cases reassigned without escalation, reopened episodes, merged disputes, missing reason codes and contradictory timestamps. Have reviewers apply the written rule independently before reconciling differences. Record agreement, unresolved interpretations and the source needed to settle them. Do not use reviewer consensus to rewrite an inconvenient history without maintaining the original and the correction basis.

Where a label is a proxy, name it honestly. “Recorded operational escalation within 30 days” is narrower than “true dispute severity.” The latter might require evidence that the organisation does not collect. A model cannot remove this measurement gap simply by learning more complex patterns. If the proxy would lead the sponsor to a different action from the intended outcome, reconsider the problem before funding development.

Check coverage across the proposed users and operating periods

Compare the historical population with the proposed deployment population. Relevant differences may include business unit, dispute reason, invoice type, language, channel, season, customer tenure and process version. Select dimensions because they can change the workflow or measurement, not because every available demographic field deserves to become a feature. Sensitive or unnecessary information should not be included without a defined, permitted purpose.

A dataset with many rows can still have little useful support for a rare but important route. Repeated monthly disputes from a few customers may create apparent volume without representing many independent situations. If related episodes share documents or negotiated terms, account for those relationships when designing evaluation. The appropriate grouping depends on whether the intended use is prediction for existing relationships, new relationships or both.

Separate a representative sample used to estimate data defects from a deliberately difficult sample used to test definitions. Combining them produces a rate with an unclear meaning. Representative-sample data-quality measurement provides the sampling discipline. For predictive readiness, additionally report where there are insufficient mature outcomes to make a credible evaluation plan, even when individual records are technically complete.

Choose a baseline and a capacity-aware evaluation question

Readiness includes knowing what a model must improve. Compare the proposed approach with the existing queue, a simple documented rule and, where appropriate, a simple statistical baseline. A model that predicts no escalations would appear correct on most records in the illustrative cohort, but it would identify none of the 96 observed escalations. Overall accuracy would therefore be a poor sole measure of usefulness.

If supervisors can review only 20 cases per working day, an evaluation should examine the cases the proposed policy would place in those 20 slots. Record the review capacity, what happens to cases below the cutoff and whether additional review changes outcomes. Measures such as precision among selected cases and recall of eventual escalations describe different consequences. Neither establishes financial benefit without evidence about the review action and its cost.

Set a comparison protocol before inspecting final results. Specify the future-like evaluation period, eligible cohort, baseline, metrics, uncertainty reporting and treatment of tied scores or missing inputs. A business owner should define what improvement would justify the next bounded experiment. This is a proposed internal acceptance condition, not a universal accuracy threshold or a guarantee that a profitable model exists.

Separate forecasting quality from the effect of intervention

Historical outcomes may already reflect staff intervention. Cases flagged by an old rule might have received extra attention and avoided escalation. A new model trained on the resulting labels can appear to assign low risk to cases that were high risk before that intervention. Record what actions occurred, when they occurred and why they were taken; do not treat the observed history as an untouched natural experiment.

A predictive evaluation asks whether estimates correspond to later outcomes under a defined process. An intervention evaluation asks whether acting on those estimates improves the process. They require different evidence. The readiness decision can authorise a prospective, human-reviewed learning exercise with agreed safeguards while leaving any claim of causal improvement unresolved. It should not describe an association in a historical table as proof that a proposed action will work.

Also consider changed incentives. If a future performance target discourages staff from recording escalation, apparent model performance could improve while record quality deteriorates. Agree who owns outcome recording and how material process changes will be communicated. The goal at this stage is an interpretable evaluation opportunity, not a permanent monitoring implementation.

Use a predictive-readiness evidence matrix

Minimum evidence for the next predictive experiment

Readiness question	Evidence to request	Reason to restrict or pause
Is the future event defined?	Outcome rule, source owner and adjudicated edge cases	Teams label the same event differently
Can inputs be reconstructed?	Opening-time snapshots or supported availability history	Current rows contain untraceable later changes
Are outcomes mature?	Horizon, recording allowance and cohort reconciliation	Pending cases are coded as negatives
Does history cover the proposed boundary?	Period, entity, language and process coverage	A target population has no suitable evidence
Can usefulness be tested?	Baseline, review capacity and predeclared evaluation plan	No permitted action follows a prediction
Is evidence use authorised?	Purpose, access, retention and responsible approvals	Required records cannot be used for the proposed test

The matrix supports a decision; it is not a scoring formula. One missing mandatory prerequisite can prevent the proposed experiment even when other rows look strong. Conversely, a gap affecting one business unit may permit a narrower study if the exclusion is real and documented. Readiness gates and scores explains why those decisions should remain separate.

Investigate missing inputs before choosing a replacement rule

Missingness has a business explanation that may change over time. An absent dispute-reason field could mean that the customer has not replied, that a branch uses a different application, or that an export omitted a populated source value. Those situations have different implications. Record the reason where it can be established and distinguish an unknown value from a genuine “not applicable” state. Do not replace every blank with a single category and assume the problem is solved.

Inspect the information that will actually arrive at the proposed prediction moment. A field may be nearly complete in closed cases but largely absent when a dispute first opens. That is a readiness mismatch even if a conventional completeness report looks excellent. Review how the workflow should respond when an essential input is missing: wait, request clarification, use a defined fallback or decline to produce a score. The assessment should establish that one of those responses is operationally acceptable.

Any statistical replacement or encoding method belongs inside the evaluation design and must be learned without access to the held-out evidence. More fundamentally, test whether the missingness pattern is stable enough to support the proposed study. A new mandatory field introduced last month does not provide years of historical information simply because all current records now display it. Preserve the implementation date and avoid backfilling invented opening-time values.

Arrange a decision meeting around unresolved evidence

Bring the process owner, data custodian, evaluation lead and intended supervisor together with the actual evidence matrix. Ask each to resolve a different question. The process owner accepts the target definition; the custodian explains reconstruction limits; the evaluator states what the available outcomes can support; the supervisor confirms the permitted response and practical capacity. A presentation by the modelling team alone cannot establish all four responsibilities.

Walk through one positive episode, one observed negative and one pending episode from the masked sample. For each, reconstruct the prediction moment and trace the label to its authoritative record. Then introduce a counterexample, such as a case reassigned without formal escalation. If the group cannot agree on the expected treatment, record the uncertainty and its scope. Do not settle it by choosing whichever rule produces a stronger historical metric.

Decide whether more evidence is worth collecting before modelling. A narrow uncertainty about one input may be resolved by a small source-history exercise. A target that does not correspond to any consistent operational event may require process redesign first. These are different investments. Document the proposed activity, owner and stopping condition so that a feasibility study does not become an open-ended effort to rescue an ill-defined prediction.

Finally, distinguish permission to inspect data from permission to use it in a model or expose predictions to users. Approval for this readiness exercise may cover only a masked extract in a restricted workspace. The next experiment needs its own stated access and use boundary. Carry that condition into the decision record rather than letting access obtained for assessment become an assumed production entitlement.

Define the next step and the evidence that will reopen it

A useful conclusion might be: “Ready for a retrospective feasibility study on mature disputes from the two named business units under the current escalation definition; not assessed for the newly acquired unit or automated action.” It should name the data snapshot, unresolved limitations, evaluation owner and conditions for expanding scope. Avoid a company-wide “predictive AI ready” label that outlives the evidence.

If the conclusion is not ready, turn the blockers into specific work. Replace “improve data” with “preserve opening-time status history for the approved dispute population and demonstrate reconstruction on the agreed sample.” Replace “get more labels” with “resolve the meaning of reassignment versus escalation and reconcile the affected outcome periods.” A data remediation backlog makes these dependencies, owners and closure evidence visible.

For an assessment discussion, prepare a masked decision contract, source inventory, outcome-state counts and examples of ambiguous records. Do not send customer disputes or confidential invoices through an unapproved channel. Discuss predictive AI readiness with a clearly defined next decision, so the evidence work tests whether an honest, useful experiment is possible before the organisation commits to a model or an automation promise.