Evaluate with the information available then
Time-Based Separation in Predictive AI
Design as-of evidence, mature-outcome windows and chronological evaluation that exclude late corrections and future information.
A historical prediction test should give the model the information that the real workflow could have known at the decision moment. Sorting rows by date is only one part of that requirement. Later corrections, delayed feeds, future outcomes and preprocessing learned from the test period can all cross the time boundary while a dataset still looks chronological.
Time-based separation is therefore a readiness question about evidence, not just a choice of validation function. This guide shows how to define an as-of record, separate learning and evaluation periods, and explain what a retrospective result can establish. For the wider decision about whether outcomes and actions are meaningful, use predictive AI readiness checks. For the full business and access boundary, start with enterprise data and AI readiness assessment.
Fix the moment when the prediction must be useful
Consider a hypothetical restaurant group exploring a forecast of reservation no-shows. A supervisor would receive an estimate at 12:00 local time on the day before service, solely to plan staff-led confirmation calls. The example uses a documented UTC+03:00 clock. The prediction does not cancel bookings, change deposits or deny service. Those actions would require separate decisions and evidence.
The eligible population consists of reservations already present in the authorised booking feed at that noon cutoff. Reservations made afterwards are outside this particular forecast. The service date alone is not the decision time: information entered on the morning of service would be too late for the previous day's prediction. Define whether reservations that are already cancelled at noon are excluded, and how rescheduling creates or replaces an eligible episode.
Write the horizon and outcome rule alongside that cutoff. For the worked protocol below, the relevant service is at 20:00 the following day, and a no-show outcome is finalised 48 hours after that scheduled service. This is an illustrative business convention, not a recommendation for every restaurant. The assessment must confirm that the organisation actually preserves the evidence needed to apply its chosen convention.
Distinguish event time, availability time and outcome maturity
Event time describes when something happened in the business. Availability time describes when the prediction workflow could access that information in usable form. A payment might occur at 11:30 but reach the approved booking feed at 12:05. At a noon prediction, the payment is not available through that feed. Using the earlier payment time alone gives the historical model information that the intended live workflow would not have had.
A correction can have an old effective date and a new availability time. A seating-capacity correction may apply to the morning's configuration but be approved and received in the afternoon. Reconstructing the noon state from the latest corrected table would quietly import future knowledge. Maintain versions, arrival records or snapshots capable of answering both “what period does this describe?” and “when could this workflow know it?”
Outcome maturity is different again. A booking can pass its scheduled service time before the final attended, cancelled or no-show classification is reliable. Reconciliation may resolve late check-ins or duplicate reservations. Specify the delay required for the intended label and show when that label became available for model training. Do not confuse an outcome's event date with the date when it was safe to learn from the outcome.
A small as-of ledger exposes large errors
| Record | Business or effective time | Available to workflow | Allowed at noon? |
|---|---|---|---|
| Reservation created | 10 May, 11:00 | 10 May, 11:01 | Yes, using the version available then |
| Deposit payment | 10 May, 11:30 | 10 May, 12:05 | No; the feed had not delivered it |
| Corrected seating capacity | 10 May, 09:00 effective time | 10 May, 14:00 | No; retain the earlier known version |
| Actual attendance | 11 May service at 20:00 | After service and reconciliation | No; it is a future outcome |
For every candidate input, the ledger should name the source, join key, version rule and evidence supporting availability. An analyst's recollection that a feed is “normally fast” is not enough to establish a timestamp for every historical record. Where arrival evidence is unavailable, state the uncertainty. A conservative delay assumption may support a limited sensitivity exercise, but it should not be presented as observed historical availability.
Use a few intentionally difficult cases: a late-arriving payment, an amended booking, an outcome reclassification and a reservation crossing midnight between systems. If the team cannot reconstruct these cases, a large backtest may amplify the same ambiguity. The readiness finding should identify the unsupported feature or population and whether it can be removed without destroying the proposed question.
Choose calendar windows before looking at final results
Also check whether source clocks were synchronised and whether an arrival timestamp records receipt, processing completion or export. These are distinct observations; use the one that supports the actual prediction boundary.
A forward-looking evaluation uses earlier information to develop a model and later information to examine its behaviour. Reserve separate development, validation and final test purposes. The exact dates depend on volume, seasonality, label delay and the deployment cadence. Do not select dates because they create the most attractive result. A sponsor should be able to see the rationale before the final test is opened.
The following illustrative windows use only the next-day 20:00 reservations defined above. Forecasts issued through 28 March concern service no later than 29 March at 20:00; their 48-hour outcome window ends by 31 March at 20:00. They can therefore support a 1 April, 00:00 fit only if the labels are actually available by that fit time. The calendar arithmetic establishes eligibility, not evidence that ingestion occurred on time.
| Purpose | Forecast issue dates | Required learning boundary |
|---|---|---|
| Initial training | 1 January–28 March | Use only inputs available at each forecast and labels available before 1 April, 00:00 |
| Validation for choices | 1–27 April | Latest defined outcome matures by 30 April, 20:00; verify availability before choices freeze |
| Freeze choices and refit | 1 May, 00:00 | Use eligible prior evidence only; record the final specification |
| Untouched final test | 1–28 May | Evaluate after outcomes mature and arrive; latest defined maturity is 31 May, 20:00 |
The omitted end-of-month forecast dates are deliberate in this simplified protocol because their labels would not yet be ready at the next stated fit boundary. They are not proof that all studies require a fixed three-day exclusion. Different service times, horizons and recording delays require different boundaries. Document excluded dates and their effect on coverage rather than quietly discarding inconvenient cases.
Keep feature construction inside the time boundary
A “previous no-show rate” must use only prior reservations whose outcomes were available at the current forecast time. Counting a customer's entire history, including later bookings, leaks the future into an apparently historical feature. Rolling averages, customer segments, venue rankings and aggregate language categories can have the same problem. Write the lookback period, eligible outcomes and update delay for each aggregate.
Vocabulary selection, scaling, missing-value replacement and feature selection can also learn from the future. The scikit-learn leakage guidance recommends fitting such learned transformations on training data rather than the test set. Applying a pipeline correctly helps with this boundary, but it does not repair source tables that already contain later business information.
Freeze the specification used for the final test: feature definitions, input sources, missing-data behaviour, model choices and decision policy. If a test result prompts a change, that test has become development evidence. It may still be useful, but it is no longer an untouched estimate for the revised approach. Reserve new evaluation evidence or describe the resulting uncertainty honestly.
Check what a point-in-time join actually guarantees
A join described as “point in time” needs a precise contract. Does it select the latest event before the decision, or the latest version actually available before the decision? Does its time-to-live limit refer to the event or to processing? How are ties, corrected records, missing timestamps and deleted values handled? The product name or API label does not answer these business questions.
The current Feast point-in-time documentation distinguishes event-time matching from created-time filtering. Its default created timestamp is used for deduplication; enabling filter_by_created_timestamp adds an availability-related cutoff where supported. This still depends on the timestamp representing real availability, and support varies by offline store. Verify the selected configuration rather than assuming all historical joins exclude late corrections.
This is a general readiness lesson even when Feast is not used. Ask the data team to demonstrate the ledger's late-arrival example through the actual extraction process. The expected result should be declared before the query runs. Save the query or transformation version, masked input and output, and explanation of any exception. Catalogue and lineage evidence helps connect the result to the source and derivative versions.
Handle repeated entities and amended reservations deliberately
Time separation does not automatically remove every form of dependence. The same customer may appear in training and test periods. That can be appropriate if the intended product predicts future bookings for existing customers using legitimately available history. It is a different task from predicting for entirely new customers. Report both populations separately if both matter to the planned use.
More serious leakage can occur when copies of one booking episode appear on both sides of a split. A reservation moved from one venue to another may retain a related identifier, deposit record and notes. Define a stable episode relationship so that an amendment is not treated as an independent historical example with privileged knowledge of the original outcome. Keep the raw relationships available for review.
Ask what independence the evaluation needs to support its claim. A venue expansion question may require evaluation on venues with different operating conditions, while a month-ahead refresh question may focus on existing venues over time. No single split answers all these questions. If the data cannot support the desired claim, narrow the claim instead of decorating one backtest with several broad interpretations.
Use gaps and rolling tests for a stated reason
A gap between training and evaluation can prevent overlapping outcome windows or allow labels to become available. It is not a magic protection against leakage. Determine the gap from the prediction horizon, label availability and any information overlap. A 30-row gap might span minutes in a busy restaurant and days in a quiet one. Calendar duration and row count should not be treated as equivalent.
The TimeSeriesSplit documentation describes ordered folds and a sample-count gap, and notes the role of equally spaced samples in comparable fold durations. Irregular booking events therefore need an explicitly justified calendar design if the evaluation question is framed in days or months. Do not assume the default splitter matches the service schedule.
Rolling evaluation can show how the proposed learning process would behave at several historical decision dates. At each date, repeat the availability and outcome-maturity rules rather than reusing a fully corrected dataset. Record which choices were fixed and which could legitimately be relearned then. A rolling test that chooses its best configuration using all later periods still contains future-informed selection.
Investigate time-related errors before interpreting performance
Inspect missing inputs, delayed feeds and outcome corrections by period. A sudden performance change may reflect a new booking system, a revised no-show definition or a change in confirmation-call practice. These explanations matter before attributing the result to model deterioration or improvement. Keep a timeline of material source and process changes alongside the evaluation periods.
Review timezone and calendar conversion at system boundaries. Store enough information to distinguish a timestamp's offset, its original meaning and the local service date used for business decisions. A date-only export may be insufficient for a noon cutoff. If exact times were never retained, avoid manufacturing midnight timestamps and treating them as evidence that the information was available before the decision.
Late outcomes require a revision policy. State when an evaluation snapshot is considered mature, what happens if a later correction arrives, and whether previously reported metrics will be restated. Preserve the original snapshot and the correction reason. This protects interpretability: a changed number should be traceable to changed evidence or method, not appear as an unexplained improvement.
Apply two different clocks to historical learning
A training example has an input clock and a learning clock. Its inputs must reflect what was available at that example's prediction moment. Its label is necessarily learned later, because the prediction concerns a future outcome. The label may be used only once it is sufficiently mature and available before the model's training cutoff. Requiring the future label to exist at the original prediction time would make supervised prediction impossible; allowing future inputs would make evaluation misleading.
Suppose a March reservation's attendance classification is corrected in June. A model actually fitted on 1 April could not have learned from that June correction. A later research dataset may use the corrected label to investigate a different question, but it must not claim to reproduce the April training state without qualification. Keep these views separate: the evidence available to a historical learning process and the best later understanding of the event.
The same distinction matters when comparing repeated fits. Each historical fit needs a record of which labels were available then, not merely a list of which reservations occurred earlier. If this history is missing, the report can describe a retrospective study using subsequently adjudicated outcomes. That is a narrower claim than a faithful simulation of what the organisation could have trained and operated at each earlier date.
Rehearse the boundary with expected answers
Before commissioning the full evaluation, prepare a compact set of timestamp cases with expected inclusion decisions. Include information arriving one minute before the cutoff, exactly at the cutoff and one minute afterwards. State whether the boundary is inclusive and how transaction completion is established. Two systems may display the same second while one update was not committed or delivered until later; timestamp formatting alone cannot settle ordering.
For each case, request the selected value and the reason it was selected, not just a successful query status. A correction case should show the earlier eligible version; a late first arrival should show the defined missing-input response. A future attendance record should be absent from inputs but eligible as a label only at the appropriate learning cutoff. Review these results with someone who understands both the booking process and the data representation.
Preserve the exercise as a readiness artifact with source versions, expected outcomes and observed deviations. It is a focused demonstration of the information boundary, not a comprehensive software test suite or an estimate of error prevalence. If a case fails, identify whether the defect is in the source history, timestamp meaning, extraction rule or stated expectation. The remedy and responsible owner depend on that distinction.
Turn temporal uncertainty into an explicit readiness conclusion
A strong handover contains the prediction contract, as-of ledger, outcome-availability rule, split calendar, feature definitions, frozen evaluation specification and unresolved source-history limitations. Include examples showing that a late input and a future outcome were excluded correctly. These artifacts make the proposed study reviewable without pretending that a retrospective test proves future operation will be identical.
If opening-time versions are unavailable for one optional feature, removing it may permit a narrower experiment. If the central target cannot be dated or finalised reliably, the appropriate next step may be evidence collection rather than model development. Remediation backlog design turns these different issues into owned tasks with demonstrable closure conditions. Readiness versus a proof of concept explains why a technically impressive prototype cannot fill missing historical evidence.
Prepare a masked timeline of a few representative and difficult cases before discussing temporal readiness. The useful question is whether the organisation can recreate the information boundary of the intended decision and evaluate on outcomes that were genuinely observed. That question should be settled before historical performance is used to justify a rollout or a claim of predictable business value.
FAQ
Your questions, answered.
Is sorting a dataset by date enough to prevent predictive leakage?
No. A chronologically ordered row can still contain later corrections, future aggregates or an outcome that was unavailable when the model would have been trained. Define the prediction time and reconstruct each feature’s availability, then separate learning and evaluation purposes. Review related episodes and preprocessing as well. A date sort is a useful operation within a temporal protocol, not evidence that the protocol is valid.
What is the difference between event time and availability time?
Event time is when the business event happened or became effective. Availability time is when the intended prediction workflow could use the information. A payment occurring before noon but arriving in the approved feed after noon cannot support a noon prediction through that feed. Preserve both meanings where they differ, including the treatment of amendments, so a historical extract does not silently use future knowledge.
How long should the gap between training and test data be?
Derive it from the prediction horizon, outcome availability, overlapping observations and the proposed retraining schedule. There is no universal number of rows or days. For irregular events, a row-count gap may correspond to very different durations. State the rule in business time, verify actual label arrival and explain which observations are excluded. A gap cannot fix a feature that already contains later information.
Can the same customer appear in training and test periods?
Sometimes, if the intended task predicts future activity for existing customers using legitimately available history. That evaluation does not automatically establish performance for completely new customers. Related copies of the same event create a different problem and may need to remain within a consistent episode boundary. Match the grouping and temporal design to the claim, and report important existing-versus-new population differences explicitly.
Does a point-in-time feature join always exclude late corrections?
No. Check whether the implementation constrains only business event time or also the time a version became available. Some systems use created timestamps only to select among duplicates unless additional filtering is enabled. Even an availability filter is trustworthy only if its timestamp has the intended meaning and the selected store supports it. Demonstrate a late-correction case through the actual extraction configuration.
When can outcomes from a validation period be used to refit a model?
After their defined horizons and maturity conditions are satisfied, the labels have actually arrived, and the protocol permits their use before the final test begins. Preserve an untouched final evaluation period for the resulting specification. If the team changes features or thresholds after inspecting final-test results, those results become development evidence and should not be presented as an independent estimate for the revised model.
What if historical arrival timestamps were never stored?
Describe the resulting limitation rather than inventing precision. Preserved snapshots, source logs or documented version history may support a reconstruction. A conservative delay assumption can be explored as a sensitivity case, with its status made explicit. If the missing history affects essential inputs, collect prospective evidence or narrow the experiment. Current corrected tables alone may be insufficient to establish what the workflow knew at the time.
What evidence should a temporal-readiness reviewer receive?
Provide the prediction and outcome contract, source availability ledger, calendar windows, label-maturity reconciliation, aggregate-feature definitions, preprocessing boundary and frozen evaluation specification. Include masked demonstrations of late arrivals, amendments and future outcomes being handled correctly. State unresolved history gaps and the permitted conclusion. The reviewer should be able to trace both a feature value and a training label to the information legitimately available at the relevant time.