Events, order and recovery
Event-Driven AI Integration
Design AI consumers around trustworthy event meaning, transaction boundaries, duplicates, late arrival and controlled replay, with a worked registration-state example.
Event-driven AI integration lets a model-assisted process react to a fact recorded elsewhere: a document revision, a registration change or an approved application result. Its value is independent consumption and durable processing. Its difficulty is that events can arrive late, repeat or describe a state that has already changed by the time inference finishes.
A reliable design therefore needs more than a broker and a model worker. It needs an event contract, a transaction boundary, a rule for ordering, a record of processing and a safe way to replay history. This guide gives integration engineers those decisions in a form that can be reviewed within AI integration and delivery.
Define the fact before choosing the transport
An event should identify what the source application established. “Workshop registration cancelled” is a fact about an authoritative registration record. “Write a cancellation explanation” is a command to another capability. The distinction matters because the source owns the first fact, while the consumer can accept, reject or fail to complete the second task.
The CloudEvents specification provides a common format for event data across services and systems. A standard envelope can improve interoperability, but it does not supply your business meaning, permission model or recovery guarantee. Define those explicitly alongside the schema.
Keep the payload proportionate to its purpose. A consumer that only needs to detect a changed registration may receive an identifier, revision and changed-state description, then retrieve current authorised detail. Putting an entire personal record into every event expands the copies that must be protected and maintained.
A worked event contract for a training provider
The example is illustrative. A commercial training provider wants an AI service to prepare bilingual instructor briefing notes when workshop registrations change. The registration system remains authoritative for enrolment and cancellation. AI may prepare a briefing draft, but it cannot restore a cancelled booking, alter fees or communicate a final attendance decision.
| Field | Illustrative value | Contract meaning |
|---|---|---|
| Event identity | event-registration-218-17 | Identity of this emitted fact, stable across redelivery |
| Producer | registration-service | Authoritative source within the verified organisational scope |
| Event type | registration.status.changed | A status change has committed in the source |
| Schema version | 1 | Version of the payload meaning and structure |
| Entity identity | registration-218 | The registration whose state changed |
| Entity revision | 17 | Source ordering value for this entity |
| New status | cancelled | State established by the source application |
| Occurrence time | A source-recorded timestamp with an offset | When the source change occurred, separate from receipt time |
| Correlation reference | change-request-62 | Link to the wider business request, not a duplicate-suppression key |
Do not assume the sample field names form a complete implementation of a particular event standard. They illustrate the semantic questions the producer and consumer must answer. Establish identifier uniqueness within a documented producer and organisational scope; two tenants can legitimately use the same local registration number.
Authenticate the producer and verify the event's organisational scope at ingestion. A correctly shaped payload is not proof that its claimed source is authoritative. Define which producers may emit each event type and which consumer identities may receive it. This prevents an otherwise sound ordering and replay design from processing a forged or misrouted business fact.
Close the gap between a database change and event publication
A common failure occurs when the application commits a change but crashes before publishing its event. Publishing first creates the opposite risk: consumers react to a change that never commits. These are two separate operations unless the design explicitly coordinates them.
The transactional outbox pattern documented by AWS stores the business change and the intent to publish in the same database transaction. A relay subsequently publishes the recorded event. This closes the local database-to-publication-intent gap, but the relay can still publish more than once.
Define who monitors unpublished outbox entries, how the relay resumes and how long event history remains available. If the source application cannot be changed, investigate its supported events or change-data-capture interface. Do not assume a row-level database change expresses a complete business event: several row changes may belong to one logical registration update.
Give consumers their own duplicate and completion rules
The AI consumer should track event identity and its local processing state. Where possible, persist the inbox record and the local work item together so a crash cannot leave a recorded event without the work it was meant to create. Acknowledgement to the broker should follow the documented persistence boundary.
Do not keep a database transaction open while waiting for a model response. Create durable work, perform inference separately and persist the result against that work identity. If the worker crashes after inference but before storing the result, a retry may incur another model call; the design should still prevent duplicate accepted briefing drafts.
A local duplicate check does not automatically protect an external side effect. If a consumer submits a draft to another application, use the receiving API contract and reconcile unknown outcomes. The event transport, local database and destination application have different guarantees. Record the boundary of each rather than promising end-to-end exactly-once behaviour.
Choose ordering rules for the entity, not the entire enterprise
For a current-state briefing, ordering generally matters within a registration, not between unrelated registrations. Partition or route accordingly where the selected platform supports it, and still inspect source revisions. Arrival order is not sufficient evidence of business order when messages can be delayed or replayed.
Suppose revision 17 cancels a booking and revision 18 reinstates it through an authorised source action. If revision 18 arrives first and 17 arrives later, a projection of current registration state should not move backwards. It can record the late event for audit while retaining revision 18 as the current state.
That rule applies to a state projection, not every kind of consumer. A consumer responsible for processing every financial adjustment or every required notification may need each transition, not only the latest state. Specify whether the consumer builds a current view, processes every transition or aggregates a window. Skipping older events without that distinction can lose legitimate work.
Check freshness again when AI finishes
The model may start from revision 17 and finish after revision 18 exists. Store the source revision used for inference and compare it with the current authorised state before releasing the briefing draft. If the change invalidates the content, supersede or regenerate it according to a defined rule.
For this example, a draft explaining a cancellation must not remain current after reinstatement. A small correction to an unrelated internal note might not require regeneration. Define material fields so the system does not waste work on every incidental update or ignore changes that alter meaning.
Use event time for the business sequence where appropriate and receipt or processing time to measure operational delay. Preserve both. A delayed cancellation should not appear to have happened at the time the AI worker finally received it. Locale presentation may change, but the stored timestamp and offset must remain unambiguous.
Replay history without repeating real-world actions
Replay can rebuild a projection, test a changed prompt or recover lost processing. Declare which purpose applies. A replay for evaluation should write to an isolated result set with the model, prompt and source revisions recorded. It should not quietly update production briefing drafts or trigger notifications.
Separate replay identity from the original business event identity. Preserve the event identity for lineage and use a replay-run identity to distinguish the evaluation execution. Decide whether the objective is historical reconstruction using then-available evidence or a current reinterpretation using newer information. Those produce different results and should not be compared as if they were the same test.
Before production replay, define the interval, entities, side-effect policy, duplicate ledger, expected output population and rollback or correction plan. Replaying a dead-letter queue without examining why records failed can simply reproduce the incident at higher speed.
Operate schema changes, backlogs and poison events
Version schema meaning as well as shape. Changing a status from “withdrawn by participant” to “cancelled by provider” alters the briefing even if the field remains a string. Test old and new producers and consumers under the supported compatibility policy. Preserve a route for events that cannot be decoded or safely interpreted.
Classify failures before retrying: temporary infrastructure failure, unsupported schema, missing permission, missing source and uninterpretable content require different handling. A dead-letter record needs an owner, reason, original identity and next action. Restrict sensitive payload access rather than treating the exception queue as an unrestricted debugging store.
Monitor oldest pending event, consumer lag, unresolved outcomes, stale drafts and completed useful work. A high event-throughput graph can coexist with an unusable briefing backlog. The monitoring guide and audit-logging guide explain the operational and reconstructive evidence to retain.
Use integration testing to rehearse duplicates, reordered revisions, producer crashes, interrupted consumers and controlled replay. Compare the pattern with other integration architectures when the operating burden exceeds the value of event-driven behaviour. Trusted Data's streaming guide is related commercial reading on event-processing design.
Discuss an event-driven AI integration with an event sample, the authoritative source, ordering requirements and the actions a replay must never repeat.
FAQ
Your questions, answered.
Does a transactional outbox guarantee exactly-once AI processing?
No. It coordinates the local business change with the recorded intent to publish. Publication may repeat, consumers may retry and external applications may have separate semantics. Use durable event identity, consumer processing records and destination reconciliation. State the boundary of each guarantee rather than extending a local database property to the complete distributed workflow.
How should AI consumers handle out-of-order events?
Define whether the consumer builds current state or must process every transition. A current-state projection can use source revisions to prevent moving backwards, while preserving late events for evidence. A transition-processing consumer may need ordered recovery or explicit handling of missing steps. Arrival timestamps alone are insufficient when transport delay or replay can change delivery order.
Can we replay events to evaluate a new prompt?
Yes, using an isolated evaluation destination and a declared replay-run identity. Preserve original event and source revisions, record the prompt and model configuration, and disable production side effects. Specify whether the evaluation reconstructs what was known historically or uses current evidence. Those are different experiments and should be labelled accordingly.
What if the source changes while the model is generating a result?
Record the source revision used for inference and check relevant current state before releasing the result. Supersede, regenerate or review the draft when a material change invalidates it. Define material fields so unrelated edits do not cause unnecessary work. A result can be correct for its original event and still be stale for the current business decision.
Should an event contain the complete source document?
Only when that is justified by the consumer contract and permitted data handling. Often an identifier, revision and limited change description are sufficient, with authorised retrieval for detail. Consider event retention, consumer access and deletion or correction handling. Replicating full documents across every topic can create unnecessary exposure and lifecycle complexity.
What belongs in an AI event dead-letter process?
Include the original event identity, failure class, relevant schema and processing versions, owner, next action and controlled access to evidence. Separate transient failures from unsupported schema, denied permissions and irrecoverable input. Review the cause before replaying. A dead-letter queue is an exception-management mechanism, not a substitute for accountable resolution.