AI Agent Telemetry Design

AI agent telemetry design is the decision about which operating facts to collect, where they may go, how they are interpreted and when they are removed. Good telemetry supports a specific investigation or control. A large collection of messages without field definitions, access rules or coverage information can be expensive and still leave the important question unanswered.

This guide is for platform engineers and service owners designing the evidence layer before production volume grows. It provides a field contract, a cardinality example and a transparent storage calculation. The examples are design exercises, not measured benchmarks or prescribed retention periods.

Begin with decisions, then select the signal

List the decisions operators need to make: identify stalled work, locate a failing tool, distinguish a rejected result, compare a release or reconcile an external action. For each, identify the smallest reliable evidence set. A counter can reveal a trend, a trace can explain execution and a durable business record can establish task state.

Do not force every purpose into one signal. A sampled trace may support diagnosis but cannot necessarily supply a complete denominator for task reporting. A quality score may support comparison but cannot prove that an external permission was checked. Keep the authoritative record and the analytical view connected through explicit references.

For an illustrative service, an access-catalogue review assistant prepares a recommendation for task AC-91. It reads approved entitlement metadata and asks a security reviewer to decide whether an entry needs further investigation. It does not grant or revoke access. The telemetry needs to explain the recommendation and review state without reproducing the entire directory or credentials.

Write a field contract that an engineer can implement

Illustrative telemetry field contract for AC-91

Field group	Purpose	Handling rule
Task and attempt references	Connect executions and review decisions	Opaque identifiers in controlled traces or records; avoid per-task metric labels
Service and configuration versions	Identify the behaviour under investigation	Record stable version references with a documented owner
Operation and outcome categories	Aggregate failure patterns	Use a bounded vocabulary with an explicit unknown value
Entitlement source reference	Identify the approved evidence used	Retain a controlled reference, not the whole directory response
Reviewer disposition	Explain acceptance, rejection or investigation	Store the decision and authorised role; restrict personal detail
Timing and resource measures	Diagnose waiting, latency and abnormal consumption	Define units, start/end events and measurement origin
Raw content and exceptions	Support specifically approved diagnosis	Minimise, classify and restrict; exclude secrets before export

For each actual field, add its type, null meaning, producer, permitted destinations, access roles, retention class and schema version. Define whether a missing value means unknown, not applicable, not collected or lost. Collapsing these into an empty string makes later analysis unreliable.

Choose a custom namespace for organisation-specific fields and use applicable standards where they fit. OpenTelemetry for agents covers convention versions. Do not reuse a standard-looking attribute name with a different private meaning; that makes downstream interpretation fragile.

Keep unique identifiers out of aggregate metric dimensions

Prometheus explains that each distinct combination of metric labels creates a separate time series. Unbounded identifiers can therefore multiply storage and query work. A task ID can be valuable in a trace while being unsuitable as a metric label. The same distinction applies to email addresses, free-text error messages and full document names.

As an illustrative upper-bound calculation, six service types, two environments, four outcome categories and three language categories allow 144 label combinations for one metric. Adding a dimension with 10,000 task identifiers creates a potential cross-product of 1,440,000 combinations. Not all combinations will occur, and histogram representation can add further series; this is a design warning, not a capacity forecast.

Use bounded outcome and error categories for aggregation, with a controlled drill-down to task evidence. Preserve an explicit unknown category rather than creating a new label value from every exception string. Review the vocabulary when a new agent or tool is introduced so a seemingly harmless field does not become an unbounded dimension.

Put content controls at the correct boundary

OpenTelemetry's sensitive-data guidance recommends minimisation and warns that hashing predictable identifiers may not achieve anonymisation. A hash can support correlation while the underlying person or record remains identifiable. Assess the actual re-identification risk and permitted use rather than treating a transformation as automatic anonymity.

For AC-91, record the entitlement category and controlled source reference needed to investigate the recommendation. Do not export authentication tokens, directory credentials or unnecessary staff details. Review tool exceptions carefully: an error may echo a request body even when normal responses are minimised.

Decide whether filtering occurs in the application, a trusted local exporter, a collector or the backend. If a field must not leave the application environment, backend masking is too late for that requirement. Test all paths, including debug logs and failed exports, with distinctive synthetic sensitive values.

Do not place sensitive correlation content in propagation baggage merely to avoid configuring attributes. Baggage can travel across service boundaries. Define allowed keys and destinations, and keep authority checks independent of propagated values. The PDPL operating-data guide covers the separate review of relevant personal-data flows.

Separate completeness requirements from sampling choices

Decide which records must exist for every admitted task and which diagnostic detail may be sampled. The task-state ledger and a required approval record may have different completeness requirements from a verbose model trace. Document those requirements before selecting a sampling rate.

OpenTelemetry distinguishes head sampling, decided early, from tail sampling based on later trace evidence. Early sampling cannot guarantee retention of every error discovered later. Tail sampling needs buffering, timing and capacity decisions, and cannot recover data already discarded upstream. Long-running tasks can also span several traces and exceed a sampler's decision window.

Keep routine diagnostic samples separate from deliberately retained incidents or unusual cases. If failures are retained more often, an unweighted success rate calculated from those traces is biased. Use an appropriate complete event source for operational counts, or document a valid estimation method and its limits. Do not disguise a selected investigation set as all production traffic.

Calculate volume before negotiating a bill

Use measured serialized event sizes from a representative test, then model the proposed workload. In an illustrative calculation, assume 2,400 tasks per day, fourteen spans per task and 1,800 bytes per span. Multiplication gives 60,480,000 bytes, or 60.48 decimal MB of raw span data per day. At an illustrative thirty-day retention period, that is about 1.81 decimal GB.

This estimate excludes indexes, replication, backups, logs, metrics, attachments, evaluation data and protocol overhead. Compression can reduce some stored bytes; duplication and indexing can increase others. It is not a vendor quote. Replace the assumptions with observed distributions, including large tool outputs and retry-heavy tasks, before sizing a production service.

Set budgets for event size, events per task, label cardinality and exporter queue capacity. When a limit is reached, preserve an explicit truncation or loss indicator. Silent clipping can remove the field that explains an incident while leaving a trace that appears complete. Route oversized content to an approved controlled store only when the investigation purpose justifies it.

Design retention as a lifecycle

Assign retention by purpose and data class. Short-lived diagnostic detail, approved test fixtures, aggregate trends and accountability records may require different treatment. The duration must follow the organisation's applicable requirements and decisions; the thirty-day arithmetic example above is not a legal or operational recommendation.

Map deletion across primary storage, search indexes, exports, caches and backups. Document any controlled exception, such as an approved investigation hold, including its owner and release condition. A record disappearing from the main interface does not establish that all copies have been removed.

Preserve interpretability when detail expires. An aggregate trend should retain its definition, period and coverage limitations. A test fixture should retain its approved source and expected outcome if it remains in use. Avoid a situation where a score survives but the organisation can no longer explain what was evaluated.

Protect evidence access as carefully as application access

Separate people who administer the platform from people who need to inspect content. Grant operators enough evidence to act, with a controlled escalation route for restricted detail. Test search, direct record links, exports and APIs, because interface filtering alone does not establish record-level access control.

Log material changes to capture policy, retention, export destinations and schema mappings. A telemetry configuration change can expose data or destroy visibility without changing the agent application. Use an approval process proportionate to that effect, and preserve a rollback route.

For records serving accountability, align with the distinct audit-logging contract. For financial attribution, preserve the measurement origin and units needed by AgentOps FinOps. The telemetry design should support those processes without pretending one sampled diagnostic stream fulfils all requirements.

Accept the design through a field-level review

Review one normal task, one rejected recommendation, one tool exception and one interrupted export. For each, inspect captured fields, destinations, access, loss indicators and expiry behaviour. Ask the operator to explain AC-91 using only the permitted evidence. If the explanation is impossible, refine the contract; do not respond by enabling unrestricted content capture.

Maintain the field contract, volume model and review fixtures with the service. Production monitoring then turns the approved signals into decisions and response routes. AgentOps services can be scoped around the evidence architecture and operating handover required.

Discuss a telemetry field contract and storage model with a sanitised event sample, expected workload and current retention policy. Those inputs reveal where to reduce unnecessary data, where stronger evidence is needed and which operating responsibilities must be assigned.