Model calls and task outcomes
LLM Observability vs Agent Observability
Distinguish model-response evidence from authorised actions, external confirmations and the complete agent task outcome.
LLM observability explains interactions with a language model. Agent observability must also explain the task surrounding those interactions: which tools actually ran, which decisions were authorised, what changed in external systems and whether the intended outcome was accepted. A high-quality model response can coexist with an incomplete or incorrect agent task.
The boundary is functional, not a fixed division between vendors. Many products support both model and agent workflows, and their capabilities evolve. Evaluate the evidence your implementation produces. A dashboard labelled agent observability may still show only model calls if the application never records tool outcomes or task state.
Start with two different units of observation
A model interaction has a request, response or stream, configuration, duration and reported resource use. Depending on permitted capture, it may also have instruction and context references. These signals help investigate response quality, provider errors and model-level performance.
An agent task has a goal, an applicable authority boundary, execution attempts, decisions, tool interactions, interruptions and an end-state decision. It may contain several model interactions or deterministic steps without a model. The task can continue after a model response finishes and after the initiating application request has returned.
The current OpenTelemetry conventions distinguish model operations from agent and tool operations. Those technical concepts support the distinction, but the organisation still defines its business-task acceptance contract. Do not equate a provider response, a framework invocation and a customer-accepted task.
Illustrative case: studio-session rescheduling task IS-58
Consider a hypothetical studio-session scheduling assistant. An authorised production coordinator asks it to move a recording session to an agreed new slot and notify the sound engineer. The example uses synthetic identities and a test calendar. The operating contract requires both a confirmed calendar update and the defined notification outcome.
The model correctly identifies the recording session, returns valid tool arguments and produces a clear message. The calendar update is confirmed. The notification tool then returns an uncertain result after its request was accepted. The agent's final message says the rescheduling is complete, although the notification state has not been established.
A model-only view may show a correct response and ordinary latency. A task-level view must preserve the confirmed calendar change, the unresolved notification and the premature completion statement. The right recovery is to establish the notification state and apply the approved next step. Repeating the entire task could create unnecessary or conflicting changes.
Use an evidence ladder
| Level | Question answered | What it does not establish alone |
|---|---|---|
| Model response | What did the model return under the observed input? | That a proposed tool call was executed |
| Tool request | What operation did the application attempt? | That the caller had authority or the external effect occurred |
| Authorisation result | Was the action permitted under the applied rule? | That the permitted action completed |
| External confirmation | What result did the receiving system confirm? | That every other task requirement was satisfied |
| Quality judgement | Did the observed result meet a defined criterion? | That omitted or unobserved actions were correct |
| Task disposition | Was the complete task accepted, escalated or unresolved? | Why it reached that state without linked execution evidence |
This ladder prevents a common diagnostic error: promoting evidence from one level into a stronger conclusion than it supports. Schema-valid arguments are useful, but they do not prove permission. A delivery-service acceptance may be useful, but its meaning depends on the interface contract. A final natural-language statement does not override authoritative external state.
Reconstruct the task from the end state backwards
Begin an investigation with the acceptance requirement. For IS-58, identify the calendar reference, approved new slot and notification condition. Check the receiving systems for their confirmed states, then locate the requests and authorisation decisions that produced them. Only then examine model context and response choices to explain the difference.
This order avoids blaming the model for a missing callback or assuming a correct model response means the whole service worked. It also prevents a team from changing instructions to compensate for an integration or state-management defect. The repair should address the component that controls the failed requirement.
Preserve a durable task reference and separate attempts. Link model interactions to tool execution, review events and external references without relying on matching text. Task tracing explains that relationship model. The model-call record remains valuable, but it becomes one part of a wider evidence graph.
Measure duration at the intended boundary
A model-call duration measures the model operation. An agent invocation may include several calls and tools. A business-task duration may additionally include queueing, review and external confirmation. Define the start and end events for each measure. Comparing them without those definitions can make a normal waiting state look like a slow model.
The current GenAI metric conventions distinguish an in-process agent invocation from a provider-facing client operation. Your business service may be broader than either. If IS-58 waits for a notification acknowledgement after the invocation ends, preserve that interval in the task lifecycle rather than silently excluding it from service reporting.
Use the same care with cost and call counts. Parent and child agents can generate overlapping summaries; summing both without checking the aggregation rule can double-count work. Define where measurements originate and which level includes which operations. Cost per accepted task addresses the outcome-based economic measure.
Evaluate the response and the trajectory separately
Response evaluation asks whether the text or structured output is suitable for a criterion. Trajectory evaluation examines the observable sequence of tool use and decisions. Task evaluation asks whether the complete outcome meets the service contract. These can disagree for legitimate reasons and should remain distinguishable.
For IS-58, the model's calendar arguments may be correct, the tool sequence may be authorised, and the task may still be unresolved because notification confirmation is missing. The final completion statement can also be wrong even when the earlier model output was correct. Record each finding instead of replacing them with one opaque quality score.
LangSmith's evaluation documentation distinguishes evaluation before deployment from evaluation of production activity. Whichever tool you use, define the property being judged and preserve evaluator or reviewer versions. Model-assisted judges require their own calibration; a model's confidence is not a substitute for external confirmation.
Watch the tool boundary, not just the tool-call text
Instrument the adapter that actually executes the operation. Record permitted arguments or safe references, the applicable authorisation result, the attempt and the response meaning. Keep requested, accepted, executed and confirmed states distinct where the interface exposes them. If the external system cannot establish a state immediately, record uncertainty.
Do not infer safe replay from a failed-looking span. The receiving system may have acted before a timeout. In the scheduling example, a notification may have been sent even though confirmation was lost. The retry and idempotency guide explains how the application can control repeated effects; observability supplies the evidence for that decision.
Also inspect actions the model did not directly choose. Frameworks, middleware and deterministic business logic may perform additional work. An agent history containing only model-requested tools can omit important side effects. Inventory the actual action surface and ensure each relevant component emits evidence.
Preserve the human decision as a separate event
If a coordinator approves a changed slot, record the scope and applicable version of that approval. A later change to attendees or time may require a new decision under the service's rules. Do not treat a historical approval flag as unrestricted permission for subsequent tool calls.
A human rejection is not automatically a system failure. It may be the intended safe disposition when information is incomplete. The task view should show the review request, decision, reason and resumed or stopped state. A model-call view can provide context, but it cannot establish the reviewer's authority by itself.
The preventive checks belong to action authorisation controls. Keep them outside the model's generated explanation. Recording that an action was unauthorised after it happened is useful for investigation, but does not replace checking permission before execution.
Choose tooling by the missing evidence
If the immediate issue is provider latency or response formatting, model-level instrumentation may answer the question. If the issue is incomplete work, inappropriate tool execution or uncertainty about external results, extend the evidence to the task and action layers. Avoid buying a broader label without demonstrating the relevant workflow.
Ask the proposed platform to reconstruct IS-58 from a synthetic fixture: correct model output, confirmed calendar update, uncertain notification and premature completion text. The operator should identify the unresolved requirement without being told the answer. Test the same fixture with missing telemetry so absence of evidence is not mistaken for a completed action.
Use agent observability versus application monitoring for the separate infrastructure and application-health comparison. Model, agent and application signals can share tooling, but each must have an explicit interpretation and owner. The decision is about coverage and operating usefulness.
Build a combined operating view
A useful task view shows the accepted request, authority, current state, significant model and tool operations, review decisions and confirmed external results. It also links to application health and identifies evidence gaps. Different roles may see different content, but they should agree on the state definitions.
Do not require unrestricted transcripts to establish every outcome. Safe references and authoritative confirmations can support many decisions. Keep additional content available only where the approved investigation needs it, with appropriate access and retention. A complete conceptual view does not mean copying every payload into every dashboard.
Discuss the gap between your LLM monitoring and agent outcomes with one masked task that looked successful at the model layer but remained unresolved operationally. AgentOps services can be scoped around the evidence and controls needed to close that gap. The goal is to know what the service actually accomplished and what still requires a decision.
FAQ
Your questions, answered.
What does agent observability add to LLM observability?
It connects model interactions to the wider task: authority, actual tool execution, human decisions, external confirmations and final disposition. A model record remains useful for response-quality and provider investigations. The addition is evidence about what the service actually accomplished, including work that continues after a model response or framework invocation has ended.
Can a correct LLM answer still result in an agent failure?
Yes. A downstream tool may fail, a request may remain unconfirmed, permissions may be inappropriate or another task requirement may be incomplete. In the studio-session example, correct calendar arguments coexist with an unresolved notification. Evaluate the response and the complete task separately, and use authoritative external evidence to establish the end state.
Is agent latency the same as LLM response latency?
No. A model operation is one timing boundary. An agent invocation may contain several model and tool operations, while a business task may also include queueing, human review and external confirmation. Define each start and end event. A task waiting for a required acknowledgement should not disappear from service reporting because the model response was fast.
Should tool-call suggestions be counted as executed actions?
No. A model can propose a tool call that is rejected, never dispatched or fails before execution. Capture the actual adapter and receiving-system evidence, with requested, accepted and confirmed states distinguished where applicable. Keep authorisation separate as well; valid arguments do not establish permission to perform the action.
Can one platform provide LLM and agent observability?
Yes, depending on its capabilities and your instrumentation. The product label alone is insufficient. Demonstrate a task with several model calls, actual tools, a human decision and an uncertain external result. Verify that the operator can reconstruct it with appropriate access. Additional application events may be needed even when the platform supports agent views.
How should we evaluate an agent that correctly escalates to a person?
Treat escalation according to the task contract. It can be the correct disposition when authority or information is insufficient. Record why review was requested, who may decide and what happened next. Do not mark every escalation as failure or every model response as success; assess the behaviour against the intended service outcome.