LangSmith vs Langfuse vs Arize: Agent Observability Compared

LangSmith, Langfuse and Arize can support agent observability, but a meaningful comparison starts with the operational investigation your team must perform. Can an operator reconstruct an interrupted task, explain an evaluation failure, identify the affected release and export enough evidence for a corrective decision? A platform demonstration should answer those questions with your workflow.

In this guide, Arize means Arize AX, the enterprise platform. Phoenix, whose current repository uses Elastic License 2.0, is a separate Arize project with a different deployment and licensing discussion, covered in Phoenix versus Langfuse. This comparison is for engineering and platform leaders selecting an observability environment. It is an evaluation protocol, not a claim that any product has won a benchmark on your workload.

What the current product documentation establishes

Documented product scope and the corresponding purchase question

Platform	Established scope	What to verify in your trial
LangSmith	Tracing and observability; separate offline and online evaluation workflows	Whether the complete task and review workflow is visible with your frameworks and deployment
Langfuse	Observability, evaluation and prompt-management capabilities; core code under MIT with enterprise exceptions	Which operating and administration features your selected edition includes
Arize AX	Enterprise ML and LLM observability, including ingestion of OpenTelemetry traces	How agent traces, evaluation evidence and existing model monitoring fit your operating process

The table draws on LangSmith observability, LangSmith evaluation types, Langfuse observability, its repository licence, and Arize AX architecture. A documented capability establishes that an evaluation is worth attempting. It does not establish coverage of your custom tools, the quality of your evaluators or the availability of a particular permission in every commercial edition.

For deployment, LangSmith self-hosting requires an enterprise licence. Langfuse provides self-hosting guidance, with production operation requiring more than a local demonstration. Arize AX documents a self-hosted option subject to commercial arrangements. Request the exact architecture and entitlement schedule in each proposal; “private” and “self-hosted” are not interchangeable descriptions of control-plane location, support access or outbound traffic.

Build one comparison pack before opening three dashboards

Choose a bounded agent workflow that resembles production. Include its application version, model configuration, instruction version, tools, source fixtures and expected end states. Use synthetic or appropriately de-identified records. Freeze the pack while comparing platforms, so a changed workload cannot be mistaken for a product difference.

For an illustrative evaluation, use a technical-document review agent preparing a release-note draft. It retrieves an approved component specification, checks two change tickets, identifies a missing dependency and requests an engineer's decision before marking the draft ready. The comparison examines observability of that process, not whether the platform writes better release notes.

Prepare five variants: a clean completion; an outdated specification; one delayed ticket lookup; an engineer rejection followed by correction; and a worker restart before completion. Give each variant an expected event map. Keep the business task identifier constant across attempts where they belong to the same task, but assign separate attempt identifiers. This allows the test to distinguish genuine reconstruction from a visually attractive but incomplete trace.

Run five operator exercises

1. Reconstruct the task without asking its developer

Give an operator only the task identifier and the investigation question. Ask them to locate the relevant attempts, retrieval evidence, tool calls, engineer decision and final draft status. Record missing events and manual steps. If a dashboard shows only model calls, require the team to instrument the remaining application events before crediting task-level coverage.

Check asynchronous boundaries explicitly. A queue consumer may run in another process; an engineer decision may arrive hours later. The operator must be able to connect those events without treating every long-lived task as one continuously open request. A trace viewer's ability to display a hierarchy does not prove your instrumentation preserves it.

2. Explain a quality failure with inspectable evidence

Run the outdated-specification case and attach an evaluation that checks the cited revision against the approved fixture. The operator should see the input, applicable expectation, result and evaluator version, subject to content permissions. A vague score such as “quality: poor” is less useful than a failure linked to a specific evidence mismatch.

Keep deterministic checks distinct from judgement. A revision identifier can be checked mechanically; whether a dependency explanation is adequate may require calibrated human or model-assisted review. Use the same evaluation logic for all platforms, and preserve reviewer disagreements. Outcome evaluation explains why the acceptance contract must be defined independently of the dashboard.

3. Turn the failure into a repeatable test

Ask the operator to create or identify a test case without copying sensitive production content indiscriminately. Preserve the source version, expected disposition and reason for inclusion. Run an unchanged application first, then a deliberately corrected version. Check that the experiment identifies which dataset, evaluator and application versions produced each result.

This exercise tests the working relationship between tracing and evaluation. Count the manual transformations needed, such as restructuring tool messages or extracting expected outcomes. Those transformations can be legitimate, but they need ownership and tests. A one-click workflow that silently drops relevant state is less dependable than a documented conversion that preserves meaning.

4. Exercise access, masking and export

Introduce a synthetic restricted field and verify where it is removed or transformed: application, exporter, collector, ingestion layer or user interface. A field hidden in a browser may still exist in storage or exports. Test an operator, an evaluator and an administrator separately using the permissions actually available in the proposed edition.

Export one complete investigation pack. It should retain identifiers, timestamps, event relationships, version references and evaluation meanings. Identify anything requiring a proprietary interface to interpret. The goal is usable portability, not a promise that every platform has an identical internal data model. OpenTelemetry for AI agents explains the boundary between transport interoperability and semantic compatibility.

5. Observe the observability system failing

Make the exporter unavailable in a controlled test. Inspect buffering, dropped records, application impact and recovery behaviour. Then interrupt a worker before its buffered events are flushed. Record whether the platform shows partial evidence distinctly or leaves an apparently normal trace. Telemetry loss must not be silently counted as successful business work.

For self-hosted candidates, include an upgrade and a restore exercise on a representative test environment. For hosted candidates, request documented retention, incident and export arrangements. Test responsibilities vary by deployment, so distinguish what you observed from what the supplier undertakes contractually.

Use an evidence ledger instead of a feature-count winner

Reusable ledger for the three-platform evaluation

Exercise	Required evidence	Record separately
Task reconstruction	Correct event sequence and unresolved-state identification	Instrumentation gaps, operator effort and query limitations
Quality investigation	Failure linked to source and evaluator versions	Judgement uncertainty and access restrictions
Repeatable experiment	Versioned test case and comparable run results	Manual conversion and maintenance work
Privacy and export	Verified field handling and usable exported relationships	Edition rights and retained copies
Operational resilience	Visible telemetry loss and documented recovery	Supplier responsibilities versus customer responsibilities

Mark each requirement demonstrated, demonstrated with additional work, not demonstrated, or outside scope. Do not convert an untested claim into a high score. A missing mandatory privacy control can eliminate a deployment even if its trace viewer performs well. Conversely, a cosmetic preference should not outweigh a reliable investigation workflow.

Compare the total operating burden

Request costs against the same workload envelope: trace volume, event size, retention, evaluation runs, users and growth assumptions. Include application instrumentation, evaluator maintenance, infrastructure, upgrades, incident handling and data export. Avoid comparing one hosted subscription with another product's licence cost while omitting the people and systems needed to operate it.

For a Saudi enterprise, review the complete data flow with the relevant security, privacy and procurement owners. This guide does not establish that any named cloud service is hosted in Saudi Arabia. The Saudi enterprise observability shortlist covers the procurement gates that should precede a final choice. Separate the technical result of your trial from legal and contractual acceptance.

If your team mainly needs a workable incident-to-test loop, select the candidate that demonstrates that loop with acceptable maintenance effort. If the organisation already operates extensive model monitoring, test whether a shared environment improves investigation without obscuring agent-specific task outcomes. If infrastructure control is essential, prove the chosen deployment under the actual operating team's constraints.

Write the decision so it can be revisited

The decision record should name the product, edition, version or service date, deployment model, mandatory controls, trial evidence and unresolved exceptions. Include why other candidates were not selected for this workload. That is more useful than declaring a universal winner that becomes stale when a product changes or a new agent is introduced.

Assign review triggers: a new data category, framework, region requirement, action type or material volume change. Preserve the trial pack for future upgrades and migrations. The AgentOps tooling stack guide helps place observability alongside controls it cannot replace, such as action authorisation and incident ownership.

Discuss an observability platform evaluation with a masked task diagram, proposed deployment boundaries and your mandatory investigation questions. OpenForAI's AgentOps service can be scoped around the evidence needed for that decision, with deliverables agreed for the engagement.