Arize Phoenix vs Langfuse

Arize Phoenix and Langfuse are credible candidates for teams evaluating agent traces and experiments, but the decision has two distinct parts: permission to use the proposed software arrangement, and evidence that the operating workflow works. A licence comparison cannot replace an experiment, and a successful experiment cannot replace a licence review.

This guide is for platform engineers, evaluation leads and technical procurement owners considering self-managed or hosted arrangements. Phoenix is a separate project from Arize AX; do not transfer AX enterprise features or commitments into a Phoenix comparison. LangSmith, Langfuse and Arize AX covers that different shortlist.

Read the actual licences before using an open-source label

The current Phoenix repository licence is Elastic License 2.0. Its terms restrict offering the software to third parties as a hosted or managed service exposing a substantial set of its functions. Describe Phoenix as source-available and self-hostable under its applicable terms; do not equate those terms with MIT.

The Langfuse repository licence applies MIT to its core while reserving enterprise areas under separate terms. Identify the components and edition you intend to deploy. For a service provider, record who operates the instance, who accesses its functions and what the customer receives. Have the intended arrangement reviewed against the exact licence and any commercial agreement.

This distinction does not imply that every consulting engagement involving Phoenix is prohibited, or that every Langfuse capability is unrestricted. It means that product labels are too coarse for a purchasing or service-design decision. Preserve a copy or immutable reference to the applicable release's licence in the procurement record.

Compare a complete experiment rather than a dashboard screenshot

Phoenix documents tracing and evaluation capabilities, with versioned datasets whose examples can include inputs, reference outputs and metadata. Langfuse documents datasets and experiments, including item version selection. These are starting points for a trial; they do not establish that identifiers, scores or review histories have equivalent meanings across the products.

Use an experiment that includes retrieval, a model response and a decision to answer or request clarification. An agent's end state may depend on all three. Preserve the test application's implementation independently of either platform so you can distinguish application behaviour from the platform's representation of it.

For an illustrative case, consider a photography-equipment support assistant answering questions about camera housing Q-5. It checks the product variant, retrieves the relevant manual and drafts an answer for review. Some questions intentionally omit the variant; the correct outcome is clarification. The example uses synthetic manuals and does not provide real equipment instructions.

Create a portability manifest for experiment EXP-R9

The manifest should explain what must survive an export, a backup restoration or a move to another platform. It belongs to your organisation's evaluation process, even when a platform stores copies. Give the illustrative experiment a stable business reference, EXP-R9, separate from whichever internal run identifiers each platform assigns.

Portability manifest for an agent experiment

Object	Meaning to preserve	Verification
Case set	Exact included cases and excluded cases	Compare stable case IDs and content hashes
Source fixture	Manual revision and product variant	Resolve each reference to the intended immutable fixture
Application candidate	Instructions, model configuration and tool contracts	Identify the approved combination, including any unavailable provider version
Evaluation result	Score scale, direction, eligibility and missing-value meaning	Recalculate selected cases from the retained evidence
Review history	Who judged what and whether the judgement was superseded	Recover the controlling decision and reason
Execution evidence	Task, attempt and causal relationships	Reconstruct one answer, one clarification and one interrupted case

A content hash helps detect changed bytes; it does not prove that the content is correct or that a source reference remains accessible. Keep the authorised fixture and its retrieval procedure as well. Where a model service cannot reproduce a historical version, state that limit rather than promising an identical rerun.

Test three kinds of equivalence

Data equivalence

Export the selected case set and load it into the other candidate using a documented mapping. Check nested structures, arrays, empty strings, absent fields, language tags and source references. A missing value must not silently become a zero score or an empty reference answer. Preserve the difference between “not evaluated” and “evaluated and failed”.

Inspect a difficult case manually in each interface. Arabic text, product codes and mixed-language quotations should remain intact. Do not infer content correctness from a successful file import. If the application expects structured tool messages, verify the order and role information rather than flattening everything into one text field.

Evaluation equivalence

Keep the same evaluator implementation and rubric when comparing platforms. For Q-5, use separate checks for correct variant selection, source support and appropriate clarification. If one platform displays a larger-is-better score while a custom evaluator returns an error count, make the conversion explicit. Otherwise the same result can appear better in one view and worse in another.

Define the denominator for each result. A source-support check may apply only to cases that produced an answer; a clarification check applies to intentionally incomplete requests. Reporting one average across both without an eligibility rule can conceal a behaviour change. The outcome evaluation guide develops those acceptance decisions beyond the portability test.

Operational equivalence

Ask the future operator to reproduce an investigation after the data move. They should find the task, inspect permitted evidence, identify the controlling review and explain the comparison between application candidates. Record the manual work and permissions needed. A technically complete export can still be operationally unusable if nobody can interpret it.

Test the reverse direction or an independent readout of the exported package. This reveals assumptions hidden in the initial import. Do not promise a lossless, automatic migration until the required objects and relationships have been verified. OpenTelemetry can assist trace interoperability, but it is not a universal format for every experiment or permission model.

Decide where the evidence should live

Keep raw diagnostic content, reviewed test fixtures and business acceptance records conceptually distinct. Their access and retention needs may differ. A reviewer may need a sanitised manual excerpt without the original production conversation; an engineer may need a restricted trace while investigating a specific incident.

Define the system of record for each object. If the case registry is authoritative in a repository, platform edits need a controlled reconciliation route. If the platform is authoritative, require a usable export and restoration procedure. Two editable copies without a precedence rule create uncertainty about what was actually tested.

The telemetry design guide covers field classification and retention, while agent audit logging covers evidence intended to support accountability. Do not assume the same store, permissions and sampling policy are appropriate for all three purposes.

Evaluate self-hosting as an operating service

For each candidate, review the actual release architecture and production configuration. Langfuse's self-hosting guidance distinguishes local setup from production operation. Apply the same practical discipline to a Phoenix deployment: identify persistence, authentication, network access, backups, upgrades and failure recovery for the arrangement you will run.

Perform a restore test using EXP-R9. Recover the case set, source references, evaluation outputs and review decisions, then ask an operator to explain the experiment. Separately interrupt ingestion and verify whether delayed or lost events are visible. Restoring a user interface does not establish that your evidence history survived.

Plan capacity with representative event sizes and query patterns. Large retrieved documents can dominate storage even when task counts are modest. Avoid retaining raw documents merely because a default integration captures them. A reference and an approved excerpt may be sufficient, provided they support the defined investigation and are accessible to the right role.

What should determine the final choice?

First, the licence and deployment must fit the intended arrangement. Next, the experiment must remain interpretable and maintainable. Finally, the operating team must be able to support the service, including upgrades and recovery. Treat these as separate decisions so strong performance in one cannot hide a failure in another.

If your priority is a particular research or evaluation workflow, demonstrate it with your reviewers and data representation. If your priority is a shared production environment, test access, retention and operating responsibilities in the exact edition proposed. Do not infer enterprise controls from the existence of an open repository, and do not infer poor suitability from a commercial feature boundary without testing your actual requirements.

For a Saudi deployment, document the complete processing path and the responsible parties. Self-hosting describes who operates software; it does not by itself establish where every model call, backup, support session or evaluation runs. Saudi enterprise tool selection provides the procurement gates for those questions.

Discuss a Phoenix or Langfuse evaluation and migration plan with an example experiment, the intended service arrangement and a masked evidence export. AgentOps services can be scoped to the operating and evidence requirements around that choice. The resulting decision should be reproducible when either platform or your requirements change.