Skip to main content

Available in arize-phoenix-client 3.4.0+, with trajectory traces in 3.5.0+

Phoenix now ships a job plugin for Harbor. Select it on a Harbor run and the job lands in Phoenix as a dataset, one experiment per resolved agent configuration, a run per trial, and every verifier reward as a named evaluation. No tracing code goes inside the sandbox.

Phoenix dataset page for a Harbor dataset showing eight experiments, with annotation score comparison, run latency, and cost charts above a table listing each experiment's agent configuration, infra_ok score, and a link to its traces
Each agent and model configuration in a Harbor job becomes its own experiment on the job's dataset, so reward, latency, and cost line up across runs.

Rewards Become Scores#

reward distributions show up in experiment comparison. An infra_ok evaluation helps distinguish low task scores from execution errors. health, which tells a failed agent apart from a broken sandbox.

Phoenix experiment comparison for a single Harbor trial, showing the task input and reference output beside the agent's response, with reward, infra_ok, agent_turn_count, and tool_call_count evaluations and their averages
One trial in experiment comparison: the task instruction, the optional reference output, and the agent's final response, with the verifier's rewards recorded as evaluations.

Trajectories Become Traces#

Phoenix experiment run trace for a Harbor trial, with the harbor.trial root span above the codex agent span and its iteration, openai/gpt-5.6 LLM, and exec_command tool spans, beside the selected tool span's input and output
A Harbor trial's trajectory as a Phoenix trace, opened from its experiment run: the `harbor.trial` root, the agent beneath it, and an LLM and a tool span for each iteration.
  • trace_mode="atif" is the default. The plugin reads the saved trajectory after the final trial attempt and converts it into one trace and one session, with AGENT, LLM, and TOOL spans under a harbor.trial root. A multi-step trial adds a span for each attempted step, and context-management operations appear as compaction spans.
  • Tracing never changes a score. If the agent saves no valid trajectory, the plugin logs a warning and records the run and its evaluations without a trace. Pass trace_mode=null on the Harbor command line to turn tracing off entirely.
  • The sandbox stays offline. The plugin converts trajectories after the trial, so the agent’s environment needs no Phoenix endpoint and no Phoenix credentials.

Run a Job with the Plugin#

Install Harbor and the plugin into the same Python environment:

The plugin registers under the name arize-phoenix, so a Harbor run picks it up directly:

Configure it in Python when you want the dataset name or the naming template under version control:

  • Experiment names follow a template. experiment_name_template accepts fields like {job.name}, {agent.name}, {agent.model}, and {agent.short_digest}, and
  • Resuming the same Harbor job reuses its experiments without duplicating recorded runs.
  • Resuming is idempotent, so a re-run adds to the same experiment instead of duplicating it. The plugin requires Harbor 0.21.0+, Python 3.12+, and Phoenix server 15.0+. Endpoint and API key fall back to The plugin requires Harbor 0.21.0+ on Python 3.12+. Endpoint and API key fall back to PHOENIX_COLLECTOR_ENDPOINT and PHOENIX_API_KEY.
Harbor

Install the plugin, map its objects, and read its scores correctly

Importing ATIF Trajectories

The trajectory format the plugin converts