How to: Experiments

Configure prompts and evaluators in the Playground and compare results.

Run experiments programmatically with tasks and evaluators in code.

Start experiments from the Playground that keep running after you close the browser, and stop or resume them anytime.

Load your test cases into Phoenix to use as inputs for experiments.

Define the function or workflow you want to evaluate against your dataset.

Set up the scoring criteria to assess your task outputs.

Execute your task across all dataset examples and collect evaluation results.

Run tasks multiple times to measure variance and consistency.

Run experiments on specific subsets of your dataset.

Use LLM-as-a-judge to assess quality, correctness, and other criteria.

Built-in heuristic evaluators like exact match, JSON distance, and regex.

Build your own evaluation logic with custom prompts or code.

Attach evaluators to datasets for automatic scoring during experiments.