arize-phoenix-evals

Phoenix Evals provides:

Built-in evaluators for common tasks (faithfulness, correctness, relevance, conciseness, and more)
A unified LLM wrapper supporting OpenAI, Anthropic, Google, LiteLLM, and other providers
Batch evaluation over pandas DataFrames with async_evaluate_dataframe (or sync evaluate_dataframe)
Custom evaluator creation via create_evaluator decorator or ClassificationEvaluator
Benchmark datasets for testing evaluator accuracy

Installation#

pip install "arize-phoenix-evals>=3"

Install the LLM vendor SDK for your chosen provider:

pip install openai  # or anthropic, google-genai, litellm, etc.

Quick example#

import osfrom phoenix.evals import LLM, async_evaluate_dataframefrom phoenix.evals.metrics import RetrievalRelevanceEvaluatoros.environ["OPENAI_API_KEY"] = "<your-openai-key>"llm = LLM(provider="openai", model="gpt-4o")relevance_evaluator = RetrievalRelevanceEvaluator(llm=llm)# DataFrame must have columns matching the evaluator's input fields: 'input' and 'context'results_df = await async_evaluate_dataframe(    dataframe=df,    evaluators=[relevance_evaluator],    concurrency=10,)

Core API#

Symbol	Description
`LLM(provider=..., model=...)`	Unified LLM wrapper for all supported providers
`evaluate_dataframe(dataframe, evaluators)`	Run evaluators over a DataFrame (sync)
`async_evaluate_dataframe(dataframe, evaluators)`	Async variant with concurrency control
`create_evaluator(name, kind=...)`	Decorator to turn a function into an `Evaluator`
`ClassificationEvaluator(name, prompt_template, llm, choices)`	Custom classification evaluator for LLM-as-a-judge tasks
`bind_evaluator(evaluator, input_mapping)`	Bind field mappings to an evaluator

Built-in evaluators#

All built-in evaluators live in phoenix.evals.metrics and accept llm: LLM as their first argument:

Evaluator	Input fields	Labels
`FaithfulnessEvaluator`	`input`, `output`, `context`	faithful / unfaithful
`CorrectnessEvaluator`	`input`, `output`	correct / incorrect
`RetrievalRelevanceEvaluator`	`input`, `context`	relevant / irrelevant
`ConcisenessEvaluator`	`input`, `output`	concise / verbose / too_concise
`RefusalEvaluator`	`input`, `output`	refusal / no_refusal
`ToolSelectionEvaluator`	`input`, `output`, `tool_name`, `expected_tool`	correct / incorrect
`ToolInvocationEvaluator`	`input`, `tool_name`, `tool_call_args`	correct / incorrect
`ToolResponseHandlingEvaluator`	`input`, `tool_response`, `output`	good / bad
`exact_match`	`output`, `expected`	(code-based, no LLM)
`MatchesRegex(pattern=...)`	`output`	(code-based, no LLM)

To learn more about LLM Evals, see the evals quickstart.

Reference Documentation#

Full API Reference

Complete API documentation for evaluators, metrics, and LLM classification