Pydantic Evals - Phoenix

Pydantic Evals is an evaluation library that provides preset direct evaluations and LLM Judge evaluations. It can be used to run evaluations over dataframes of cases defined with Pydantic models. This guide shows you how to use Pydantic Evals alongside Arize Phoenix to run evaluations on traces captured from your running application.

Launch Phoenix#

uvx arize-phoenix serve

No uv? pip install arize-phoenix && phoenix serve does the same thing. See Terminal setup for customization.

docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest

Images are published to Docker Hub. See Docker for volumes, PostgreSQL, and other options.

Phoenix serves its UI and OTLP HTTP on port 6006, and OTLP gRPC on port 4317. For a local instance that’s http://localhost:6006. Leave it running while you work.

Install packages:

pip install arize-phoenix-otel

Point your code at the Phoenix instance you started. The endpoint below is the default for a local phoenix serve; for a deployment running elsewhere, use its hostname instead.

import osos.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"# Only if the deployment has authentication enabled# os.environ["PHOENIX_API_KEY"] = "your-api-key"

Install#

pip install pydantic-evals arize-phoenix openai openinference-instrumentation-openai

Setup#

Enable Phoenix tracing to capture traces from your application:

from phoenix.otel import registertracer_provider = register(    project_name="pydantic-evals-tutorial",    auto_instrument=True,  # Automatically instrument OpenAI calls)

Basic Usage#

1. Generate Traces to Evaluate#

First, create some example traces by running your AI application. Here’s a simple example:

from openai import OpenAIimport osclient = OpenAI()inputs = [    "What is the capital of France?",    "Who wrote Romeo and Juliet?",    "What is the largest planet in our solar system?",]def generate_trace(input):    client.chat.completions.create(        model="gpt-4o-mini",        messages=[            {                "role": "system",                "content": "You are a helpful assistant. Only respond with the answer to the question as a single word or proper noun.",            },            {"role": "user", "content": input},        ],    )for input in inputs:    generate_trace(input)

2. Export Traces from Phoenix#

Export the traces you want to evaluate:

import phoenix as pxfrom phoenix.client.types.spans import SpanQueryquery = SpanQuery().select(    "llm.input_messages",    "llm.output_messages",)# Query spans from Phoenixfrom phoenix.client import Clientclient = Client()spans = client.spans.get_spans_dataframe(query=query, project_name="pydantic-evals-tutorial")spans = spans.rename(columns={"llm.input_messages": "input", "llm.output_messages": "output"})spans["input"] = spans["input"].apply(lambda x: x[1].get("message").get("content"))spans["output"] = spans["output"].apply(lambda x: x[0].get("message").get("content"))

3. Define Evaluation Dataset#

Create a dataset of test cases using Pydantic Evals:

from pydantic_evals import Case, Datasetcases = [    Case(        name="capital of France",        inputs="What is the capital of France?",        expected_output="Paris"    ),    Case(        name="author of Romeo and Juliet",        inputs="Who wrote Romeo and Juliet?",        expected_output="William Shakespeare",    ),    Case(        name="largest planet",        inputs="What is the largest planet in our solar system?",        expected_output="Jupiter",    ),]

4. Create Custom Evaluators#

Define evaluators to assess your model’s performance:

from pydantic_evals.evaluators import Evaluator, EvaluatorContextclass MatchesExpectedOutput(Evaluator[str, str]):    def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:        is_correct = ctx.expected_output == ctx.output        return is_correctclass FuzzyMatchesOutput(Evaluator[str, str]):    def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:        from difflib import SequenceMatcher        def similarity_ratio(a, b):            return SequenceMatcher(None, a, b).ratio()        # Consider it correct if similarity is above 0.8 (80%)        is_correct = similarity_ratio(ctx.expected_output, ctx.output) > 0.8        return is_correct

5. Setup Task and Dataset#

Create a task that retrieves outputs from your traced data:

import nest_asyncionest_asyncio.apply()async def task(input: str) -> str:    output = spans[spans["input"] == input]["output"].values[0]    return output# Create dataset with evaluatorsdataset = Dataset(    cases=cases,    evaluators=[MatchesExpectedOutput(), FuzzyMatchesOutput()],)

6. Add LLM Judge Evaluator#

For more sophisticated evaluation, add an LLM judge:

from pydantic_evals.evaluators import LLMJudgedataset.add_evaluator(    LLMJudge(        rubric="Output and Expected Output should represent the same answer, even if the text doesn't match exactly",        include_input=True,        model="openai:gpt-4o-mini",    ),)

7. Run Evaluation#

Execute the evaluation:

report = dataset.evaluate_sync(task)print(report)

Advanced Usage#

Upload Results to Phoenix#

Upload your evaluation results back to Phoenix for visualization:

# Extract results from the reportresults = report.model_dump()# Create dataframes for each evaluatormeo_spans = spans.copy()fuzzy_label_spans = spans.copy()llm_label_spans = spans.copy()for case in results.get("cases"):    # Extract evaluation results    meo_label = case.get("assertions").get("MatchesExpectedOutput").get("value")    fuzzy_label = case.get("assertions").get("FuzzyMatchesOutput").get("value")    llm_label = case.get("assertions").get("LLMJudge").get("value")    input = case.get("inputs")    # Update labels in dataframes    meo_spans.loc[meo_spans["input"] == input, "label"] = str(meo_label)    fuzzy_label_spans.loc[fuzzy_label_spans["input"] == input, "label"] = str(fuzzy_label)    llm_label_spans.loc[llm_label_spans["input"] == input, "label"] = str(llm_label)# Add scores for Phoenix metricsmeo_spans["score"] = meo_spans["label"].apply(lambda x: 1 if x == "True" else 0)fuzzy_label_spans["score"] = fuzzy_label_spans["label"].apply(lambda x: 1 if x == "True" else 0)llm_label_spans["score"] = llm_label_spans["label"].apply(lambda x: 1 if x == "True" else 0)# Upload to Phoenixfrom phoenix.client import Clientclient = Client()client.spans.log_span_annotations_dataframe(dataframe=meo_spans, annotation_name="Direct Match Eval", annotator_kind="CODE")client.spans.log_span_annotations_dataframe(dataframe=fuzzy_label_spans, annotation_name="Fuzzy Match Eval", annotator_kind="CODE")client.spans.log_span_annotations_dataframe(dataframe=llm_label_spans, annotation_name="LLM Match Eval", annotator_kind="LLM")

Custom Evaluation Workflows#

You can create more complex evaluation workflows by combining multiple evaluators:

from pydantic_evals.evaluators import Evaluator, EvaluatorContextfrom typing import Dict, Anyclass ComprehensiveEvaluator(Evaluator[str, str]):    def evaluate(self, ctx: EvaluatorContext[str, str]) -> Dict[str, Any]:        # Multiple evaluation criteria        exact_match = ctx.expected_output == ctx.output        # Length similarity        length_ratio = min(len(ctx.output), len(ctx.expected_output)) / max(len(ctx.output), len(ctx.expected_output))        # Semantic similarity (simplified)        from difflib import SequenceMatcher        semantic_score = SequenceMatcher(None, ctx.expected_output.lower(), ctx.output.lower()).ratio()        return {            "exact_match": exact_match,            "length_similarity": length_ratio,            "semantic_similarity": semantic_score,            "overall_score": (exact_match * 0.5) + (semantic_score * 0.3) + (length_ratio * 0.2)        }

Observe#

Once you have evaluation results uploaded to Phoenix, you can:

View evaluation metrics: See overall performance across different evaluation criteria
Analyze individual cases: Drill down into specific examples that passed or failed
Compare evaluators: Understand how different evaluation methods perform
Track improvements: Monitor evaluation scores over time as you improve your application
Debug failures: Identify patterns in failed evaluations to guide improvements

The Phoenix UI will display your evaluation results with detailed breakdowns, making it easy to understand your AI application’s performance and identify areas for improvement.