> ## Documentation index > Fetch the complete documentation index at: https://arize.com/llms.txt > Ask it with GET https://arize.com/ask?q=. Short keywords rank best. > Use these to discover all available pages before exploring further. # Evaluate > Build a dataset, write an LLM judge, and compare two prompts as experiments, in Python or TypeScript. An evaluation scores your agent's output so you can compare prompts or models and catch regressions. It has three parts: - **Dataset**: test cases, with inputs and expected outputs. - **Task**: the change you want to test, such as a new prompt. - **Evaluator**: code or an LLM judge that scores each output. The example below runs two experiments over three airline-policy questions. The first prompt scores 0; the second adds the policy and scores 1. You need an OpenAI API key. To have a coding agent or [PXI](/docs/phoenix/pxi) do this instead, see [Connect Your Coding Agent](/docs/phoenix/integrations/developer-tools/coding-agents#example-prompts). ## Run two experiments ```bash pip install arize-phoenix-client arize-phoenix-evals openai export OPENAI_API_KEY= export PHOENIX_ENDPOINT=http://localhost:6006 ``` Save as `evaluate.py` and run `python evaluate.py`. {/* snippet: examples/quickstarts/python/evaluate.py#main */} ```python from openai import OpenAI from phoenix.client import Client from phoenix.evals import LLM, bind_evaluator, create_classifier phoenix = Client() # reads PHOENIX_ENDPOINT, default http://localhost:6006 openai = OpenAI() dataset = phoenix.datasets.create_dataset( name="travel-support", examples=[ { "input": {"question": "How long do I have to cancel a Basic fare for a full refund?"}, "output": { "expected_answer": "Basic fares can be cancelled for a full refund within 48 hours of booking." }, }, { "input": {"question": "Can I change the date on a Basic fare, and what does it cost?"}, "output": { "expected_answer": "Basic fares can be changed once for a 40 dollar fee, plus any difference in price." }, }, { "input": { "question": "How many checked bags come with a Basic fare, and what does a second bag cost?" }, "output": { "expected_answer": "Basic fares include one checked bag. A second bag is 45 dollars." }, }, ], ) judge = create_classifier( name="correctness", llm=LLM(provider="openai", model="gpt-5-mini"), prompt_template="""Compare the provided answer to the reference answer. Label true if it states the same policy: same time windows, fees, and quantities, in any wording. Label false if it contradicts the reference, hedges, or leaves out a fee, limit, or condition. Reference answer: {{reference}} Provided answer: {{output}}""", choices={"true": 1.0, "false": 0.0}, ) # The expected output is {"expected_answer": ...}; hand the judge just the answer. correctness = bind_evaluator( evaluator=judge, input_mapping={"reference": "reference.expected_answer"} ) def run(prompt, name): def task(input): response = openai.chat.completions.create( model="gpt-5-mini", messages=[{"role": "user", "content": prompt.format(**input)}], ) return response.choices[0].message.content return phoenix.experiments.run_experiment( dataset=dataset, task=task, evaluators=[correctness], experiment_name=name ) BASELINE = "You are a travel support agent. Answer the customer's question.\n\nQuestion: {question}" POLICY = """You are a travel support agent. Answer the customer's question using this policy. Basic fare policy: - Full refund if cancelled within 48 hours of booking. - Can be changed once for a 40 dollar fee, plus any price difference. - Includes one checked bag. A second bag is 45 dollars. Question: {question}""" run(BASELINE, "baseline") run(POLICY, "with-policy") ``` ```bash npm install @arizeai/phoenix-client @arizeai/phoenix-evals ai @ai-sdk/openai tsx export OPENAI_API_KEY= export PHOENIX_ENDPOINT=http://localhost:6006 ``` Save as `evaluate.ts` and run `npx tsx evaluate.ts`. {/* snippet: examples/quickstarts/typescript/evaluate.ts#main */} ```typescript import { openai } from "@ai-sdk/openai"; import { generateText } from "ai"; import { createClient } from "@arizeai/phoenix-client"; import { createDataset } from "@arizeai/phoenix-client/datasets"; import { asExperimentEvaluator, runExperiment } from "@arizeai/phoenix-client/experiments"; import { createClassificationEvaluator } from "@arizeai/phoenix-evals/llm/createClassificationEvaluator"; const client = createClient(); // reads PHOENIX_ENDPOINT, default http://localhost:6006 const model = openai("gpt-5-mini"); const dataset = await createDataset({ client, name: "travel-support", description: "Questions about a fictional airline's Basic fare.", examples: [ { input: { question: "How long do I have to cancel a Basic fare for a full refund?" }, output: { expected_answer: "Basic fares can be cancelled for a full refund within 48 hours of booking." }, }, { input: { question: "Can I change the date on a Basic fare, and what does it cost?" }, output: { expected_answer: "Basic fares can be changed once for a 40 dollar fee, plus any difference in price." }, }, { input: { question: "How many checked bags come with a Basic fare, and what does a second bag cost?" }, output: { expected_answer: "Basic fares include one checked bag. A second bag is 45 dollars." }, }, ], }); const judge = createClassificationEvaluator({ name: "correctness", model, choices: { true: 1, false: 0 }, promptTemplate: `Compare the provided answer to the reference answer. Label true if it states the same policy: same time windows, fees, and quantities, in any wording. Label false if it contradicts the reference, hedges, or leaves out a fee, limit, or condition. Reference answer: {{reference}} Provided answer: {{output}}`, }); const correctness = asExperimentEvaluator({ name: "correctness", kind: "LLM", evaluate: ({ output, expected }) => judge.evaluate({ reference: (expected as { expected_answer: string }).expected_answer, output: String(output), }), }); async function run(prompt: string, experimentName: string) { await runExperiment({ client, dataset, experimentName, evaluators: [correctness], task: async (example) => { const { question } = example.input as { question: string }; const { text } = await generateText({ model, prompt: prompt.replace("{question}", question), }); return text; }, }); } const BASELINE = "You are a travel support agent. Answer the customer's question.\n\nQuestion: {question}"; const POLICY = `You are a travel support agent. Answer the customer's question using this policy. Basic fare policy: - Full refund if cancelled within 48 hours of booking. - Can be changed once for a 40 dollar fee, plus any price difference. - Includes one checked bag. A second bag is 45 dollars. Question: {question}`; await run(BASELINE, "baseline"); await run(POLICY, "with-policy"); ``` ## Compare the runs Open the dataset's **Experiments** tab to compare the two runs side by side. A higher score on the same dataset and judge means the change helped; a run that still scores 0 has the judge's explanation attached. ## Learn more - [Datasets and Experiments](/docs/phoenix/datasets-and-experiments/overview-datasets): building datasets, comparing variants, and tracking quality over time. - [Server Evals](/docs/phoenix/evaluation/server-evals/overview): evaluators you manage in the UI, including templates and code evaluators. - [Evaluate a Prompt Change in Code](/docs/phoenix/datasets-and-experiments/how-to-experiments/evaluate-a-prompt-change): the same loop with a deterministic code evaluator, example ids and metadata, and a dataset you built in the UI. - [Evaluation](/docs/phoenix/evaluation/evals): writing your own evaluators in code. The code on this page lives in [examples/quickstarts](https://github.com/Arize-ai/phoenix/tree/main/examples/quickstarts), ready to run.