Correctness - Phoenix

Overview#

The Correctness evaluator assesses whether an LLM’s response is factually accurate, complete, and logically consistent. It evaluates the quality of answers without requiring external context or reference responses.

When to Use#

Use the Correctness evaluator when you need to:

Validate factual accuracy - Ensure responses contain accurate information
Check answer completeness - Verify responses address all parts of the question
Detect logical inconsistencies - Identify contradictions within responses
Evaluate general knowledge responses - Assess answers that don’t rely on retrieved context
Get a quick gut-check - Capture a wide range of potential problems quickly

Info

For evaluating responses against retrieved documents, use the Faithfulness evaluator instead. Correctness is best suited for evaluating general knowledge.

Supported Levels#

The level of an evaluator determines the scope of the evaluation in OpenTelemetry terms. Some evaluations are applicable to individual spans, some to full traces or sessions, and some are applicable at multiple levels.

Level	Supported	Notes
Span	Yes	Apply to LLM spans where you want to evaluate the response quality.
Trace	Yes	Evaluate the final response of the entire trace.
Session	Yes	Evaluate responses across a conversation session.

Relevant span kinds: LLM spans, particularly ones where the LLM response is not grounded in retrieved context.

Input Requirements#

The Correctness evaluator requires two inputs:

Field	Type	Description
`input`	`string`	The user’s query or question
`output`	`string`	The LLM’s response to evaluate

Formatting Tips#

For best results:

Use human-readable strings rather than raw JSON for all inputs

For multi-turn conversations, format input as a readable conversation:

User: What is the capital of France?Assistant: Paris is the capital of France.User: What is its population?

Output Interpretation#

The evaluator returns a Score object with the following properties:

Property	Value	Description
`label`	`"correct"` or `"incorrect"`	Classification result
`score`	`1.0` or `0.0`	Numeric score (1.0 = correct, 0.0 = incorrect)
`explanation`	`string`	LLM-generated reasoning for the classification
`direction`	`"maximize"`	Higher scores are better
`metadata`	`object`	Additional information such as the model name. When tracing is enabled, includes the `trace_id` for the evaluation.

Interpretation:

Correct (1.0): The response is factually accurate, complete, and logically consistent
Incorrect (0.0): The response contains factual errors, is incomplete, or has logical inconsistencies

Usage Examples#

from phoenix.evals import LLMfrom phoenix.evals.metrics import CorrectnessEvaluator# Initialize the LLM clientllm = LLM(provider="openai", model="gpt-4o")# Create the evaluatorcorrectness_eval = CorrectnessEvaluator(llm=llm)# Inspect the evaluator's requirementsprint(correctness_eval.describe())# Evaluate a single exampleeval_input = {    "input": "What is the capital of France?",    "output": "Paris is the capital of France."}scores = correctness_eval.evaluate(eval_input)print(scores[0])# Score(name='correctness', score=1.0, label='correct', ...)

import { createCorrectnessEvaluator } from "@arizeai/phoenix-evals";import { openai } from "@ai-sdk/openai";// Create the evaluatorconst correctnessEvaluator = createCorrectnessEvaluator({  model: openai("gpt-4o"),});// Evaluate an exampleconst result = await correctnessEvaluator.evaluate({  input: "What is the capital of France?",  output: "Paris is the capital of France.",});console.log(result);// { score: 1, label: "correct", explanation: "..." }

Using Input Mapping#

When your data has different field names or requires transformation, use input mapping.

from phoenix.evals import LLMfrom phoenix.evals.metrics import CorrectnessEvaluatorllm = LLM(provider="openai", model="gpt-4o")correctness_eval = CorrectnessEvaluator(llm=llm)# Example with different field nameseval_input = {    "question": "What is the speed of light?",    "answer": "The speed of light is approximately 299,792 km/s."}# Use input mapping to match expected field namesinput_mapping = {    "input": "question",    "output": "answer"}scores = correctness_eval.evaluate(eval_input, input_mapping)

For more details on input mapping options, see Input Mapping.

import { bindEvaluator, createCorrectnessEvaluator } from "@arizeai/phoenix-evals";import { openai } from "@ai-sdk/openai";const correctnessEvaluator = createCorrectnessEvaluator({  model: openai("gpt-4o"),});// Bind with input mapping for different field namesconst boundEvaluator = bindEvaluator(correctnessEvaluator, {  inputMapping: {    input: "question",    output: "answer",  },});const result = await boundEvaluator.evaluate({  question: "What is the speed of light?",  answer: "The speed of light is approximately 299,792 km/s.",});

For more details on input mapping options, see Input Mapping.

Configuration#

For LLM client configuration options, see Configuring the LLM.

Viewing and Modifying the Prompt#

You can view the latest versions of our prompt templates on GitHub. The evaluators are designed to work well in a variety of contexts, but we highly recommend modifying the prompt to be more specific to your use case. Feel free to adapt them.

from phoenix.evals.metrics import CorrectnessEvaluatorfrom phoenix.evals import LLM, ClassificationEvaluatorllm = LLM(provider="openai", model="gpt-4o")evaluator = CorrectnessEvaluator(llm=llm)# View the prompt templateprint(evaluator.prompt_template)# Create a custom evaluator based on the built-in templatecustom_evaluator = ClassificationEvaluator(    name="correctness",    prompt_template=evaluator.prompt_template,  # Modify as needed    llm=llm,    choices={"correct": 1.0, "incorrect": 0.0},    direction="maximize",)

import { CORRECTNESS_CLASSIFICATION_EVALUATOR_CONFIG, createCorrectnessEvaluator } from "@arizeai/phoenix-evals";import { openai } from "@ai-sdk/openai";// View the prompt templateconsole.log(CORRECTNESS_CLASSIFICATION_EVALUATOR_CONFIG.template);// Create a custom evaluator with a modified templateconst customEvaluator = createCorrectnessEvaluator({  model: openai("gpt-4o"),  promptTemplate: CORRECTNESS_CLASSIFICATION_EVALUATOR_CONFIG.template, // Modify as needed});

Using with Phoenix#

Evaluating Traces#

Run evaluations on traces collected in Phoenix and log results as annotations:

Running Experiments#

Use the Correctness evaluator in Phoenix experiments:

Using Evaluators in Experiments

API Reference#

Python: CorrectnessEvaluator
TypeScript: createCorrectnessEvaluator

Faithfulness Evaluator - For evaluating responses against retrieved context
Tool Selection Evaluator - For evaluating LLM tool selection accuracy

Overview#

When to Use#

Supported Levels#

Input Requirements#

Formatting Tips#

Output Interpretation#

Usage Examples#

Using Input Mapping#

Configuration#

Viewing and Modifying the Prompt#

Using with Phoenix#

Evaluating Traces#

Running Experiments#

API Reference#

Related#