Completeness

Overview#

The Completeness evaluator classifies whether an assistant completed every active request the user made over the course of a conversation. It measures finished work — a delivered answer, a delivered artifact (including every required component), or an action whose success is visible in the record — not whether a request was merely acknowledged.

A conversation can finish its main task and still be incomplete when a secondary request is dropped. For example, if the user asks the assistant to reset a password and update a billing address, and the assistant only resets the password, the conversation is incomplete.

Supported Levels#

Relevant span kinds: AGENT, CHAIN, and LLM spans that preserve multi-turn conversation history and tool activity.

Input Requirements#

Include early-turn requests. For agent traces, include tool calls and tool results in conversation so action success can be verified. If tools are omitted, the judge falls back to the visible dialogue.

Output Interpretation#

Usage Examples#

from phoenix.evals import LLMfrom phoenix.evals.metrics import CompletenessEvaluatorevaluator = CompletenessEvaluator(    llm=LLM(provider="openai", model="gpt-4o-mini"),    temperature=0.0,)scores = evaluator.evaluate({    "conversation": (        "User: Reset my password and update the billing address.\n"        "Assistant: Your password has been reset."    ),})print(scores[0])# Score(name='completeness', score=0.0, label='incomplete', ...)

import { createCompletenessEvaluator } from "@arizeai/phoenix-evals";import { openai } from "@ai-sdk/openai";const evaluator = createCompletenessEvaluator({  model: openai("gpt-4o-mini"),});const result = await evaluator.evaluate({  conversation:    "User: Reset my password and update the billing address.\nAssistant: Your password has been reset.",});console.log(result);// { score: 0, label: "incomplete", explanation: "..." }

Using Input Mapping#

Map your trace or dataset fields into the evaluator’s conversation field.

input_mapping = {    "conversation": lambda row: render_messages(row["messages"]),}scores = evaluator.evaluate(dataset_row, input_mapping)

See Input Mapping for additional mapping options.

Viewing and Modifying the Prompt#

The default prompt is maintained in the classification evaluator config. Adapt it when your application has domain-specific notions of what counts as an intention.

import { createCompletenessEvaluator } from "@arizeai/phoenix-evals";const evaluator = createCompletenessEvaluator({  model,  promptTemplate: `Conversation: {{conversation}}Did the assistant complete every active user intention?`,  choices: { complete: 1, incomplete: 0 },});

Configuration#

For model and provider options, see Configuring the LLM. Judge model choice changes accuracy on this evaluator; see Benchmarks for a comparison of gpt-4o-mini, gpt-5.6, and claude-sonnet-4-6.

Using with Phoenix#

• Evaluating Phoenix Traces
• Logging LLM Evaluations
• Using Evaluators in Experiments

Benchmarks#

The default prompt was scored on a 45-example categorized synthetic suite. Every model used the same prompt. Runs were local with PHOENIX_TEST_TRACKING=false. See completeness.eval.ts for the example set.

On this suite, gpt-5.6 matched gold on every example. The other models mainly missed cases that mix completeness with correctness, tool evidence, or multi-part asks:

• Correctness vs completeness. Gold treats a delivered answer as complete even when it is wrong. claude-sonnet-4-6 missed on these cases.
• Claims without tool evidence. Gold treats the visible reply as sufficient when no matching tool record exists. gpt-4o-mini and claude-sonnet-4-6 missed here.
• Partial delivery. gpt-4o-mini labeled several multi-part and partial-tool cases complete.

API Reference#

• Python: CompletenessEvaluator
• TypeScript: createCompletenessEvaluator

Related#

• Correctness Evaluator
• User Friction Evaluator
• Hallucination Evaluator