User Friction

Overview#

The User Friction evaluator classifies whether the latest user message expresses friction with an assistant’s preceding behavior. It detects corrections, retries after an unsuccessful response, frustration, and challenges to unrequested or unexplained actions.

Use it to monitor conversational assistants, identify turns worth reviewing, and measure whether product changes reduce expressed user friction.

Supported Levels#

Relevant span kinds: AGENT, CHAIN, and LLM spans that preserve multi-turn conversation history.

Input Requirements#

Keep the target message separate from conversation. Include enough preceding history to distinguish retries and corrections from ordinary follow-ups. Render tool activity compactly and remove non-human payloads before evaluation.

Output Interpretation#

Usage Examples#

from phoenix.evals import LLMfrom phoenix.evals.metrics import UserFrictionEvaluatorevaluator = UserFrictionEvaluator(    llm=LLM(provider="openai", model="gpt-4o-mini"),    temperature=0.0,)scores = evaluator.evaluate({    "conversation": (        "User: Show orders from this week.\n"        "Assistant: Here are last month's orders."    ),    "user_message": "No, I asked for this week.",})print(scores[0])# Score(name='user_friction', score=1.0, label='friction', ...)

import { createUserFrictionEvaluator } from "@arizeai/phoenix-evals";import { openai } from "@ai-sdk/openai";const evaluator = createUserFrictionEvaluator({  model: openai("gpt-4o-mini"),});const result = await evaluator.evaluate({  conversation:    "User: Show orders from this week.\nAssistant: Here are last month's orders.",  userMessage: "No, I asked for this week.",});console.log(result);// { score: 1, label: "friction", explanation: "..." }

Using Input Mapping#

Map your trace or dataset fields into the evaluator’s two-field contract. The conversation should end immediately before the target user message.

input_mapping = {    "conversation": lambda row: render_messages(row["messages"][:-1]),    "user_message": lambda row: row["messages"][-1]["content"],}scores = evaluator.evaluate(dataset_row, input_mapping)

See Input Mapping for additional mapping options.

Viewing and Modifying the Prompt#

The default prompt is maintained in the classification evaluator config. Adapt it when your application has domain-specific conversational conventions.

import {  createUserFrictionEvaluator,} from "@arizeai/phoenix-evals";const evaluator = createUserFrictionEvaluator({  model,  promptTemplate: `Conversation: {{conversation}}Latest user message: {{userMessage}}Does the latest message express friction?`,  choices: { friction: 1, no_friction: 0 },});

Configuration#

For model and provider options, see Configuring the LLM.

Using with Phoenix#

• Evaluating Phoenix Traces
• Logging LLM Evaluations
• Using Evaluators in Experiments

Benchmarks#

On a 40-example public synthetic benchmark using gpt-4o-mini, the default prompt achieves 0.97 accuracy, 0.98 macro precision, 0.97 macro recall, and 0.97 macro F1. See user_friction.eval.ts for the categorized example set.

API Reference#

• Python: UserFrictionEvaluator
• TypeScript: createUserFrictionEvaluator

Related#

• Refusal Evaluator
• Correctness Evaluator