Precision / Recall / F-Score

Overview#

The PrecisionRecallFScore evaluator computes precision, recall, and F-beta scores for comparing predicted labels against expected labels. It supports both binary and multi-class classification with various averaging strategies, and is available in Python and TypeScript.

When to Use#

Use the PrecisionRecallFScore evaluator when you need to:

• Evaluate classification performance - Measure how well your model predicts correct labels
• Compare label sequences - Assess predicted vs expected labels for multi-item outputs
• Binary classification metrics - Compute metrics for spam/ham, positive/negative, etc.
• Multi-class evaluation - Evaluate across multiple categories with different averaging strategies

How These Metrics Work#

Every prediction for a given class falls into one of four buckets: true positive (TP, correctly predicted that class), false positive (FP, predicted that class but it wasn’t), false negative (FN, was that class but predicted something else), and true negative (not that class and not predicted as it). Precision, recall, and F-beta are all built from TP, FP, and FN:

Precision and recall trade off against each other: a model that predicts the positive class more aggressively tends to raise recall (catches more true positives) at the cost of precision (more false alarms), and vice versa. Which one matters more is a property of the task, not the model:

• Spam filtering — a false positive (a real email marked as spam) is usually worse than a false negative (spam that slips through), so precision matters more.
• Medical screening / fraud detection — a false negative (a missed disease or fraudulent transaction) is usually worse than a false positive (an unnecessary follow-up), so recall matters more.

F-beta (the F-measure introduced by Van Rijsbergen, see References) combines them into a single number, where beta sets how much more recall is weighted than precision: beta = 1 (F1, the default) weights them equally, beta > 1 (e.g. F2) favors recall, and beta < 1 (e.g. F0.5) favors precision. Because it’s a harmonic mean, F-beta stays low if either precision or recall is low.

Averaging Strategies#

Precision, recall, and F-beta are inherently per-class metrics. For multi-class classification, average controls how the per-class scores combine into a single number:

macro and weighted compute F-beta per class first and then average the per-class F-beta values — they don’t derive F-beta from the averaged precision and recall. This matches scikit-learn’s precision_recall_fscore_support semantics (see References), so results are directly comparable to the equivalent sklearn.metrics call.

Supported Levels#

This evaluator is not tied to specific tracing levels. It operates on lists of predicted and expected labels, making it useful for:

• Comparing model predictions against ground truth labels
• Evaluating classification outputs at any level where you have paired label sequences
• Batch evaluation of classification tasks in experiments

Input Requirements#

The evaluator requires two inputs:

Both sequences must have the same length and contain at least one element.

Constructor Arguments#

Output Interpretation#

Usage Examples#

from phoenix.evals.metrics import PrecisionRecallFScore# Create evaluator with default settings (F1, macro averaging)evaluator = PrecisionRecallFScore()# Inspect the evaluator's requirementsprint(evaluator.describe())# Multi-class evaluationeval_input = {    "expected": ["cat", "dog", "cat", "bird", "dog"],    "output": ["cat", "cat", "cat", "bird", "dog"]}scores = evaluator.evaluate(eval_input)for score in scores:    print(f"{score.name}: {score.score:.3f}")# precision: 0.889# recall: 0.833# f1: 0.822

import {  createPrecisionRecallFScoreEvaluators,} from "@arizeai/phoenix-evals/code";// Matching precision/recall/F1 evaluators sharing one options object// (F1, macro averaging by default)const { precision, recall, fScore } = createPrecisionRecallFScoreEvaluators();// Multi-class evaluationconst example = {  expected: ["cat", "dog", "cat", "bird", "dog"],  output: ["cat", "cat", "cat", "bird", "dog"],};console.log("precision:", (await precision.evaluate(example)).score?.toFixed(3));console.log("recall:", (await recall.evaluate(example)).score?.toFixed(3));console.log("f1:", (await fScore.evaluate(example)).score?.toFixed(3));// precision: 0.889// recall: 0.833// f1: 0.822

Binary Classification#

For binary classification, specify the positive label:

from phoenix.evals.metrics import PrecisionRecallFScore# Binary classification for spam detectionevaluator = PrecisionRecallFScore(positive_label="spam")eval_input = {    "expected": ["spam", "ham", "spam", "ham", "spam"],    "output": ["spam", "spam", "ham", "ham", "spam"]}scores = evaluator.evaluate(eval_input)for score in scores:    print(f"{score.name}: {score.score:.3f}")# precision: 0.667  (2 TP / 3 predicted spam)# recall: 0.667     (2 TP / 3 actual spam)# f1: 0.667

import {  createPrecisionEvaluator,  createRecallEvaluator,  createF1Evaluator,} from "@arizeai/phoenix-evals/code";// Binary classification for spam detectionconst precision = createPrecisionEvaluator({ positiveLabel: "spam" });const recall = createRecallEvaluator({ positiveLabel: "spam" });const f1 = createF1Evaluator({ positiveLabel: "spam" });const example = {  expected: ["spam", "ham", "spam", "ham", "spam"],  output: ["spam", "spam", "ham", "ham", "spam"],};console.log("precision:", (await precision.evaluate(example)).score?.toFixed(3));console.log("recall:", (await recall.evaluate(example)).score?.toFixed(3));console.log("f1:", (await f1.evaluate(example)).score?.toFixed(3));// precision: 0.667  (2 TP / 3 predicted spam)// recall: 0.667     (2 TP / 3 actual spam)// f1: 0.667

Using with Phoenix#

Evaluating Traces#

Run evaluations on traces collected in Phoenix and log results as annotations:

• Evaluating Phoenix Traces

Running Experiments#

Use the PrecisionRecallFScore evaluator in Phoenix experiments:

• Using Evaluators in Experiments

References#

• Van Rijsbergen, C.J. (1979). Information Retrieval (2nd ed.). Butterworth-Heinemann. — origin of the F-measure and its beta parameter.
• Precision and recall — Wikipedia overview of the underlying concepts and terminology.
• sklearn.metrics.precision_recall_fscore_support — scikit-learn’s reference implementation; the average strategies here follow the same semantics.

API Reference#

• Python: PrecisionRecallFScore
• TypeScript: Classification Metrics

Related#

• Exact Match Evaluator - For exact string comparison
• Correctness Evaluator - For semantic correctness evaluation