12.03.2025: TypeScript createEvaluator

TypeScript createEvaluator#

The createEvaluator utility in @arizeai/phoenix-evals provides a type-safe way to build custom code evaluators for experiments in TypeScript. Define evaluators with full type inference for inputs, outputs, and expected values.

Basic Usage#

Create simple evaluators that validate experiment outputs:

import { createEvaluator } from "@arizeai/phoenix-evals";const inBounds = createEvaluator<{ output: number }>(  ({ output }) => {    return 1 <= output && output <= 100 ? 1 : 0;  },  { name: "in_bounds" });

Multiple Parameters#

Access input, output, expected, and metadata in your evaluator:

import { createEvaluator } from "@arizeai/phoenix-evals";import { distance } from "fastest-levenshtein";const editDistance = createEvaluator<{ output: string; expected: string }>(  ({ output, expected }) => distance(output, expected),  { name: "edit_distance" });

Evaluator Options#

Customize display properties for better integration with the Experiments UI:

const containsLink = createEvaluator<{ output: string }>(  ({ output }) => /https?:\/\/[^\s]+/.test(output) ? 1 : 0,  { name: "contains_link", kind: "CODE" });

Running in Experiments#

Pass evaluators directly to runExperiment:

import { runExperiment } from "@arizeai/phoenix-client/experiments";import { createEvaluator } from "@arizeai/phoenix-evals";const hasGreeting = createEvaluator<{ output: string }>(  ({ output }) =>     ["hello", "hi", "hey"].some(w => output.toLowerCase().includes(w)) ? 1 : 0,  { name: "has_greeting", kind: "CODE" });const exactMatch = createEvaluator<{ output: string; expected: string }>(  ({ output, expected }) => output.trim() === expected.trim() ? 1 : 0,  { name: "exact_match", kind: "CODE" });const experiment = await runExperiment({  dataset: myDataset,  task: myTask,  evaluators: [hasGreeting, exactMatch],});

More Information:#

Using Evaluators Documentation