Phoenix provides pre-built evaluation metrics that can be used out of the box to assess LLM application quality. These metrics are available in both Python and TypeScript and are designed to work seamlessly with Phoenix’s tracing and experiment infrastructure.
All LLM evaluation templates are tested against golden datasets and achieve an F1 score of 85% or higher on benchmarks.
LLM Evaluators#
LLM evaluators use a judge model to assess the quality of outputs. These are useful for subjective or nuanced evaluations where simple rules don’t suffice.
Measures whether a response is faithful to (grounded in) the provided context. Detects hallucinations and unsupported claims.
Detects claims in a response that are unsupported by, or contradict, the conversation. The conversation-level counterpart to Faithfulness.
Checks whether every active user request in a conversation was actually completed, not merely acknowledged.
Evaluates whether a response is concise and free of unnecessary content like filler, hedging, and meta-commentary.
Evaluates the general correctness of an LLM response.
Assesses whether externally retrieved information is relevant to the request, from any source: RAG, tools, MCP, or web search.
Determines whether the correct tool was selected for a given context from the available options.
Checks if a tool was invoked correctly with proper arguments, formatting, and safe content.
Evaluates whether an agent correctly processed a tool’s result, including error handling, data extraction, and safe information disclosure.
Detects when an LLM refuses, declines, or avoids answering a user query.
Detects corrections, retries, frustration, and challenges expressed in a user’s follow-up message.
Screens a conversation record for personally identifiable information.
Detect hateful, demeaning, abusive, or threatening text in model outputs or user inputs.
Code Evaluators#
Code evaluators use deterministic logic for evaluation. These are faster, cheaper, and provide consistent results for objective criteria.
Looking to create custom evaluators? See the Building Custom Evaluators guide.