Amazon Bedrock Evals

Using Amazon Bedrock with Phoenix Evals#

Info

Requires litellm and AWS credentials configured via environment variables or boto3.

pip install "arize-phoenix-evals>=3" litellm boto3

Use the "litellm" provider with a Bedrock model ID:

from phoenix.evals import LLMllm = LLM(provider="litellm", model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0")

LiteLLM reads AWS credentials from the standard environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION_NAME).

Using with evaluators#

from phoenix.evals import LLM, evaluate_dataframefrom phoenix.evals.metrics import FaithfulnessEvaluatorllm = LLM(provider="litellm", model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0")evaluator = FaithfulnessEvaluator(llm=llm)results_df = evaluate_dataframe(dataframe=df, evaluators=[evaluator])

Authentication with assumed roles#

If you need to assume a role, configure your credentials via environment variables before creating the LLM:

import osimport boto3from phoenix.evals import LLMsts_client = boto3.client("sts")response = sts_client.assume_role(    RoleArn="arn:aws:iam::123456789012:role/MyRole",    RoleSessionName="PhoenixEvalsSession",)creds = response["Credentials"]os.environ["AWS_ACCESS_KEY_ID"] = creds["AccessKeyId"]os.environ["AWS_SECRET_ACCESS_KEY"] = creds["SecretAccessKey"]os.environ["AWS_SESSION_TOKEN"] = creds["SessionToken"]llm = LLM(provider="litellm", model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0")