Together AI Tracing

Together AI provides fast inference for a wide range of open-source models. The Together and AsyncTogether chat completions clients can be instrumented using the openinference-instrumentation-together package, which traces calls as OpenInference LLM spans.

Note

This integration requires together >= 2.0.0.

Install#

pip install openinference-instrumentation-together "together>=2.0.0"

Setup#

Set the TOGETHER_API_KEY environment variable to authenticate calls made using the SDK.

export TOGETHER_API_KEY=[your_key_here]

Connect your application to Phoenix with the register function:

from phoenix.otel import register# configure the Phoenix tracertracer_provider = register(project_name="my-llm-app", # Default is 'default'auto_instrument=True # Auto-instrument your app based on installed OI dependencies)

auto_instrument=True picks up the installed openinference-instrumentation-together package automatically. To instrument explicitly instead, call TogetherInstrumentor().instrument(tracer_provider=tracer_provider).

Run Together AI#

A simple chat completion that is now instrumented:

from together import Togetherclient = Together()response = client.chat.completions.create(    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",    messages=[{"role": "user", "content": "Why is the sky blue?"}],)print(response.choices[0].message.content)

Streaming#

When stream=True, the span stays open until the stream is fully consumed, then records the accumulated output, tool calls, and token counts from the chunks.

stream = client.chat.completions.create(    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",    messages=[{"role": "user", "content": "Write a haiku about observability."}],    stream=True,)for chunk in stream:    if chunk.choices and chunk.choices[0].delta.content:        print(chunk.choices[0].delta.content, end="", flush=True)

Tool calls#

Tools passed on the request (llm.tools.*) and any tool calls returned on the response (message.tool_calls.*) are captured on the span.

tools = [    {        "type": "function",        "function": {            "name": "get_current_weather",            "description": "Get the current weather in a given city",            "parameters": {                "type": "object",                "properties": {                    "city": {"type": "string", "description": "The city, e.g. Paris"},                },                "required": ["city"],            },        },    }]response = client.chat.completions.create(    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",    messages=[{"role": "user", "content": "What is the weather in Paris right now?"}],    tools=tools,)message = response.choices[0].messageif message.tool_calls:    for tool_call in message.tool_calls:        print(f"tool call: {tool_call.function.name}({tool_call.function.arguments})")else:    print(message.content)

Observe#

Now that you have tracing setup, all invocations of Together AI chat completions (sync and async, streaming and non-streaming) will be streamed to your running Phoenix for observability and evaluation.

Install#

Setup#

Run Together AI#

Streaming#

Tool calls#

Observe#

Resources#