Skip to content

LLM Client Autotracing

Tracy provides automatic tracing for several popular LLM clients and HTTP libraries. By instrumenting these clients, you can capture detailed information about every request and response, including prompts, completions, token usage, and execution time, without manually creating spans.

How it works

To enable autotracing, you use the instrument() function provided by the specific module for the client you are using. This function modifies your client instance in-place to record OpenTelemetry spans for every API call.

Supported Clients

OpenAI

To trace the OpenAI client, use the instrument function from the tracy.openai module.

val instrumentedClient = OpenAIOkHttpClient.builder()
    .apiKey("api-token")
    .build()
    .apply { instrument(this) }

See the full example: OpenAIClientAutotracingExample.kt

Anthropic

To trace the Anthropic client, use the instrument function from the tracy.anthropic module.

val instrumentedClient = AnthropicOkHttpClient.builder()
    .apiKey("api-token")
    .build()
    .apply { instrument(this) }

See the full example: AnthropicClientAutotracingExample.kt

Google Gemini

To trace the Gemini client, use the instrument function from the tracy.gemini module.

val instrumentedClient = Client.builder()
    .apiKey("api-token")
    .build()
    .apply { instrument(this) }

See the full example: GeminiClientAutotracingExample.kt

Ktor HTTP Client

If you are using Ktor's HttpClient for manual LLM calls or custom integrations, you can instrument it using the tracy.ktor module. Although you need to explicitly pass an LLM-specific adapter that parses internal structures of requests and responses of the given LLM provider (see instrument). The usage is similar to other clients:

val client = HttpClient(CIO)
val instrumentedClient = instrument(client, adapter = OpenAILLMTracingAdapter())

Currently, Tracy supports the following LLM providers and provides the corresponding adapters (see the implementation of LLMTracingAdapter and its inheritors):

  1. OpenAI: OpenAILLMTracingAdapter.
  2. Anthropic: AnthropicLLMTracingAdapter.
  3. Gemini: GeminiLLMTracingAdapter.

See the full example: KtorClientAutotracingExample.kt

OkHttp

For applications using OkHttp, Tracy provides an interceptor-based approach via the tracy.core module (as it's used by many SDKs). Same as with Ktor, you need to pass an LLM-specific adapter to the instrument function:

val okHttpClient = OkHttpClient.Builder().build()
val instrumentedClient = instrument(
    okHttpClient, adapter = OpenAILLMTracingAdapter()
)

See the full example: OkHttpClientAutotracingExample.kt

What is Captured?

Tracy follows the OpenTelemetry Generative AI Semantic Conventions. The following attributes are typically captured (depending on the client and provider):

  • gen_ai.system: The AI system name (e.g., "openai").
  • gen_ai.request.model: The name of the model requested.
  • gen_ai.request.temperature: Sampling temperature.
  • gen_ai.response.model: The model that actually generated the response.
  • gen_ai.usage.input_tokens: Number of tokens in the prompt.
  • gen_ai.usage.output_tokens: Number of tokens in the completion.
  • gen_ai.prompt.[index].content: The content of the prompt (if capturing sensitive data is enabled).
  • gen_ai.completion.[index].content: The content of the completion (if capturing sensitive data is enabled).

For more information on how to control what content is captured, see the Configuration & Sensitivity page.