OpenLLMetry is a set of extensions built on OpenTelemetry that adds observability to LLM applications. Because it emits standard OpenTelemetry data, traces can be sent to an existing observability stack such as Datadog, Honeycomb, Grafana, or an OpenTelemetry Collector rather than a dedicated LLM tool.
The Python repository contains OpenTelemetry instrumentations for LLM providers, vector databases, and agent frameworks, plus a Traceloop SDK that initializes everything with a single Traceloop.init() call. If an application already uses OpenTelemetry, individual instrumentations can be added directly. A separate OpenLLMetry-JS project covers JavaScript and TypeScript. The project's semantic conventions for LLM spans have been contributed to OpenTelemetry.
| Instrumented | Examples |
|---|---|
| LLM providers | Aleph Alpha, Anthropic, AWS Bedrock, Cohere, Google Gemini, Groq, Hugging Face, IBM Watsonx, Mistral AI, Ollama, OpenAI and Azure OpenAI, Replicate, AWS SageMaker, Together AI, Vertex AI, WRITER |
| Vector databases | Chroma, LanceDB, Marqo, Milvus, Pinecone, Qdrant, Weaviate |
| Frameworks | Agno, AWS Strands, CrewAI, Haystack, LangChain, Langflow, LangGraph, LiteLLM, LlamaIndex, OpenAI Agents |
| Protocols | MCP |
Features
- OpenTelemetry-native: standard span data that plugs into existing tracing pipelines
- Destinations: Traceloop, Axiom, Azure Application Insights, Braintrust, Dash0, Datadog, Dynatrace, Google Cloud, Grafana, Honeycomb, HyperDX, IBM Instana, New Relic, OpenTelemetry Collector, Sentry, SigNoz, Splunk, and others
- Provider instrumentation: automatic spans for calls to OpenAI, Anthropic, Bedrock, Gemini, Mistral, Ollama, and more
- Vector database instrumentation: Chroma, Pinecone, Qdrant, Weaviate, Milvus, LanceDB, and Marqo
- Framework instrumentation: LangChain, LangGraph, LlamaIndex, CrewAI, Haystack, Agno, LiteLLM, and OpenAI Agents
- Traceloop SDK: one-line initialization with an option to disable batching for immediate local traces
- Standard OpenTelemetry coverage: databases, API calls, and anything else OpenTelemetry already instruments
- JS/TS variant: OpenLLMetry-JS for JavaScript and TypeScript applications
