llm-keypool is a key pool manager for free-tier LLM APIs. You register API keys from several providers once, and it round-robins requests across them, puts a key into cooldown when it returns a 429, and retries on a different key. It exposes a CLI, a Textual terminal UI, a LangChain drop-in chat model (AggregatorChat), and a local OpenAI-compatible proxy so any agent or tool that speaks the OpenAI API can use the pool without code changes.
Keys carry capability tags (agentic, fast, generalpurpose, code, vision, largecontext), and each proxy instance serves only the capabilities it is started with. A two-tier setup runs one proxy for agentic traffic and another for fast general-purpose traffic. The documented Hermes Agent integration points the main loop at one proxy and delegate calls at the other. Each key uses its own assigned model, so the model field a client sends is ignored.
Cooldown timestamps are derived from provider rate-limit headers where available (Groq, Cerebras, Mistral) and otherwise fall back to a config-driven strategy in providers.json, such as next UTC midnight for OpenRouter and Cloudflare or a rolling 65 seconds for SambaNova. State lives in a SQLite database in WAL mode, so rotation position and cooldowns survive restarts.
Features
- Multi-provider pooling: keys for Groq, Cerebras, Mistral, OpenRouter, SambaNova, Google, Cohere, Cloudflare, Jina, Hugging Face, and more
- Capability tagging: tag keys with one or more capabilities and filter per proxy instance or per request via X-Keypool-Capabilities
- Automatic rotation: round-robin across keys, rotating every N requests (default 5)
- 429 handling: rate-limited keys enter a header-derived or config-driven cooldown while the next call retries another key
- OpenAI-compatible proxy: POST /v1/chat/completions with SSE streaming, GET /v1/models, GET /health, and GET /audit
- Audit log: every call recorded with subscriber ID, provider, model, token counts, and latency, queryable with llm-keypool audit
- Subscriber tracking: attribute calls with an X-Subscriber-ID header or subscriber_id= parameter
- LangChain integration: AggregatorChat as a BaseChatModel with sync and async invocation, chains, and LangSmith tracing
- Think-token stripping: removes ... blocks from reasoning model output
- Terminal UI: tabular key view with inline deactivate, clear-cooldown, and add-key form
- Persistent state: SQLite in WAL mode at a configurable database path
