GPTCache

GPTCache stores model responses so that repeated or similar questions are answered from a cache instead of another API call. Exact-match caching works poorly for language-model traffic, because the same question arrives phrased many ways, so GPTCache converts a query into an embedding and runs a similarity search over stored queries in a vector store; a close enough match returns the earlier response. The effects are fewer billed requests, faster answers for cached queries, headroom against provider rate limits, and a local stand-in for the provider while developing and testing.

The design is modular, and each module has several interchangeable implementations. An adapter presents the provider's own interface so existing calls keep their shape; an embedding generator produces the vectors; cache storage holds the responses; a vector store finds the nearest stored queries; a cache manager coordinates storage and eviction; and a similarity evaluator decides whether a candidate counts as a hit, using vector distance, an exact comparison, a small classifier model, or a norm over the embeddings.

Because a semantic cache can return false positives on hits and false negatives on misses, three metrics are offered for tuning — hit ratio, latency and recall — and a sample benchmark is included. A temperature parameter between 0 and 2 controls how often a request skips the cache and goes straight to the model: at 0 the cache is always searched first, at 2 it is always bypassed. A server image makes the cache reachable from any language, and integrations exist for LangChain and LlamaIndex. New provider APIs and models are no longer being added; a generic get and set interface is the recommended path for them.

Module	Implementations
Cache storage	SQLite, DuckDB, PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, DynamoDB
Vector store	Milvus, Zilliz Cloud, Milvus Lite, FAISS, Hnswlib, PGVector, Chroma, DocArray, Qdrant, Weaviate
Embeddings	OpenAI, ONNX, Hugging Face transformers, Cohere, fastText, SentenceTransformers, Timm
Eviction	LRU, FIFO, LFU and random replacement in memory; Redis or memcached when distributed
Adapters	OpenAI chat, LangChain, MiniGPT-4, llama.cpp, Dolly, plus image generation and audio transcription

Features

• Semantic matching: queries are embedded and matched by similarity, so paraphrases of a cached question still hit
• Drop-in adapters: the cache presents the provider's own interface, so existing calls are unchanged
• Pluggable modules: embeddings, storage, vector store, eviction and similarity evaluation are each swappable
• Temperature control: a 0-to-2 parameter sets how often a request bypasses the cache entirely
• Tuning metrics: hit ratio, latency and recall, with a sample benchmark to measure them
• Distributed caching: Redis or memcached keep cache state consistent across replicas
• Multimodal adapters: experimental support for image generation and audio transcription APIs
• Language-agnostic server: a server image exposes the cache over the network for non-Python callers
• Framework integrations: LangChain and LlamaIndex, with worked examples for question answering, SQL chains and web-page question answering