CocoIndex

CocoIndex is a Python framework for building data pipelines that keep an index or other target store continuously in sync with its sources, so AI agents and LLM applications read current context rather than a stale batch. Sources include codebases, PDFs, meeting notes, inboxes, Slack, and videos; targets include relational databases, data warehouses, vector databases, graph databases, message queues, and feature stores.

The programming model is declarative: you write a Python transformation that declares what the target should contain, and the engine keeps that target consistent with the latest source data and code. The mental model is Target = F(Source), described by the project as React for data engineering. Functions decorated with @coco.fn(memo=True) are cached by a hash of their input and their code, so when a source file changes only the affected records are reprocessed, and when the transformation code changes only the outputs that depend on the changed code re-run.

The engine tracks per-row provenance end to end. When a source changes it identifies the affected records, propagates the change across joins and lookups, updates the target, and retires stale rows without touching unchanged data. The core is written in Rust, with parallel chunking and failure isolation so one bad record does not stall the flow; the Python API targets Python 3.10 to 3.13.

CocoIndex-code is an MCP (Model Context Protocol) server built with the framework for AI coding agents such as Claude Code. It maintains an AST-aware, incremental semantic code index with call graphs, symbols, vectors, and chunks for Python, TypeScript, Rust, and Go. A CocoIndex skill file is also provided so a coding agent can write correct v1 pipeline code.

Features

• Incremental processing: only the delta is recomputed after a source or code change, with memoisation keyed on input hash plus code hash
• Declarative Python flows: pipelines are ordinary Python functions that declare target rows and vector indexes, not a DAG definition
• Lineage: per-row provenance lets every target row trace back to its exact source, and stale rows are retired automatically
• Source connectors: local files, S3, Google Drive, git repositories, Hacker News via the Algolia API, and CSV folders among the documented examples
• Target stores: Postgres with pgvector, LanceDB, Neo4j, Kuzu, SurrealDB, Kafka, and other database, warehouse, and queue targets
• Built-in operations: recursive text splitting, AST-aware code chunking, embedding, LLM extraction with typed schemas (via BAML or DSPy), transcription with speaker diarization, and entity deduplication
• Rust engine: parallel by default, zero-copy transforms where possible, and failure isolation per record
• CocoIndex-code MCP server: semantic code search, call graphs, and blast-radius analysis for Claude Code and other MCP-aware agents
• Control plane subsystems: live caching, pipeline catalog, version tracking, lineage, task scheduling, metrics collection, and failure management
• Example pipelines: code embedding, PDF RAG indexing, HN trending topics, conversation and podcast knowledge graphs, multi-repo summarisation, structured form extraction, and CSV to Kafka