DataChain is a Python library that indexes files in S3, Google Cloud Storage, Azure, or a local filesystem into versioned, typed datasets without copying the files. A dataset is a named, versioned result of a pipeline step, holding Pydantic-typed metadata and pointers to the original files, and every save records its schema, version, and lineage in a local Dataset DB. Pipelines then reference datasets by name instead of paths.
The compute engine runs Python functions over files in parallel with async I/O. Processing state lives in the Dataset DB, so a crashed run resumes from its last committed batch, and incremental runs process only new or changed files while bumping the dataset version. Filters, joins, aggregations, and vector similarity search run as vectorised operations against the Dataset DB without downloading files, and results can be exported to pandas for a filtered subset.
For agent workflows, an optional skill plugs DataChain into Claude Code, Cursor, Codex, GitHub Copilot, and Pi. The agent reads a Knowledge Base of Markdown summaries derived from the Dataset DB and enriched by an LLM, which describes each dataset, its schema, joinable fields, and column meaning, and the agent uses it to generate pipelines and save each step as a reusable dataset. The Knowledge Base can also be browsed as linked Markdown, for example in Obsidian.
DataChain Studio, the hosted team product, adds a shared dataset registry, distributed compute across attached clusters, an MCP endpoint for agents, access control, viewers for video, medical imaging, and point clouds, lineage graphs, and reproducible runs. It can run in your own cloud on AWS, GCP, Azure, or on-premises Kubernetes.
Features
- Storage indexing: turns objects in S3, GCS, Azure, or local storage into typed datasets without moving the data
- Typed schemas: Pydantic models define columns, including nested types, and are enforced at dataset creation
- Versioning and lineage: every save registers a named dataset version with recorded inputs and schema
- Parallel compute: Python functions run over files with async I/O and prefetching
- Checkpoint recovery: failed runs resume from the last committed batch
- Incremental updates: re-runs process only new or changed files
- Fast queries: filter, join, group, and count over millions of records locally without reading files
- Vector search: embeddings stored alongside metadata and queried with similarity functions
- Agent skill: gives Claude Code, Cursor, Codex, GitHub Copilot, and Pi awareness of your datasets
- Knowledge Base: LLM-enriched Markdown summaries of datasets and buckets for agents and people
- Studio: shared registry, distributed jobs, MCP endpoint, access control, and bring-your-own-cloud deployment
