DataChain is a Python library that indexes files in S3, GCS, Azure, or a local filesystem into versioned, typed datasets and queries them without copying the bytes. Indexing stores typed metadata and file pointers, and a re-run processes only new or changed files. Every .save() registers a dataset such as pets_embeddings@1.0.0 in a Dataset DB kept locally in SQLite, recording schema, version, lineage, and processing state, so pipelines reference datasets by name rather than by path.
Column shapes are Pydantic models: the return type of a user-defined function becomes the dataset schema, nested models stay queryable as dotted fields, and the schema is enforced when the dataset is created. Filters, aggregations, and joins run as vectorized operations against the Dataset DB without loading rows into memory or downloading files, and embeddings stored as list[float] columns support cosine-distance search in the same query as ordinary metadata filters. A filtered subset can be exported to pandas.
Long runs are checkpointed: when a pipeline fails partway, re-running it resumes from the start of the last uncommitted batch rather than recomputing. Setting delta=True on a source makes later runs process only newly arrived files while bumping the dataset version.
For agent workflows, an optional Knowledge Base derives structured markdown from the Dataset DB, one file per dataset and bucket, browsable as files, through wikilinks, or in Obsidian. An installable skill plugs the datasets, chain operations, and Knowledge Base into Claude Code, Cursor, Codex, GitHub Copilot, and Pi, so an agent reads existing schemas and lineage before generating a pipeline and saves each step as a named dataset. The hosted Studio adds a shared dataset registry, distributed compute across attached clusters, an MCP endpoint for agents, access control, viewers for video, DICOM, NIfTI, and point clouds, lineage graphs, and reproducible runs, with data and compute staying in your own cloud or on-premises Kubernetes.
Features
- Storage indexing: S3, GCS, Azure, and local files become typed datasets of metadata and file pointers, with no bytes copied out of storage
- Pydantic schemas: a function's return type defines the dataset schema, nested models included, enforced at dataset creation
- Versioned datasets: every .save() records schema, version, inputs, author, time, and lineage, and a code or data change bumps the version
- Fast queries: filters, joins, and group-by run vectorized against the Dataset DB, over millions of records locally and hundreds of millions on Studio
- Vector search: embeddings live as columns beside the metadata, so similarity search combines with ordinary filters without a separate store
- Checkpoint recovery: a failed run resumes from the last uncommitted batch instead of recomputing finished work
- Incremental updates: delta=True processes only new or changed files on a re-run
- Knowledge Base: markdown derived from the Dataset DB, one file per dataset and bucket, navigable with wikilinks or in Obsidian
- Agent skill: installs into Claude Code, Cursor, Codex, GitHub Copilot, and Pi so agents read schemas and lineage before writing code
- Studio: shared registry, distributed jobs across attached clusters, an MCP endpoint, access control, lineage graphs, and viewers for video, DICOM, NIfTI, and point clouds
