DataFlow is a Python system for preparing LLM training data. It generates, refines, evaluates, and filters data drawn from noisy sources such as PDFs, plain text, and low-quality question-answer sets, producing datasets for pre-training, supervised fine-tuning, and reinforcement learning in specific domains.
Work is organized in a Pipeline, Operator, Prompt hierarchy modeled on PyTorch. Each operator takes structured input (JSON, JSONL, or CSV), performs one processing task, and writes its result to a named output column. Operators fall into categories such as generation, evaluation, filtering, and refinement, with about 10 core operators and more than 100 pipeline-specific ones. You can write custom operators and distribute them through GitHub or PyPI. LLM steps call an API backend or a local GPU model through vLLM.
Ready-made pipelines cover text question-answer mining for SFT and RL, reasoning enrichment (chain-of-thought, category, and difficulty), Text2SQL, knowledge-base cleaning from tables, PDFs, and Word documents, and Agentic RAG data. A DataFlow Agent analyses data, writes operators, and assembles them into pipelines for a stated goal. The wider suite adds a web UI with drag-and-drop pipeline editing and MCP access, a skills collection for coding agents, an ecosystem layer for domain modules, and RayOrch for distributed scheduling on Ray.
Published results include fine-tuning with a 10K-sample DataFlow instruction set:
| Qwen2.5-7B | Math avg | Code avg | Knowledge avg |
|---|---|---|---|
| Base | 37.1 | 76.5 | 76.0 |
| + Infinity-Instruct-10K | 22.6 | 77.6 | 75.8 |
| + Infinity-Instruct-1M | 33.3 | 78.0 | 75.8 |
| + DataFlow-Instruct-10K | 46.7 | 78.6 | 76.2 |
Features
- Operator pipelines: reproducible, reusable pipelines built from operators that read and write named data columns
- Operator library: about 10 core operators and more than 100 pipeline-specific operators for generation, evaluation, filtering, and refinement
- Custom operators: write your own and distribute them through GitHub or PyPI
- Ready-made pipelines: text QA mining, reasoning enrichment, Text2SQL, knowledge-base cleaning, and Agentic RAG data
- Document extraction: large-scale PDF to QA and book PDF to visual-QA conversion
- DataFlow Agent: analyses data, writes operators, and orchestrates them into pipelines for a task
- Model backends: LLM APIs or local GPU inference through vLLM
- Web UI: drag-and-drop pipeline editing with natural-language assistance and MCP access
- Distributed runs: RayOrch schedules large data jobs on Ray
