Metaflow is a Python framework for building and operating machine-learning, AI, and data-science systems, covering the path from a prototype in a notebook to a scheduled production deployment. Flows are written as plain Python classes whose steps pass data to one another; the framework versions code, data, and artifacts at every step automatically, so experiment tracking and debugging come from the run history rather than separate instrumentation.
The same flow runs on a laptop and on remote compute without code changes. Steps can be scaled vertically or horizontally in a cloud account, using CPUs or GPUs, for both embarrassingly parallel fan-out and gang-scheduled distributed workloads, with checkpointing and retry behaviour for failures. Dependencies are declared per step and resolved for local and remote execution alike.
For production, flows deploy to highly available orchestrators with one command and can be triggered reactively by events from other systems. The infrastructure runs in the user's own account: Amazon EKS with S3 or AWS Batch and Step Functions, Azure AKS with Blob Storage, Google GKE with Cloud Storage, or a custom Kubernetes cluster on-premises. Metaflow was developed at Netflix and open-sourced in 2019, and is now supported by Outerbounds.
Features
- Pythonic flow API: workflows are plain Python steps, developed and debugged locally before deployment
- Automatic versioning: code, data, and artifacts are tracked per run for experiment comparison and debugging
- Notebook support: flows can be run and inspected from notebooks, with built-in result visualization
- Scaling: vertical and horizontal scaling to cloud compute with CPUs or GPUs, fast data access, and distributed training
- Reliability: step-level retries, failure handling, and checkpointing for long-running compute
- Dependency management: per-step library dependencies resolved consistently for local and remote runs
- One-command deployment: production deployment to highly available orchestrators without changing flow code
- Event triggering: flows react to updated data and events from other systems
- Multi-cloud infrastructure: deploys on AWS, Azure, Google Cloud, or a custom Kubernetes cluster in your own account
