CascadeFlow

cascadeflow is a runtime layer that runs inside an agent's execution loop rather than at the HTTP boundary, so it can act on individual model calls, tool calls, and sub-agent handoffs. Every step is scored across cost, latency, quality, budget, compliance, and energy, then resolved into one of four runtime actions: allow, switchmodel, denytool, or stop. It ships as a Python package and a TypeScript package, and the project measures its in-process overhead at under 5ms, against 10-50ms of network round trip for an external proxy.

Routing works by speculative execution with quality validation. A small, fast model drafts the answer; the response is checked against configurable thresholds for length, confidence from logprobs, output format, and semantic alignment; and only a failed check escalates to a larger model. A domain pipeline classifies each query first — code, math, data, legal, and other domains, by rule or optional ML semantic classification — and routes it to the cascade configured for that domain.

Integration comes in tiers. cascadeflow.init(mode="observe") tracks existing OpenAI and Anthropic SDK calls with no code changes; a scoped cascadeflow.run(budget=..., maxtoolcalls=...) context adds per-run caps and prints a cost, latency, energy, and step summary plus a full decision trace; and enforce mode applies the policy actions. Adapters cover LangChain, the OpenAI Agents SDK, CrewAI, Pydantic AI, Google ADK, n8n, the Vercel AI SDK, OpenClaw, and Hermes Agent, over a provider layer spanning more than 17 providers.

Reported cost savings against flagship-only execution, retaining 96% of GPT-5 quality:

Benchmark	Cost saving
MT-Bench	69%
GSM8K	93%
MMLU	52%
TruthfulQA	80%

Features

• Speculative cascading: a cheap drafter runs first and escalates to a flagship model only when quality validation fails
• Quality validation: length checks, logprob confidence scoring, format validation, and semantic alignment against the request
• Domain routing: queries are classified into domains and routed to the model cascade configured for that kind of work
• Runtime enforcement: allow, switchmodel, denytool, and stop act on a step before it executes
• Harness modes: off, observe, and enforce, so a rollout can start as tracking only and switch over later
• Budget enforcement: per-run and per-user budget caps with automatic stop actions when a limit is passed
• Compliance gating: GDPR, HIPAA, and PCI rules plus strict model allowlists block non-compliant models before execution
• KPI-weighted routing: business weights for quality, cost, latency, and energy feed into each model decision
• Decision traces: a per-step audit trail records the action, reason, model, cost, budget state, and enforcement status
• Framework adapters: LangChain, OpenAI Agents SDK, CrewAI, Pydantic AI, Google ADK, n8n, Vercel AI SDK, OpenClaw, and Hermes Agent
• Provider coverage: a unified API over more than 17 providers, including OpenAI, Anthropic, Groq, Together, Ollama, and vLLM
• Cost tracking: built-in analytics with vendor-neutral OpenTelemetry export