Crawl4AI is a Python web crawler and scraper that converts pages into Markdown and structured data for retrieval-augmented generation (RAG), agents, and data pipelines. It drives Playwright browsers asynchronously through an AsyncWebCrawler class, and it also exposes a crwl command-line interface for single crawls, deep crawls, and question-driven extraction.
Markdown generation produces headings, tables, code, and citation-style reference lists, with a heuristic "fit Markdown" filter and BM25-based filtering that strip boilerplate before content reaches a model. Structured extraction works two ways: schema-based CSS or XPath selectors that return JSON without a model, and LLM-driven extraction that accepts any open or proprietary model, with topic, regex, and sentence-level chunking and cosine-similarity selection of relevant chunks.
Browser handling covers Chromium, Firefox, and WebKit; persistent profiles with saved authentication state; session reuse across multi-step crawls; proxies with authentication; custom headers, cookies, and user agents; connection to remote browsers over the Chrome DevTools Protocol; and a stealth mode. Deep crawling supports breadth-first strategies with page limits, a prefetch mode for URL discovery, and crash recovery through resumestate and onstate_change callbacks for long runs.
A Docker image runs a FastAPI server with JWT authentication, a monitoring dashboard, a request playground, browser pooling with page pre-warming, endpoints for HTML extraction, screenshots, PDF generation, and JavaScript execution, and an MCP integration for connecting AI tools such as Claude Code. Since v0.9.0 the server enables authentication by default and binds to loopback unless given a token. A hosted cloud API is in closed beta.
Features
- Markdown output: structured Markdown with headings, tables, code blocks, and numbered citations, plus fit-Markdown and BM25 noise filtering
- Schema extraction: CSS and XPath selector schemas that emit JSON from repetitive page patterns
- LLM extraction: model-driven structured extraction with chunking strategies and cosine-similarity chunk selection
- Browser control: Chromium, Firefox, and WebKit; managed user-owned browsers; persistent profiles; sessions; proxies; custom headers and cookies
- Dynamic pages: JavaScript execution, async waits, lazy-load handling, full-page scroll simulation, and dynamic viewport sizing
- Media and links: extraction of images, audio, video, srcset/picture variants, iframe content, and internal and external links
- Deep crawling: BFS strategies with --max-pages, prefetch URL discovery, and resumable crash recovery
- Hooks and caching: hooks at each crawl step (string or function based) and cached fetches
- Inputs: live URLs, raw HTML via raw:, and local files via file://
- Docker API server: FastAPI service with JWT auth, dashboard, playground, browser pool, screenshot and PDF endpoints, and MCP integration
- CLI: crwl for Markdown output, deep crawls, and natural-language extraction queries
