Crawl4AI is a Python web crawler and scraper that turns web pages into Markdown and structured data for LLM use: retrieval-augmented generation, AI agents, and data pipelines. It drives real browsers (Chromium, Firefox, or WebKit) through an async browser pool, so it handles JavaScript-rendered pages, infinite scroll, and lazy-loaded media, and it outputs Markdown that keeps headings, tables, and code blocks, with page links turned into a numbered citation list.
Extraction runs either without a model, through CSS, XPath, or regex schemas, or with an LLM that fills a typed JSON schema. Any provider LiteLLM supports works for LLM extraction, including hosted models and local ones through Ollama. A schema generator writes a reusable schema from a one-time description, and content filters (pruning, BM25 for a query, or an LLM filter) strip menus, footers, and boilerplate to produce a shorter fit Markdown.
It runs three ways: as a library inside a Python process, as a self-hosted Docker server with a REST API, an MCP endpoint, a monitoring dashboard, and a playground, or as the hosted Crawl4AI Cloud API, which runs the browsers and proxies and adds web search, direct answers with sources, and extraction without a user-supplied LLM key.
| Library | Self-hosted server | Crawl4AI Cloud | |
|---|---|---|---|
| Runs the browsers | Your Python process | Docker on your machine | The service |
| JS-heavy pages and bot walls | Your settings and proxies | Your settings and proxies | Handled automatically |
| Web search | No | No | Search and answer endpoints |
| Price | Free | Free (your hosting) | Pay as you go |
Features
- Clean Markdown: headings, lists, tables, and code blocks, with links converted into a numbered reference list
- Content filters: pruning, query-driven BM25, and LLM filters remove boilerplate to produce fit Markdown
- Schema extraction: CSS, XPath, and regex strategies extract structured data without an LLM
- LLM extraction: fills a typed JSON schema with any LiteLLM-supported provider, hosted or local
- Deep crawl: BFS, DFS, and best-first strategies with crash recovery for long crawls
- Adaptive crawling: stops once it has gathered enough to answer a given query
- URL discovery: seeds URLs from sitemaps and Common Crawl and maps domains
- Browser control: persistent profiles, CDP remote browsers, sessions, proxies with rotation, stealth mode, and hooks at every crawl step
- Dynamic pages: runs JavaScript, waits for elements, scrolls full pages, and captures screenshots and PDFs
- Batch crawling: many URLs at once through a memory-adaptive dispatcher
- Self-hosted server: REST API, MCP endpoint for agents, monitoring dashboard, and playground in AMD64 and ARM64 images
- Hosted API: scrape, extract, search, and answer endpoints with an MCP server for Claude Code, Codex, Cursor, and OpenCode
