Firecrawl

Firecrawl is a web data API that searches, scrapes, and crawls sites, returning clean Markdown, structured JSON, screenshots, or HTML that an agent can consume directly. It is open source and also runs as a hosted service, and it handles proxy rotation, orchestration, rate limits, and JavaScript-rendered pages behind the API.

Three core endpoints cover most use: search runs a web search and returns full page content for the results, scrape converts a URL into Markdown, HTML, a screenshot, or structured JSON, and interact acts on a scraped page — clicking, scrolling, typing, waiting, and pressing keys — from an AI prompt or code. Four more endpoints extend that: agent gathers data from a prompt without requiring URLs, crawl walks a whole site in one request, map lists a site's URLs, and batch scrape handles thousands of URLs asynchronously.

The agent endpoint accepts a prompt and an optional schema for structured output, and it can be pointed at specific URLs. An effort setting of low, medium, or high changes the reasoning budget while running the spark-2 model; a legacy model field still selects spark-1-mini, spark-1-pro, or spark-2, and sending both fields returns a 400 error. A request with neither runs spark-1-pro.

Client libraries exist for Python, Node.js, .NET, and PHP, and they poll asynchronous jobs for you. A CLI and an installable agent skill wire it into coding agents, and an MCP server connects any MCP-compatible client. Platform integrations include n8n, Zapier, and Lovable. Firecrawl respects robots.txt directives by default, and the project states that respecting a site's policies remains the user's responsibility.

Features

• Search: web search that returns full page content, not just links
• Scrape: any URL to Markdown, HTML, screenshots, or schema-shaped JSON
• Interact: click, scroll, write, wait, and press on a scraped page from a prompt or code, with a live view URL
• Agent endpoint: prompt-driven data gathering with optional URL focus, structured output schemas, and an effort setting
• Crawl and map: whole-site crawling with job status polling, and URL discovery with optional relevance search
• Batch scrape: thousands of URLs scraped asynchronously in one job
• Media parsing: content extracted from web-hosted PDFs, DOCX, and similar files
• SDKs: Python, Node.js, .NET, and PHP clients that handle polling for async operations
• Agent access: a CLI, an installable skill, and an MCP server for MCP-compatible clients
• Integrations: n8n, Zapier, and Lovable, alongside the hosted playground and API reference