ScrapeGraphAI

ScrapeGraphAI is a Python web-scraping library that combines a large language model (LLM) with direct graph logic to build extraction pipelines for websites and local documents such as XML, HTML, JSON, and Markdown. You describe the information you want in a prompt and supply a source URL or file; the pipeline fetches the page (with Playwright for live websites), runs the LLM over it, and returns structured output, typically a JSON-like dictionary.

Pipelines are exposed as graph classes. SmartScraperGraph handles a single page from a prompt and a source; the others extend that to search results, multiple pages, audio output, or generated Python scripts. Each graph also has a multi variant that runs LLM calls in parallel.

Pipeline	What it does
SmartScraperGraph	Single-page scraper that needs only a user prompt and an input source
SearchGraph	Multi-page scraper over the top n results of a search engine
SpeechGraph	Single-page scraper that also generates an audio file
ScriptCreatorGraph	Single-page scraper that generates a Python script for the extraction
SmartScraperMultiGraph	Multi-page scraper over a list of sources with a single prompt
ScriptCreatorMultiGraph	Generates a Python script for extracting from multiple pages and sources

The LLM is configured per graph. Hosted providers such as OpenAI, Groq, Azure, Gemini, and MiniMax are selected by changing the model string and API key in the config, and local models run through Ollama. The library integrates with LangChain, LlamaIndex, CrewAI, Agno, and CAMEL, with low-code platforms including Pipedream, Bubble, Zapier, n8n, Dify, and Toolhouse, and it is also available as a Model Context Protocol (MCP) server.

The open-source library is self-hosted: you bring your own LLM keys, configure the browser, and handle proxies, anti-bot measures, and scaling. A separate managed cloud API, used through Python and JavaScript SDKs, handles the LLM, JavaScript rendering, proxies, and scaling for you and adds Scrape, Extract, Search, Crawl, Monitor, and History capabilities on a pay-as-you-go credit model. The library collects anonymous usage metrics by default; set SCRAPEGRAPHAITELEMETRYENABLED=false to opt out.

Features

• Prompt-driven extraction: state which fields you want and the pipeline returns them as structured data
• Website and file sources: scrapes live pages via Playwright as well as local XML, HTML, JSON, and Markdown documents
• Pipeline graphs: SmartScraper, Search, Speech, ScriptCreator, and their multi-page variants
• Parallel multi graphs: the multi version of each graph runs LLM calls in parallel
• Provider choice: OpenAI, Groq, Azure, Gemini, MiniMax, and others through configuration, or local models through Ollama
• Framework integrations: LangChain, LlamaIndex, CrewAI, Agno, and CAMEL
• Low-code and MCP access: Pipedream, Bubble, Zapier, n8n, Dify, Toolhouse, and an MCP server
• Managed API option: a hosted service with SDKs that adds crawl, scheduled monitoring, and managed rendering and proxies
• Telemetry opt-out: anonymous metrics are collected unless SCRAPEGRAPHAITELEMETRYENABLED=false is set