Wigolo gives an AI agent one interface for web work: search, fetch, crawl, extract, cache, find-similar, research, and autonomous gather loops. It runs as an MCP server next to a coding agent, as a REST and MCP endpoint on the machine where self-hosted agents run, or embedded in an application through TypeScript and Python SDKs. Search, fetch, crawl, extract, cache and find-similar need no API key, and the cache, embeddings, models and configuration stay on the local machine.
Search fans out to 18 engines through direct adapters, fuses the rankings, and reranks the results with an on-device model. Each result carries a verbatim excerpt pinned to a byte-offset span in the source, a citation ID, and a score decomposed into semantic, lexical and engine-consensus parts. Weak results are flagged as junk, failed engines are reported, and stale cache entries are labeled. Fetching starts with plain HTTP and escalates to a headless browser when it detects anti-bot challenges, single-page-app shells or thin content, and it learns per domain which sites need the browser.
The research, agent and answer-format search tools call an LLM to write a cited answer. Without one configured they return a raw brief and evidence for the calling agent to assemble. Synthesis works with Gemini, Anthropic, OpenAI or Groq models, or stays local through Ollama or any OpenAI-compatible endpoint. Deterministic steps such as canonicalization, rank fusion, deduplication and schema matching stay off the LLM, and LLM-filled fields are checked against the source and nulled when absent.
| Wigolo | Firecrawl | Exa | Tavily | |
|---|---|---|---|---|
| Multi-engine web search | yes | yes | yes | yes |
| Fetch and structured extraction | yes | yes | yes | yes |
| Whole-site crawl and map | yes | yes | no | yes |
| Excerpts pinned to byte-offset source spans | yes | no | no | no |
| Per-result score decomposition | yes | no | no | no |
| Persistent local cache, queryable offline | yes | no | no | no |
| API key or account | none | required | required | required |
| Cost per query | none | metered | metered | metered |
Features
- Search: 18 engine adapters with rank fusion, ML reranking and per-result scores, query arrays for parallel breadth, and domain, time-range, exact-phrase and image filters
- Fetch: a tiered router from plain HTTP to a headless browser, returning clean markdown, metadata and links, with PDFs, single sections, authenticated sessions and page actions
- Crawl: breadth-first, depth-first, sitemap or map-only crawls with per-domain rate limits, robots.txt respect and boilerplate deduplication
- Extract: tables, metadata, JSON-LD, brand identity, named schemas such as Article, Recipe and Product, or any custom JSON Schema
- Cache and find-similar: keyword or hybrid semantic queries over everything already fetched, change detection, and similar-page lookup that fuses keyword, semantic and live web results
- Research and agent: question decomposition, parallel sub-queries and a cited report, or an autonomous plan, search, fetch, extract and synthesize loop with a step log and time budget
- Diff and watch: shows what changed on a page since the last visit and delivers changes to a webhook
- Surfaces: MCP over stdio, a REST API with an OpenAPI 3.1 contract and remote MCP endpoints, a CLI and interactive shell, and TypeScript and Python SDKs with an embedded local mode
- Framework packages: LangChain tools and retriever, CrewAI tool set, LlamaIndex reader, and AI SDK tool factories
- Agent setup: writes MCP configuration and instructions for Claude Code, Cursor, Codex, Gemini CLI, OpenCode, VS Code, Windsurf, Zed and Antigravity, plus an 11-skill pack for coding agents
- Self-hosting: Docker images, a bearer token required when the server binds beyond loopback, and an opt-in proxy for challenge-protected sites
