Wigolo

wigolo is a local-first web layer for AI agents that gives them search, fetch, crawl, extract, cache, find-similar, research, and autonomous gather loops through one surface. It runs where your agent runs: as an MCP server beside a coding agent, as a REST and MCP endpoint on a box hosting self-hosted agents, or embedded in your own app through a TypeScript or Python SDK. Search, fetch, crawl, extract, cache, and find-similar need no API keys; the reranker and embeddings run on-device, and cache, models, and configuration live under ~/.wigolo/. The project is in public beta.

Search fans a query, or an array of queries in parallel, across 18 direct engine adapters, fuses the rankings, reranks with an on-device model, and returns per-result evidence: a verbatim excerpt pinned to a byte-offset span in the source, a citation ID, a score decomposed into semantic, lexical, and engine-consensus parts, and a freshness signal. Weak results are flagged as junk by wigolo's own scorer, failed engines are reported, and stale cache is labeled.

The research and agent tools, and search format=answer, use an LLM to write a synthesized, cited answer; without one they return a raw brief and evidence for the host agent to assemble. Any provider works (Gemini, Anthropic, OpenAI, Groq), or Ollama and any OpenAI-compatible URL for a fully local setup. npx wigolo init downloads the browser engine and on-device models and can write MCP configuration for Claude Code, Cursor, Codex, Gemini CLI, OpenCode, VS Code, Windsurf, Zed, and Antigravity in one run; Node 20 or later and about 1.5 GB of disk are required.

Features

• search: multi-engine metasearch with rank fusion, ML reranking, domain and time scoping, exact-phrase matching, and image results
• fetch: a tiered router that escalates from plain HTTP to a headless browser on anti-bot challenges or SPA shells, returning clean Markdown, metadata, and links, with PDF handling, single-section extraction, authenticated sessions, and page actions
• crawl: BFS, DFS, sitemap, or map-only crawls with per-domain rate limits, robots.txt compliance, and boilerplate deduplication
• extract: tables, metadata, JSON-LD, brand identity, named schemas (Article, Recipe, Product), or any custom JSON Schema
• cache: keyword or hybrid semantic queries over everything already seen, plus stats, clearing, and change detection
• find_similar: pages similar to a URL or concept via fusion of keyword, semantic, and live web signals
• research and agent: question decomposition, sub-query fan-out, source fetching, and cited synthesis, or a plan-search-fetch-extract-synthesize loop with a step log, time budget, and optional output schema
• diff and watch: page change detection since the last visit, with on-demand re-checks delivered to a webhook
• REST API: wigolo serve exposes POST /v1/{tool}, an OpenAPI 3.1 contract, and /mcp and /sse on one port, failing closed with a bearer token off loopback
• SDKs and framework wrappers: wigolo-sdk for TypeScript and wigolo for Python with an embedded local mode, plus LangChain, CrewAI, LlamaIndex, Vercel AI SDK, and n8n integrations
• CLI and shell: every tool runs from the terminal with --json output or from an interactive NDJSON-piping shell