sift-kg is a command-line tool that turns a folder of documents into a knowledge graph. It extracts text locally from PDFs, DOCX, XLSX, PPTX, HTML, EPUB, images, and other formats (75+ via the Kreuzberg engine, with optional OCR), asks an LLM to extract entities and relationships, assembles them into a NetworkX graph, and opens an interactive viewer in your browser. Every entity and relation links back to the source document and passage.
Schema discovery runs by default: one LLM call samples the corpus and designs entity and relation types for it, saved as a YAML file you can edit and reuse. Bundled fixed-schema domains cover general document analysis, OSINT investigations (shell companies, financial accounts, beneficial ownership), and academic literature mapping (theories, methods, systems, findings), or you define your own domain in YAML with closed vocabularies and type constraints. Any LiteLLM-compatible provider works, including OpenAI, Anthropic, Mistral, and local Ollama models, and a --max-cost flag caps spending.
Entity resolution is human-in-the-loop: the LLM proposes merges and you approve or reject them in a terminal UI before they apply. The resulting JSON graph doubles as persistent memory for AI agents. Commands such as sift topology, sift query, and sift search --json return structural overviews, neighbourhood subgraphs, and entity lookups as JSON, and a bundled agent skill teaches an agent to use the graph across sessions. The graph grows incrementally as you extract new documents into the same output directory. A Python library API exposes the same pipeline.
Features
- Document ingestion: 75+ formats through Kreuzberg, with OCR via Tesseract, EasyOCR, PaddleOCR, or Google Cloud Vision and automatic detection of scanned pages
- Schema discovery: the LLM designs corpus-specific entity and relation types; fixed domains (general, osint, academic) and custom YAML schemas are alternatives
- Provider choice: OpenAI, Anthropic, Mistral, Ollama, or any LiteLLM-compatible provider, with a per-run cost cap
- Human-approved deduplication: proposed entity merges are reviewed interactively before they are applied
- Interactive viewer: force-directed graph with community regions, focus mode, keyboard navigation, a trail breadcrumb, search, and type, community, source, and confidence filters
- Exports: GraphML, GEXF, SQLite, CSV, and JSON for Gephi, yEd, Cytoscape, DuckDB, or Datasette
- Narrative generation: a Markdown report with relationship chains, a timeline, and entity profiles grouped by Louvain community
- Agent memory: JSON topology, query, search, and info commands and a bundled SKILL.md for agents that use the graph as persistent memory
- Multilingual input: extracts from documents in any language into a unified English graph with romanised non-Latin names
- Provenance: each extraction records the document and passage it came from
