Fenic

Fenic is a DataFrame query engine that puts language-model calls inside the query model. You write the PySpark and SQL-style operations you already know — select, filter, join, group_by, agg — alongside semantic operators such as extract, classify, predicate, map, reduce, summarize, and embed, configure the models once on a Session, and build the pipeline lazily. The engine compiles and runs it with automatic batching, rate limiting, retries, token and cost accounting, and a response cache.

Extraction is schema-bound: you define the shape you want as a Pydantic model, and semantic.extract returns typed columns you can query like any other. Because the work is expressed as typed operators, the pipeline itself is the output — explain() shows the logical and physical plan, lineage() traces individual rows forwards and backwards through every operation, per-query metrics report tokens and cost, and a table or view can be promoted into a named MCP (Model Context Protocol) tool with typed parameters and bounded result sizes for an agent to call.

Semantic work also covers whole DataFrames: semantic.join matches rows on a natural-language predicate rather than exact keys, simjoin does a top-k embedding-similarity join, and withcluster_labels runs k-means over an embedding column. Text-heavy data is typed rather than left as strings — Markdown, transcripts in SRT and WebVTT, JSON queried with jq expressions, HTML, PDF document paths, and fixed-dimension embeddings — with Rust-accelerated chunking, regex, and fuzzy matching. Sources include local files, S3, and Hugging Face datasets.

Two commands target coding agents writing Fenic code: fenic check statically resolves a script's fc.* symbols against the installed package and reports namespace and import mistakes without executing anything, and fenic skill install copies the fenic-mechanics skill into the skill directories of the agents it detects.

Provider	Type	Notes
OpenAI	LLM + embeddings	GPT, o-series, GPT-5 family; text-embedding-3-*
Anthropic	LLM	Claude Haiku, Sonnet, and Opus, with thinking budgets
Google	LLM + embeddings	Gemini via AI Studio and Vertex
Cohere	Embeddings	embed-v4.0
OpenRouter	LLM aggregator	provider routing, fallbacks, price and throughput controls

Features

• Semantic column operators: extract, classify, predicate, map, reduce, analyzesentiment, summarize, embed, and parsepdf
• Semantic DataFrame operators: natural-language join, embedding simjoin, and k-means withcluster_labels
• Typed extraction: a Pydantic schema defines the output columns, validated at plan time
• Lazy plans: pipelines compile and rerun, with .cache() to materialise expensive intermediates and an LLM response cache
• Inspection: explain() for the plan, row-level lineage() in both directions, and per-query token and cost metrics
• MCP tools: promote a table or view into system tools or a parameterised tool, served with fenic-serve
• Unstructured types: Markdown, transcripts, JSON with jq, HTML, PDF paths, and embeddings as first-class logical types
• Data sources: local files, S3, and Hugging Face datasets, read as CSV, Parquet, or documents
• Full SQL: session.sql("... {df} ...", df=df) runs SQL over the same DataFrames
• Agent authoring aids: fenic check lints a pipeline's symbols, and fenic skill install teaches coding agents the API
• Optional extras: pdf, cluster, and sim-join install the heavier operators only when needed