GraphGen generates synthetic training data for large language models, guided by knowledge graphs. It builds a fine-grained knowledge graph from source text, measures where a trainee model lacks knowledge using the expected calibration error metric, and prioritizes question-answer pairs that target high-value, long-tail knowledge. Multi-hop neighborhood sampling captures relational information across the graph, and style-controlled generation diversifies the output.
The generated data feeds supervised fine-tuning, pretraining rephrase, and reinforcement learning with verifiable rewards (RLVR). After generation, you fine-tune with LlamaFactory or xtuner. Two models take part in each run: a synthesizer model that builds the graph and writes the data, and a trainee model whose knowledge gaps steer generation.
Reported SFT results, where over half the SFT data came from GraphGen, against a Qwen2.5-7B-Instruct baseline:
| Domain | Dataset | GraphGen | Baseline |
|---|---|---|---|
| Plant | SeedBench | 65.9 | 51.5 |
| Common | CMMLU | 73.6 | 75.8 |
| Knowledge | GPQA-Diamond | 40.0 | 33.3 |
| Math | AIME24 | 20.6 | 16.7 |
| Math | AIME25 | 22.7 | 7.2 |
Workflows are declared as a graph of nodes in a YAML file, and the generation pipeline runs on Ray for distributed execution. A Gradio demo is hosted on Hugging Face and ModelScope.
Features
- Knowledge-graph construction: extracts entities and relations from source text into a graph that drives generation
- Knowledge-gap targeting: uses expected calibration error on the trainee model to prioritize long-tail knowledge
- QA formats: atomic, aggregated, multi-hop, chain-of-thought, visual QA, multiple-choice, multiple-answer, fill-in-the-blank, and true-or-false
- Rephrase pipeline: LLM-driven reformulation produces diverse variants of a pretraining corpus instead of repeating it
- Data sources: CSV, JSON, PDF (through MinerU), and TXT files, Hugging Face Datasets, UniProt, NCBI, and RNAcentral, plus Bing, Google, and Wikipedia search
- Inference backends: Hugging Face Transformers, SGLang, and vLLM locally, or OpenAI, Azure, SiliconFlow, Ollama, and HTTP API servers
- Storage backends: Kuzu or NetworkX for the graph and RocksDB or JSON for key-value storage
- Graph evaluation: metrics for entity and relation accuracy, conflict detection, and structural properties such as connectivity and degree distribution
- Community detection: Leiden clustering over the knowledge graph to synthesize chain-of-thought data
