Garak (Generative AI Red-teaming and Assessment Kit) is a command-line scanner that checks whether a language model or dialog system can be made to fail in unwanted ways. It probes for hallucination, data leakage, prompt injection, misinformation, toxicity, jailbreaks, and other weaknesses, playing a role for LLMs similar to what nmap or the Metasploit Framework play for networks.
A scan pairs a generator (the model under test) with probes (the attack prompts) and detectors (checks that decide whether an output shows the failure). Probes can be static, dynamic, or adaptive, and by default a run tries every known probe with the detectors each probe recommends. For every probe, the output reports a pass or fail per detector with the failure rate, and each run writes a JSONL report plus a hit log of the attempts that produced a vulnerability.
Generators cover Hugging Face models (local pipelines, the Inference API, and private endpoints), OpenAI chat and completion models, Replicate, Cohere, Groq, AWS Bedrock (Claude, Llama, Titan, AI21, Cohere, and Mistral models), NVIDIA NIM endpoints, LiteLLM, GGUF and ggml models through llama.cpp, and any REST endpoint returning text or JSON. Probes, detectors, evaluators, generators, and harnesses are all plugin classes, so new attacks or targets are added by subclassing a base class.
Features
- Probe library: DAN-style jailbreaks, encoding-based prompt injection, PromptInject, adversarial suffixes (GCG), glitch tokens, and Unicode perturbations
- Content-risk probes: toxicity (RealToxicityPrompts), Language Model Risk Cards, malware generation, misleading claims, and questions a model should refuse
- Leakage and hallucination checks: training-data replay, snowballed hallucination, package hallucination, and cross-site exfiltration attempts
- Automated attack generation: a red-teaming model probes the target and adapts to its responses
- Detectors: each probe names primary and extended detectors that judge whether an output shows the failure mode
- Model targets: Hugging Face, OpenAI, Replicate, Cohere, Groq, AWS Bedrock, NVIDIA NIM, LiteLLM, GGUF models through llama.cpp, and generic REST endpoints
- Reporting: per-probe failure rates, a JSONL report of every attempt, a hit log of successful attacks, and an analysis script for the prompts with the most hits
- Plugin architecture: probes, detectors, evaluators, generators, and harnesses inherit from base classes and can be extended with custom modules
