Giskard is a Python library for testing and evaluating agentic systems. Version 3 is a rewrite built for dynamic, multi-turn testing of AI agents, split into focused packages that each carry only their own dependencies. A test target can be an LLM, a black-box agent, or a multi-step pipeline, wrapped as any sync or async callable.
Two packages make up the main surface. Giskard Checks creates evals for non-deterministic outputs, where the same input can produce different valid responses: a scenario sends one or more interactions to the target and applies checks to the trace, and a suite runs many scenarios together. Checks range from string matching, comparisons, regex, and semantic similarity to LLM-as-judge assessments such as groundedness and conformity.
Giskard Scan handles red teaming. From a plain-language description of an agent it generates adversarial test suites covering prompt injection, jailbreaks, harmful content, stereotypes, and misinformation, mapped to the OWASP LLM Top 10 threat categories. Its quality scan evaluates a knowledge base for retrieval-augmented generation (RAG), replacing the RAGET test-set generator from version 2.
Version 2 remains available but is no longer maintained. Its automatic scan for tabular and classic ML models, which detects performance, bias, and robustness issues, has no version 3 equivalent. The same company also offers Giskard Hub, a hosted platform for team-based agent testing and continuous red teaming.
Features
- Scenario evals: interactions plus checks, grouped into suites, for single-turn and multi-turn agent conversations
- Built-in checks: string matching, comparisons, regex, semantic similarity, and LLM-as-judge checks for groundedness and conformity
- Vulnerability scan: adversarial test suites generated from a plain-language agent description, across OWASP LLM Top 10 categories
- Prompt-injection probes: a built-in dataset of injection payloads
- Custom generators: user-supplied scenario generators can be passed to suite generation or registered for reuse
- RAG quality scan: knowledge-base evaluation that generates questions, reference answers, and context for retrieval pipelines
- Provider-agnostic routing: LLM judges and generators run through a routing layer with optional provider extras such as OpenAI and Anthropic
- Modular packages: checks, scan, core utilities, LLM routing, and agent orchestration ship as separate packages
- Opt-out telemetry: optional aggregated analytics that exclude prompts and outputs
