Giskard is an open-source Python library for testing and evaluating agentic systems. A target is anything you can call — an LLM, a black-box agent, or a multi-step pipeline — wrapped as a sync or async callable from inputs to outputs, and the library builds evaluations and adversarial test suites around it. Version 3 is a rewrite organised as separate packages that each carry only their own dependencies, and it requires Python 3.12 or later.
Giskard Checks is the evaluation half. It expresses one eval as a scenario: a message or short conversation sent to the target, plus checks the reply has to satisfy. Because model outputs are non-deterministic, checks range from string matching, comparisons, regex, and semantic similarity to LLM-as-judge assessments such as Groundedness, Conformity, and LLMJudge. Scenarios group into suites, and multi-turn conversations are tested as a whole rather than exchange by exchange.
Giskard Scan is the red-teaming half. From a plain-language description of an agent it generates adversarial test suites automatically, covering prompt injection, jailbreaks, harmful content, stereotypes, and misinformation across the OWASP LLM Top 10 threat categories, with a built-in injection payload dataset. A companion quality scan evaluates a knowledge base for retrieval-augmented generation. Both scan modes and the judges need a model provider extra and an API key; the default judge model is openai/gpt-4o-mini.
Three foundational packages sit underneath and are usually pulled in rather than used directly: giskard-core for shared utilities and telemetry, giskard-llm for provider-agnostic model routing, and giskard-agents for agent and workflow orchestration. Optional aggregated telemetry sends no prompts or outputs and is disabled with the DONOTTRACK or GISKARDTELEMETRYDISABLED environment variables. Version 2 remains available for the tabular and classical machine-learning scan, which is not carried into version 3.
Features
- Scenario API: define an eval as interactions with a target plus the checks its output must satisfy
- Built-in checks: string matching, comparisons, regex, semantic similarity, and LLM-as-judge assertions
- Multi-turn testing: evaluate whole conversations rather than single request and response pairs
- Suites: run many scenarios together and print a consolidated report
- Vulnerability scan: generate adversarial suites from a description of the agent, across OWASP LLM Top 10 categories
- Prompt-injection probes: a bundled dataset of injection payloads ready to run
- Custom generators: pass your own ScenarioGenerator instances or register them on the suite generator registry
- RAG quality scan: evaluate a knowledge base for grounding and answer quality
- Modular packages: checks, scan, core, LLM routing, and agent orchestration installed independently
- Provider-agnostic routing: provider SDKs installed as extras, with an LLM judge model you choose
- Opt-out telemetry: aggregated analytics only, disabled by an environment variable
