Promptfoo is a command-line tool and library for testing LLM applications. You declare prompts, providers, and test cases in a config file, run promptfoo eval, and inspect results in a local web viewer or on the command line. The same tooling covers red teaming: it generates adversarial probes against an application and produces a vulnerability report.
Evaluations run locally, so prompts stay on your machine. Providers include OpenAI, Anthropic, Azure, Bedrock, and Ollama, among others, and the tool works with any LLM API or programming language. It runs under Node.js 22.22 or later and is also distributed through Homebrew and PyPI. Results can be shared with a team, and checks can run inside CI/CD pipelines.
Features
- Prompt and model evals: automated test cases with assertions, run across prompts and providers in a matrix
- Side-by-side comparison: the web viewer shows outputs from different models or prompt variants next to each other
- Red teaming: adversarial probing and vulnerability scanning for LLM apps, with generated security reports
- CI/CD integration: evals run as pipeline checks to catch regressions before deployment
- Code scanning: pull-request review for LLM-related security and compliance issues
- Local execution: evals run on your machine with live reload and caching
- Provider coverage: OpenAI, Anthropic, Azure, Bedrock, Ollama, and other LLM APIs
- Node package: the library can be embedded in your own scripts as well as driven from the CLI
