IntellAgent

IntellAgent is a multi-agent framework for testing conversational agents by simulation. It generates realistic, challenging user interactions tailored to a specific chatbot, runs them against the agent, and critiques the resulting conversations to find where the agent breaks its policies.

The pipeline has three steps. First, it takes the chatbot's system prompt (and optionally its tools and database schema), decomposes the prompt into a policy graph, samples subsets of policies according to how often they co-occur in real conversations, and generates a scenario for each subset, including the system database state. Second, a user agent plays out the conversation with the chatbot. Third, a critique step evaluates the conversation and gives feedback on each tested policy.

Simulations are configured per environment, from a simple one without a database to a more complex one with a database, and the number of samples and a cost limit are adjustable. Both the simulator and the tested chatbot can use OpenAI, Azure OpenAI, Vertex, or Anthropic models. Results are shown in a Streamlit dashboard with analytics for comparing experiments. The method is described in a 2025 arXiv paper.

LangGraph agents are supported as a target platform, with CrewAI and AutoGen on the roadmap alongside API integration for external agents and generation of events from existing databases. The project collects anonymous usage metrics by default, which can be switched off.