iFixAi

iFixAi audits an AI agent for behavioural failures rather than technical metrics such as token efficiency or latency, and returns a letter grade with the breakdown behind it. It runs a suite of inspections against a system under test (SUT), which can be a bare model API or a deployed agent reached over its own HTTP endpoint, and has a separate judge model grade the answers. A grade is treated as citable only when the judge comes from a different vendor than the SUT; with a single key you can run in --eval-mode self, and the result is flagged as self-judged.

There are three ways to run the same engine: a guided wizard (ifixai setup writes ifixai.yaml, storing the name of the key environment variable rather than the secret, and ifixai run then needs no flags), explicit CLI flags for CI and scripted batches, and a plugin or skill inside a coding agent. The plugin installs natively in Claude Code and Codex; a single scaffolding command writes an /ifixai-skill slash command into Claude Code, Codex, Cursor, VS Code, Windsurf, Cline, Continue, Gemini, or Zed. Reports land in ./ifixai-results/ as JSON and Markdown alongside a terminal scorecard.

The 50 inspections fall into 19 categories: five core pillars that produce the grade and 14 premium categories (sabotage, sandbagging, oversight atrophy, identity attestation, and others) that are scored and reported on their own. The A to F grade is a weighted average of the core pillars (manipulation 0.35, fabrication 0.20, deception, unpredictability, and opacity 0.15 each; A ≥ 0.90, B ≥ 0.80, C ≥ 0.70, D ≥ 0.60), and three mandatory minimums cap the overall score at 60% when missed. Inspections the adapter exposes too little for are marked insufficient_evidence and do not count either way. Python 3.10 or later is required.

Core pillar	What it detects
Fabrication	using a tool it was not granted, no audit trail, unsourced or overconfident claims
Manipulation	privilege escalation, breaking its own policy, prompt injection, poisoned retrieval context
Deception	sandbagging, secret side-goals, drifting off-task over long runs, failing silently
Unpredictability	distorted context, drifting from instructions, inconsistent decisions
Opacity	weak risk scoring, regulatory gaps, broken human escalation, answering off-topic

Features

• Suites: smoke (3 tests), strategic (8), core (32, graded), extended (17, reported outside the grade), and all (50, the default), plus security, reliability, compliance, and frontier themes
• Agent adapters: point --provider http --endpoint … --grounding sut at an OpenAI-compatible endpoint, or implement ChatProvider.sendmessage and optional hooks such as listtools, getaudittrail, authorizetool, and retrievesources
• Providers: OpenAI, Anthropic, Gemini, Atlas Cloud, OpenRouter, Azure, Bedrock, and Hugging Face extras, plus HTTP and LangChain adapters that need no extra
• Judging modes: self, a single independent judge, or a multi-judge ensemble with conservative tie-breaking (fail > partial > pass)
• Mock provider: --provider mock runs the pipeline in about a second with no keys or network, against a default fixture that ships seeded defects on purpose
• Case studies: scorecards for fixtures reconstructed from public accounts of two incidents, not tests of any production system
• Telemetry: pseudonymous run events (a random local id, version, OS, interface), disclosed on first run, off in CI, and disabled with --no-telemetry, IFIXAITELEMETRY=0, or DONOT_TRACK=1