iFixAi v4.2.0: The Open-Source Agent Auditor That Grades Your AI

iFixAi is an open-source auditor for deployed AI agents. It does not benchmark models. It grades whether your agent is actually doing the job it was given, within the permissions and process your organisation expects.

The argument is simple. Existing eval, red-teaming and observability tools measure technical capability. Token efficiency. Latency. Prompt-injection resistance. But they cannot answer the question that matters most. Is the agent doing what it is supposed to do?

iFixAi answers that with a letter grade.

I tested v4.2.0 for a week against Claude Code, Codex, and a custom agent we run internally. Here is my honest review.

iFixAi ships with several features that make it stand out:

The five pillars are the right five. Fabrication, manipulation, deception, unpredictability, opacity. These are the ways agents actually fail in production. Not benchmark drift. Not hallucination rates. Real failure modes that cost money and trust.

The cross-vendor judging is the real star. Your Claude agent gets judged by a non-Anthropic model. Your Codex gets judged by a non-OpenAI model. The subject never grades itself. This is the design decision that makes the grade trustworthy.

I tested this by running the same agent twice with different judge providers. The grades were close but not identical. The reason for each difference was inspectable. The key is that you have the choice, not iFixAi.

The offline commands are impressive. ifixai trajectory checks recorded agent actions against a business policy. ifixai policy-lab finds harmful combinations of tools that are each allowed on their own. Both run offline. Both do not change inspection scores. They are pure assurance.

iFixAi is not perfect. The telemetry is on by default. Pseudonymous telemetry, they call it. But it is still telemetry. You can opt out, and the opt-out works. But the default should be off.

The plugin support is impressive but not perfect. I had a few cases where the plugin conflicted with the agent's own hooks. The plugin works well for simple agents but complex agents with custom hooks sometimes fail to connect. Manual CLI audit is more reliable but defeats the purpose of the plugin.

The grade is useful but not perfect. I found that iFixAi sometimes penalises agents for failing inspections that are not relevant to the task. An agent that is great at coding but bad at long-run planning will get a C even if you only use it for coding. Manual inspection selection is more reliable but defeats the purpose of the grade.

A: The open-source engine is free under Apache 2.0. You run it on your own infrastructure with your own model keys. A commercial cloud offering with higher inspection counts and multi-agent support exists for enterprise teams.

Q: How does iFixAi compare to METR or SWE-bench?

A: SWE-bench measures whether a model can solve isolated tasks. METR measures task completion time. iFixAi measures whether your deployed agent behaves correctly within your organisational policy. Different tools for different questions.

A: No. Cross-vendor judging is automatic. The subject is excluded from the judge pool. This is enforced, not suggested.

iFixAi is a genuine solution to a real problem. 60 inspections, cross-vendor judging, offline policy checks - these are not marketing tricks.

If you are deploying AI agents and cannot answer "is it doing the job", try iFixAi. The open-source engine is free and the essential suite takes about 120 seconds.

The telemetry default and plugin conflicts need work. But for teams that need to verify their agents, iFixAi is one of the best options out there.

Auditing iFixAi taught me that I trust my agents too much. I have been running Claude Code and Codex for months without ever grading them. I just assumed they were behaving. Running iFixAi showed me a few behaviours I did not expect. Not catastrophic. But not nothing.

I also learned that I want a letter grade more than I want a report. A 60-inspection report is thorough but nobody reads it. An A-F grade is something I can put in a dashboard and actually look at. iFixAi gives me both. That is the right call.

The biggest mistake I see with agent auditing is auditing the model instead of the agent. People run benchmarks on the base model and call it an audit. That tells you nothing about the agent you actually deployed. The guardrails, the prompts, the tools, the permissions. That is where the risk lives.

Another mistake is auditing once and never again. Agents change. Models update. Prompts drift. An audit is a snapshot, not a guarantee. The teams that use iFixAi in CI, running it on every deploy, are the ones that catch problems early.

Do not run all 60 inspections. Start with the essential suite. Ten inspections, 120 seconds, one grade. Add more only when the essential suite flags something you care about.

Do not trust the grade alone. Read the inspection details. A C grade with one critical failure is different from a C grade with many minor failures. The grade is a summary, not the story.

Do not forget to audit the judge. Cross-vendor judging is the right design but the judge is still a model. If the judge provider has an outage or a bad update, your grade changes for reasons that have nothing to do with your agent.

The best assurance is the one you actually run. Run it before every deploy.