Measuring AI Reliability Through Robustness, Safety, and Governance

Signup Write New Story Measuring AI Reliability Through Robustness, Safety, and Governance by Sunilkumar Reddy Eraganeni by Sunilkumar Reddy Eraganeni | @ SunilKumar_n1qkl9r8 Data Engineer at Barclays Data Engineer specialising in Azure, AWS, Databricks, dbt, and scalable data platforms. Writing about Data Engineering, AI, and cloud architecture.

Data Engineer specialising in Azure, AWS, Databricks, dbt, and scalable data platforms. Writing about Data Engineering, AI, and cloud architecture.

Ask most engineers whether their model is good and you will get a number back. Ninety-four percent accuracy. An F1 of 0.88. Latency under 200 milliseconds. I have given answers like that myself, and honestly, they feel great. A number is something you can put in a slide, track in a dashboard and argue about in a stand-up.

The trouble is that a number only answers the question it was built to answer. Accuracy tells you how often the model matches a label in a test set. It does not tell you what happens in the other six percent, who ends up in that six percent, or how badly it goes wrong when it does. Those are different questions, and I suspect we skip them partly because they are harder to put on a slide.

Consider a support chatbot that answers 97 percent of questions correctly. The remaining 3 percent could be harmless typos, or one confident wrong answer about a refund policy. In 2024, a Canadian tribunal held Air Canada responsible after its chatbot gave a customer incorrect information about bereavement fares. The bot was probably "accurate" by most internal measures. It was still the wrong system to leave unsupervised.

I do not want to bash performance metrics. They are necessary, and a model that fails at its basic task is not safe, just useless. My complaint is narrower than that. Performance metrics are averages, and harm rarely lives in the average.

Three gaps keep showing up, at least in the systems I have seen or read about.

First, averages hide tails. A resume screening tool can look strong overall while quietly scoring one group worse. Amazon reportedly scrapped an experimental recruiting model around 2018 after it learned to penalise resumes that signalled being a woman. The headline accuracy looked fine. The pattern underneath did not.

Second, test sets are polite. They are usually drawn from the same distribution as the training data, while real users are not polite at all. They paste in strange formats, make typos, try to trick the system on purpose, or just ask something nobody anticipated.

Third, cost is not symmetric. Missing a spam email and missing a tumour on a scan are both "errors", but treating them as equal in a single score is a modelling choice, and a questionable one.

Anecdotes only go so far, so I went looking for wider data. Stanford's 2026 AI Index reports that the AI Incident Database recorded 362 documented incidents in 2025, up from 233 in 2024, which is roughly a 55 percent jump. The database counts harms or near harms that happened in the real world through the deployment of AI systems. It is worth remembering that these are documented cases, not a complete count.

The same report points to a lopsided habit in how we measure things. Almost all leading developers publish results on capability benchmarks such as MMLU and SWE-bench, while reporting on responsible AI benchmarks stays sparse. Transparency slipped too. The average Foundation Model Transparency Index score climbed from 37 to 58 between 2023 and 2024, then dropped to 40 in 2025.

There is some good news. The share of businesses with no responsible AI policy fell from 24 percent to 11 percent. Yet in coverage of the same report, the share of organizations reporting three to five incidents rose from 30 percent to 50 percent. Both are self-reported survey figures, so I read them as direction rather than precision. Still, policies on paper and safety in practice do not seem to be the same thing, and that gap is exactly what better metrics should help close.

So where in the build process should a better scoreboard show up? Here is a simple picture of the shift, starting with the old pipeline versus a safety-aware one.

The top row is what many of us do by default. The bottom row adds two steps that sound obvious and are surprisingly easy to skip: someone actively tries to break the system, and someone has the authority to say "not yet."

Underneath those steps sits a bigger idea. Success is not one number but a stack of them, and each layer answers a different question. I find it easier to think of it as a building rather than a scoreboard.

Performance is still the foundation, and I would never argue otherwise. The layers above it are where most teams are thin.

Robustness asks how the system behaves when inputs stop looking like the training data. For a medical triage model, that might mean testing on notes written by a different hospital, in a different style, with different abbreviations. A model that drops ten points there has taught you something a clean test set never would.

Safety behaviour is the layer people find most slippery, because it is harder to define. A few measurements do seem workable, though. You can track the rate of harmful or policy-violating outputs under a fixed set of adversarial prompts. You can also track how often the system refuses things it should answer, because over-refusal is its own failure and a system that says no to everything is not safe, just irritating. The goal is calibration, not caution for its own sake.

The top layer is governance, which sounds dull until something goes wrong. Who gets paged? Who can pause the system? Is there a log someone can read three months later? I would argue these are metrics too, even if they look more like organisational facts than numbers. "Time to detect a bad behaviour" and "time to roll back" are measurable, and they probably predict real-world harm better than another half-point of accuracy.

Metrics only matter if something in the system can act on them. This is roughly where the checks sit around a deployed model.

Input checks catch obviously bad or malformed requests. Output checks look at what the model is about to say. Everything gets logged, a sample goes to humans, and the findings feed back into the tests you run before the next release. None of these boxes is exotic. The hard part is owning them.

One thing I only appreciated after watching a few deployments is that none of this is a one-time exercise. Models drift, users change, and attackers get creative. A system that passed every check in March can behave differently in September because the world it sees has moved.

That is why I think of safety metrics as a loop instead of a checklist.

The most underrated step in that loop is review. Teams tend to log failures but rarely read them with the same care they give to a launch. Near misses, where the system almost did something bad and a human caught it, are especially valuable, and they almost never make it into a metric. Counting them is a cheap way to start.

I should be honest about the limits here, because a tidy framework can hide as much as it shows.

Safety metrics are harder to define than accuracy, and some of them are contested. What counts as "harmful" depends on context, culture and the product. Two reasonable teams could draw that line in different places, and I do not think there is a clean universal answer.

There is also Goodhart's law. Once a measure becomes a target, people start optimising for the measure instead of the thing it stands for. A model trained hard to pass a fixed set of red-team prompts may simply learn those prompts. That is a real risk, and the only mitigation I know of is rotating the tests and keeping some of them private, which is imperfect.

And yes, this costs time. A release gate slows shipping, and a small team with a deadline will feel that. I would not pretend otherwise. My view, tentatively, is that the cost scales with the stakes. A recommendation widget for a hobby app does not need a governance layer. A system that touches money, health or hiring probably does.

The shift I am arguing for is smaller than it sounds. It does not mean throwing out accuracy or latency. It means treating them as the floor of a longer list, and asking a few uncomfortable questions before launch instead of after.

If you want a starting point that fits in an afternoon, try this. Pick the single worst thing your system could possibly do. Write one test that tries to make it happen. Decide who has the authority to stop the release if that test fails. That is already more than many shipped systems have, and it is a better first step than another decimal place on a benchmark.

I do not think we will ever get one clean number for "safe." We can at least stop pretending that "accurate" was ever the whole story.

When AI Makes Medical Decisions: Designing Human-in-the-Loop Clinical Systems