How I audit an AI assistant flow: disclosure, pace, handoff

Posted on Oct 19 • Originally published at perex.design

A conversation is not a screen. The failure modes are different, the evidence is different, and a heuristic checklist alone will pass a product that is quietly losing its users' trust. This is what I do differently when the flow under audit is a chat, a voice agent or an autonomous assistant.

A screen behaves the same for everyone who reaches it. An assistant does not. Its behaviour depends on what the user says, how they say it, and what they expect. So a single happy-path walkthrough tells you almost nothing.

In the sample critical-flow report, the assistant flow from signup to first prompt to first escalation is walked in three separate sessions as three user postures: curious, task-driven, and hostile. Every path is walked, including the ones the product team did not draw. The curious user asks "what can you do?" The task-driven user arrives with a job and no patience for a tour. The hostile user is testing whether the thing will lie, stall, or dump them. Each posture surfaces a different class of leak, and a finding that appears in all three is a different severity from one that appears in one.

Standard usability heuristics still apply to an assistant: visibility of status, error prevention, recognition over recall. But they were written for deterministic interfaces. An assistant that is right most of the time needs a second set of principles for the times it is not. In the audit I score against four of them alongside the heuristics:

Every finding then carries the same five fields as any other audit: severity, evidence, trust cost, fix, and effort. Findings are ordered by impact against effort, not by where they sit in the flow.

The reason a copy-level review is not enough is that the three decisions that decide whether an assistant is trustworthy are made before the first message is sent. I wrote them up as three reusable AI-trust patterns after building them into NuvAI, an assistant that helps human real-estate agents qualify and follow up with leads.

The user should not have to work out that they are talking to a machine, or what it is authorised to do. Regulation is moving the same way: the EU AI Act treats "you are interacting with a machine" as a baseline. In an audit, I look for where disclosure happens, whether it happens early enough to matter, and whether the product's tone later contradicts it.

Every assistant has a boundary where it stops being useful. The audit question is not whether the boundary exists but what happens at it. Does the conversation escalate with its context intact, or does the user start again with a human who knows nothing? The dead-end apology, where the assistant says sorry and offers no fork, is one of the most common findings in this kind of flow.

One more thing I check that a screen audit never has to: what the assistant is allowed to do on the user's behalf, and whether the interface makes that boundary honest. The permission board project exists to make that argument concrete. An agent can have access to a capability and still not have permission to use it, and a capability that was never built cannot be permitted at all. In an audit this becomes a set of questions: what actions can this assistant take alone, does the user know, and does the product ever imply an ability it does not have?

The same deliverable as any critical-flow audit: a one-page executive summary, the full findings ordered by impact and effort, an annotated walkthrough with timestamps, a "stop doing" list of features actively costing trust, a 45-minute recorded walkthrough call, and two rounds of async questions over the following two weeks. The sample report shows the format on "Aria", a composite assistant assembled for demonstration. No client is implied; the format, depth and honesty are the real thing.

The status of the work behind this: the AI-trust patterns are defined on NuvAI's real stack, with validation as the next step. The Voice Agent Lab study has not been run. I say this on the source pages and I say it here, because an audit that scores your product on disclosure had better disclose its own limits.

One critical flow, five business days, written report plus walkthrough call. Or start smaller with the \$350 teardown of one flow in 48 hours.

See the critical-flow audit — \$550 → See the teardown — \$350 →