Five regression tests for channel-aware AI reply drafts
A reply draft can be fluent, concise, and confidently addressed to the wrong person. That is why cross-channel reply tools need tests for identity and context, not only tests for whether the model returned a sentence. I recently changed Riffwise so each channel has its own identity and tone profile
Posted on Nov 2 • Edited on Nov 2 • AI-assisted
A reply draft can be fluent, concise, and confidently addressed to the wrong person. That is why cross-channel reply tools need tests for identity and context, not only tests for whether the model returned a sentence.
I recently changed Riffwise so each channel has its own identity and tone profile, with a neutral fallback when a profile is missing. The design decision is covered in a separate post; this one is the regression checklist I use to keep those boundaries visible.
Given: One user has a casual profile with a distinctive example on one channel, and a professional profile on another.
When: They draft from a conversation on each channel.
Expect: Each draft uses only the active channel's profile. Adding an example to one profile changes nothing on the other.
Given: The active channel has no saved profile, while a different channel has a detailed one.
Expect: The draft uses a neutral baseline. It does not silently borrow the other profile. The interface makes the fallback understandable.
Given: A thread contains several named participants, including a message that thanks one person and a reply box owned by another.
When: The current user drafts a response to a selected message.
Expect: The draft responds to the selected message's author and never assumes that the account owner was the person being addressed. Repeat the case with nested replies and quoted text.
Given: The same account uses different roles and writing styles in different spaces.
Expect: The app selects the relevant channel profile. It does not change who the account holder is, infer a recipient from a saved bio, or carry over another channel's personality.
Given: A draft has been generated and is ready.
Expect: Generating a draft does not publish or send anything. Insertion, copying, editing, and ignoring remain deliberate user choices.
These cases are cheap to run and catch a class of errors that ordinary prompt-quality checks can miss. A useful log record should say which profile was selected and why, without storing more conversation text than the product needs.
The lesson from our implementation is simple to state: keep the speaker, recipient, channel, and tone as separate pieces of context. Test their boundaries explicitly before tuning the wording.
I build Riffwise, a desktop Chrome extension for drafting replies from social conversations and Gmail. Selected context goes to Riffwise and OpenAI; the person reviews and edits before inserting, and the extension does not post or send automatically. Free includes 40 successful generations per month with no card; desktop Chrome and an account are required: https://chromewebstore.google.com/detail/riffwise-%E2%80%94-ai-reply-gener/nmnmflphiibhdnnbghffjhclhlehelle?utm_source=devto&utm_medium=community&utm_campaign=riffwise_202611&utm_content=showdev_identity_regression_tests
Disclosure: I build Riffwise. AI helped edit this article; the product decisions are mine.