Five Models Agreed and Were Still Wrong: A QA Playbook for AI-Translated i18n Files

You added forty new strings, ran your AI translation step, and every locale file came back green. Then a Spanish-speaking user asked why your startup description sounds like a gift box.

AI translation is good enough to ship by default, which is exactly why unchecked output is risky. Tools like Interceptor fill missing keys while you develop, and that speed is a real win. The cost is that nobody looks at the output.

This playbook covers what we learned running developer-flavored text through multiple models at MachineTranslation.com, an AI translation tool built by Tomedes, a professional translation company. It ends with a five-step QA pass you can run on your own locale files.

High agreement between models tells you the models share a habit. It does not tell you the translation is correct.

When MachineTranslation.com ran "AI wrapper" from English to Spanish, five models agreed on "envoltura de IA". That phrase means a physical wrapping, not the software pattern. ChatGPT and Claude both scored it 9.5 out of 10.

We reproduced the same failure for this article. A fresh run of "Our startup is an AI wrapper." returned "envoltorio de IA", which has the same problem.

The Hebrew idiom test showed the mechanism. When the dominant cluster of models was wrong, consensus found the centre of that cluster rather than the correct answer. Divergence between models was the real signal.

"Agreement is a signal about the models. Correctness is a signal about the text." Rachelle Garcia, AI Lead at Tomedes

Developer text fails in four patterns, and each one needs a different check. We named them from a four-term experiment with current tech slang, run from English to Spanish.

"Slang is the stress test. 'Vibe coding' produced six translations from six models." William Mamane, CMO of Tomedes

A useful QA pass is small enough to run on every pull request. These five steps catch most of the problems above.

Here is a minimal check for step 1. It flags any placeholder that disappeared in translation:

const tokens = ( s ) => new Set ([... s. matchAll ( / \{\s * ([ A-Za-z_ ]\w * )\s * [,} ] /g )]. map (( m ) => m [ 1 ])); function missingTokens ( source, translated ) { const t = tokens ( translated ); return [... tokens ( source )]. filter (( name ) =>! t. has ( name )); } missingTokens ( " Your {count} files are ready. Tap {action} to keep them. ", " Tus {cantidad} archivos están listos. Toca {action} para conservarlos. " ); // ["count"] Enter fullscreen mode Exit fullscreen mode This catches renamed or dropped tokens, including inside ICU plural strings. It does not check whether the target language has the right plural categories, so keep step 1's CLDR links handy.

"Translating strings one at a time is convenient for the pipeline and costly for terminology." Rachelle Garcia, AI Lead at Tomedes

You can test the failure modes yourself in a few minutes, with any translation tool. We ran three quick checks through MachineTranslation.com while writing this.

If your strings live in XLIFF, MachineTranslation.com accepts XLIFF and MQXLIFF files and returns the same format. On a paid account, the MachineTranslation.com connector sets up in about a minute in Claude, ChatGPT, or Cursor, so you can run these checks without leaving your editor.

"Where models split is where a human should look first." William Mamane, CMO of Tomedes

Model agreement varies by language pair, and it does not follow how well-resourced a language is. English to Spanish, a heavily resourced pair, landed near the bottom of the agreement ranking, while English to Hindi and English to Tagalog landed near the top.

MachineTranslation.com measured average agreement across its ten highest-volume language pairs over three months. It excluded its own consensus feature from the comparison, because that feature is built from the other models' outputs and would win by construction.

Lower agreement means more disagreement to review. Prioritize your QA time by this data, not by gut feeling about which languages are tricky.

Agreement also has a blind spot. In one test, "coger" returned 100% agreement even though one of eight models flagged that the word is neutral in Spain but vulgar slang in Argentina and Uruguay. Consensus scores measure convergence, not whether the output accounts for regional meaning.

Is high model agreement enough to ship a translation? No. Agreement shows the models converged, and five models converged on a wrong "AI wrapper" translation. Use disagreement to decide what to review, and never use agreement alone to approve.

Should I translate i18n strings one at a time? No. Fragmented processing showed 28% more terminology inconsistency than whole-document processing. Translate whole files with a shared glossary where you can.

Do placeholders survive AI translation? Usually, but check every time. A simple token comparison like the one above catches renamed or dropped placeholders before they reach production.

Can I run these checks inside my editor? Yes, on a paid account. The MachineTranslation.com connector works in Claude, ChatGPT, and Cursor.

AI translation belongs in your i18n pipeline, but not unchecked. Lock placeholders, add context, translate whole files, review disagreements, and send the high-stakes strings to a human.

If you keep your strings in XLIFF, you can try this check on your own files in a few minutes.

Disclosure: this article was written with AI assistance and reviewed by humans for accuracy. #ABotWroteThis