What 1,500 graded AI runs taught us about using ChatGPT, Claude and Gemini at work

At AI Professions we test AI models on real tasks from ten professions: accountants, lawyers, real estate agents, healthcare staff, engineers, sales managers, travel agents, event planners, construction managers and interior designers. Our open benchmark now holds 1,500 graded runs: 100 prompts, 5 models, each prompt run 3 times. Every prompt has 5 pass/fail criteria, and we locked them with a hash before any model saw the task.

Here are the five lessons that matter most if you use AI for client work.

In round 1, 78 of 110 failed verdicts (71%) came from the model adding something that was not in the input. Only 24 failures were missing or changed facts, and 8 were length problems.

What to do: when you review an AI draft, don't only check whether the facts are right. Look for sentences you did not give it.

GPT-6 Astra scored 99% raw in round 1, against 92% for Claude Opus 5.5 and 91% for Claude Sonnet 5. But GPT was graded by a Claude judge, and the Claude models were graded by GPT. When we swapped judges on the same 30 Claude runs, the GPT judge passed 93% of criteria and the Claude judge passed 98%. Most of the raw gap was judge strictness, not model quality.

What to do: be careful with any "Model X wins" headline that doesn't name the judge.

The same models lost points in other languages. Under one judge, Claude Opus scored 92% on the English/Spanish set and 79% on the French/Portuguese set. Sonnet fell from 91% to 70%. Gemini 4 Argon held up best, going from 91% to 80%.

What to do: if your clients read French, Portuguese or Spanish, test your prompts in that language. Don't assume the English result carries over.

On Excel and Microsoft 365 tasks, scores ranged from 83% to 93%. Older lookup functions like VLOOKUP caused 20 failed verdicts. The formulas could work, but they didn't meet the stated requirement for XLOOKUP or INDEX/MATCH.

What to do: say exactly which functions and output format you need, and check that the answer follows it.

Our tests measured whether the outputs met written criteria. They did not measure whether a professional should sign the result. Use AI for first drafts, summaries and checklists. Keep confidential client data out of tools whose data terms you haven't checked.

All prompts, rubrics, transcripts and verdicts are free to reuse under CC BY 4.0:

AI Professions is an independent portal that tests AI tools on real professional tasks in English, Spanish, French and Portuguese.

Disclosure: this post was drafted with AI assistance and checked by the AI Professions team against the published benchmark data.