AI often passes for human, but asking follow-up questions can trip it up

edited by Lisa Lock, reviewed by Robert Egan

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

Add to Preferred Sources Turing-like tests for three language tasks. Credit: Nature Human Behaviour (2026). DOI: 10.1038/s41562-026-02590-6 Reading a conversation and deciding whether a chatbot wrote it can be surprisingly difficult. In a study involving more than 72,000 judgments of human and AI responses, participants frequently mistook machine-generated captions, word associations and conversational replies for human work. However, in a smaller experiment, direct questioning proved more successful at exposing a chatbot, though even that comes with caveats.

Mengmi Zhang of Nanyang Technological University in Singapore and colleagues report the findings in Nature Human Behaviour. Their central point is that being able to pass as human depends heavily on how the testing is conducted.

The team assembled sample responses from 636 people and 37 AI models, then they recruited a separate pool of 1,916 human judges. The responses fell into six categories ranging from writing captions and holding conversations to identifying colors and objects in images.

Initial data collection took place in 2023–2024, with additional control experiments in 2025, so the findings capture an earlier stage of AI development. Although some tests included GPT-4o, Claude 3.7 and Llama 3.3, the findings do not establish how today's newer models would perform.

In the main language tests, the judges' overall accuracy in determining the author was roughly 53% to 57%, only modestly above the 50% expected from random guessing. They correctly identified captions as AI-generated just 46% of the time and AI-written conversational responses 47% of the time. Human answers were also frequently mislabeled.

These tests involved prepared material, with some obvious AI failures removed. For example, researchers discarded certain conversations that became repetitive or ended prematurely. The results therefore describe performance under controlled conditions, rather than detection of every kind of chatbot output.

The judges' scores did reveal that being a convincing imitator is different from giving an "accurate" answer. In one generated caption example, GPT-4 correctly described people on a beach but missed a visual trick that made a man appear to hold his own detached head.

"Being right or wrong is not the same as imitation," the authors write.

In a smaller experiment, the researchers also let judges question a person and a chatbot directly, choosing their own topics and follow-up questions before deciding which was which. In seven conversations where GPT-3.5-Turbo was instructed to pose as a person, judges identified it every time.

That perfect score came with possible giveaways. Response style and timing could offer clues, and some questions could trigger a safety response revealing that the chatbot was AI. The result therefore did not necessarily mean that judges could see through an otherwise convincing imitation.

To reduce the chance of such disclosures, the team ran a larger follow-up using a different model, Llama-3-70B-Instruct, with safeguards disabled. Across 188 trials involving 28 participants, judges made mistakes. However, they became better at identifying machine responses as conversations grew longer.

Because the live conversations and transcript tests differed in their models and instructions, the experiments cannot isolate the effect of questioning alone. They do suggest that asking follow-up questions can uncover clues unavailable in a finished transcript. But recognizing one chatbot in a small test does not establish that people can reliably distinguish AI from humans across different systems and situations.

Written for you by our author Andrew Zinin, edited by Lisa Lock, and fact-checked and reviewed by Robert Egan —this article is the result of careful human work. We rely on readers like you to keep independent science journalism alive. If this reporting matters to you, please consider a donation (especially monthly). You'll get an ad-free account as a thank-you.

Mengmi Zhang et al., Turing-like tests for language and vision demonstrate a narrowing gap between humans and AI, Nature Human Behaviour (2026). DOI: 10.1038/s41562-026-02590-6 Journal information: Nature Human Behaviour

Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile →

BA art history, MA material culture. Former museum editor, paramedic, and transplant coordinator. Editing for Science X since 2021. Full profile →

Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →