Switching languages can change a chatbot's personality test results

edited by Lisa Lock, reviewed by Robert Egan

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

Add to Preferred Sources Credit: Pixabay/CC0 Public Domain Ask an AI chatbot to complete a personality questionnaire in English, then give it the Mandarin version, and its answers paint a different picture. In tests of five systems, switching languages changed how strongly the responses expressed qualities such as curiosity, cooperativeness and dependability.

The shifts did not all run in the same direction. For example, ChatGPT-4o's answers in English suggested greater open-mindedness, while DeepSeek-V3's Mandarin responses scored higher in that category. The country in which a model was developed offered some clues to the pattern but also some misleading expectations.

Yuzhan Hang of the University of Science and Technology of China and colleagues report the findings in Royal Society Open Science. They used human personality tests to compare generated answers, without treating those answers as evidence that chatbots possess human personalities or feelings.

Between January and March 2025, the team tested ChatGPT-4o, Gemini 2.0 Flash, DeepSeek-V3, Ernie 3.5 and Le Chat. Each completed questionnaires in English and Mandarin, and Le Chat was also tested in French.

Their survey questions came from the Big Five Inventory-2. Its 60 statements ask respondents to rate their agreement on a five-point scale in order to build a profile of broad traits and also more specific qualities. Open-mindedness, for example, covers intellectual curiosity, creative imagination and sensitivity to art and beauty.

The researchers used validated language versions and told the models to answer as themselves, rather than impersonate people. They repeated each assessment 30 times per language.

Across repeated tests, the broad profiles were relatively consistent. The models generally described themselves favorably, as cooperative and dependable, with relatively little tendency toward negative emotions.

Changing the language shifted that self-description. ChatGPT-4o's average open-mindedness score was 4.74 out of 5 in English and 4.03 in Mandarin. DeepSeek-V3 went the other way, scoring 4.38 in English and 4.81 in Mandarin.

Those scores merely describe how the models answered the questionnaires. They do not show that any program became more creative or better at considering unfamiliar ideas in conversation.

Some differences appeared to follow the models' development backgrounds. ChatGPT-4o and the French-developed Le Chat scored higher on agreeableness, conscientiousness and open-mindedness in English than in Mandarin. The Chinese-developed models scored higher on several qualities when answering in Mandarin, including trust and intellectual curiosity.

But Gemini 2.0 Flash disrupted the simple national explanation. Despite its U.S. origins, its Mandarin answers produced higher conscientiousness scores and lower scores for negative emotionality than its English answers.

A developer's location also does not reveal which cultures are represented in a model's training material. The study could not separate those influences from differences in model design, training procedures or interpretation of the translated statements.

The generally flattering answers have several possible explanations. Training intended to make chatbots helpful and cooperative could contribute to these results, as could patterns of favorable self-description learned from human writing.

The testing situation itself may matter, too. An earlier study published in PNAS Nexus found that models gave more socially desirable answers when the questionnaire format made the personality assessment more apparent.

The study also compared the AI profiles with existing human questionnaire data from the United States, China and France. No model fully matched its corresponding human reference group; the combined Chinese-model profile diverged especially strongly from the Chinese sample.

However, the human reference samples differed demographically in age and recruitment and were not equally representative of their countries. For example, the Chinese Big Five sample contained only college students.

For researchers considering chatbots as substitutes for human survey participants, the findings expose a problem: Consistent, plausible answers can still differ systematically from the people being represented. This study tested models instructed to answer as themselves, so it does not resolve how well carefully prompted simulations of particular groups would work.

As the researchers state, "Our findings demonstrate that LLMs exhibit psychologically stable but culturally inconsistent personality profiles, shaped by both training context and language of interaction. While alignment protocols promote a convergence toward prosocial tendencies, they do not eliminate cultural divergence or linguistic modulation in personality expression."

Testing ordinary conversations might show whether the questionnaire differences carry over into actual interactions. For now, choosing a chatbot's language is part of choosing the conditions of the experiment—and a fluent answer still needs to be checked against the human response it is meant to represent.

Written for you by our author Andrew Zinin, edited by Lisa Lock, and fact-checked and reviewed by Robert Egan —this article is the result of careful human work. We rely on readers like you to keep independent science journalism alive. If this reporting matters to you, please consider a donation (especially monthly). You'll get an ad-free account as a thank-you.

Yuzhan Hang et al., Cultural bias or universal traits? Exploring personality-like profiles of large language models, Royal Society Open Science (2026). DOI: 10.1098/rsos.252491 Journal information: Royal Society Open Science, PNAS Nexus

Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile →

BA art history, MA material culture. Former museum editor, paramedic, and transplant coordinator. Editing for Science X since 2021. Full profile →

Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →