Chatbots with personality? How AI describes itself and judges us

edited by Lisa Lock, reviewed by Robert Egan

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

Add to Preferred Sources Credit: Pixabay/CC0 Public Domain Texts generated by large language models, or AI-powered chatbots, exhibit characteristic "personality" traits. Such chatbots can also identify individual personality traits in texts. However, humans are more reliable at distinguishing between human-written and AI-generated texts, according to researchers at the Institute of Psychology at Johannes Gutenberg University Mainz (JGU).

"Chatbots are becoming part of more and more areas of our lives, from customer service and digital counseling services to personal chatbots that many people already use every day and even share their feelings with," said Sophie Körner, a research associate in the Department of Personality Psychology and Psychological Assessment at JGU and lead author of the study published in Scientific Reports. "That is why we need to understand how we perceive chatbot-generated texts and whether we can still distinguish them from human-written texts."

For their study, Körner, together with Dr. Sarah Osterholz and professor Boris Egloff, compared the self-descriptions of seven AI language models—ChatGPT-4o, ChatGPT-3.5, Llama-3, Gemma 2, Gemini 2.5, Phi-3 and Orca 2—with those of 70 humans. Using psychological questionnaires and open-ended questions, they asked the chatbots and human participants to describe their respective "personalities" in German.

For example, they were asked to rate statements about their self-esteem and the Big Five personality traits, such as openness, agreeableness and extraversion. The subsequent open-ended questions focused on their interests, hopes for the future and typical behavior in stressful situations. Each language model was questioned 10 times, giving the researchers a total of 140 texts to analyze.

The team first examined how similar the language models' responses were when questioned repeatedly. "Our results show that the language models do indeed attribute fairly stable personality profiles to themselves over time," said co-author Osterholz. Compared with the human participants, the language models presented themselves as more conscientious, agreeable and emotionally stable.

Next, the researchers asked 10 humans and six chatbots to evaluate the self-descriptions. Based on the texts, they were asked to assess personality traits such as openness, conscientiousness and self-esteem. Both humans and chatbots were able to draw conclusions about personality traits. However, the chatbots tended to rate some traits of the human authors, including openness and self-esteem, higher than the human raters did. Conversely, the human raters tended to assign higher scores to the language models than to the human-written texts for all traits examined, particularly emotional stability.

The raters were also asked to determine whether each text had been written by a human or an AI model. The human raters achieved an accuracy of 83%, compared with just 67% for the chatbots. The two best-performing AI models in this task, Gemma and Gemini, were about as reliable as the average human rater at identifying whether a text was written by a human or AI.

Both groups tended to classify texts as human-written more often, possibly to avoid misattributing human-written texts to AI. Specifically, chatbot-generated texts were more likely to be incorrectly classified as human-written than vice versa. However, AI models may become significantly more accurate at identifying whether a text was written by a human or AI as their training data increasingly include AI-generated texts.

The human raters primarily cited the length of AI-generated texts and their stilted, particularly technical or analytical language as distinguishing features. A computer-assisted linguistic analysis also revealed differences: AI-generated texts contained more time-related expressions and function words such as articles and prepositions, while human self-descriptions more often focused on leisure activities, feelings and social relationships.

According to the researchers, AI language models could also be used in psychological assessment and human resources in the future, for example, to obtain an initial picture of personality traits. Such applications are already being pursued in some workplace settings. However, it remains unclear how reliable these methods are and under what conditions they could be used.

"They could provide an automated, easily accessible initial assessment," said Körner. But since language models are trained on human-written texts, they also acquire biases, meaning distortions in judgment. It is therefore important to carefully examine whether personality assessments in sensitive areas should be left to algorithms, particularly from an ethical perspective.

Sophie T. Koerner et al., Personality expression and perception in large language models and humans, Scientific Reports (2026). DOI: 10.1038/s41598-026-75546-5 Journal information: Scientific Reports

BA art history, MA material culture. Former museum editor, paramedic, and transplant coordinator. Editing for Science X since 2021. Full profile →

Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →