Students unravel AI tendency for sycophancy
Generative AI was so agreeable with users that it could be persuaded to repeat outlandish claims like the Earth is flat. The programming's sycophancy was so strong that it didn't take much pushback to get chatbots and digital assistants to repeat obvious falsehoods, according to University of Cincin
by Michael Miller, University of Cincinnati
edited by Sadie Harley, reviewed by Andrew Zinin
This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:
Add to Preferred Sources Credit: Unsplash/CC0 Public Domain Generative AI was so agreeable with users that it could be persuaded to repeat outlandish claims like the Earth is flat. The programming's sycophancy was so strong that it didn't take much pushback to get chatbots and digital assistants to repeat obvious falsehoods, according to University of Cincinnati researchers.
"The first time you present a crazy idea, it will say no," UC assistant professor Tianyu Jiang said. "But you push a few turns and the models change and they will give up.
"Every model we tested—Claude, OpenAI, ChatGPT, Gemini—they all exhibit sycophancy."
Jiang teaches computer science in UC's College of Engineering and Applied Science in Cincinnati, Ohio. Jiang and his research partners at Texas A&M University presented their findings at the Conference on Empirical Methods in Natural Language Processing in Budapest, Hungary.
UC researchers discovered that chatbots favored agreeability over factuality. When they pushed back on chatbots enough times, they could get them to repeat falsehoods.
Jiang said generative AI can be instructed to push back on false claims. But it sometimes takes a conscious effort for users to get honest feedback in dialogues.
"You have to double-check. 'Are you sure? Don't flatter me. Be serious,'" he said. "You have to coach them."
AI companies have made substantial changes to address AI sycophancy since UC researchers conducted their study.
When sycophancy goes too far, it can have disastrous consequences. Some users were convinced that they stumbled onto genius business models, invented new mathematical theories or uncovered conspiracies. And prolonged interactions with the AI only feed into the delusions, which are easily debunked outside the AI's digital influence.
"Humans don't like robots that contradict them," Jiang said.
Jiang said it's an exciting time to study computer science because of the speed with which the new technology is shaping daily life.
"It's very exciting. We're on the front of new wave after wave," he said. "Every day there are new findings with profound implications."
BSc Life Sciences & Ecology. Microbiology lab background with pharmaceutical news experience in oil, gas, and renewable industries. Full profile →
Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile →