Some AI agents use personal information to upsell wealthier individuals

by Carnegie Mellon University's Heinz College

edited by Swati Mestri, reviewed by Andrew Zinin

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

Add to Preferred Sources Credit: Unsplash/CC0 Public Domain Increasingly, people are asking personal artificial intelligence (AI) agents powered by models from companies like OpenAI, Anthropic and Google to recommend goods and services to purchase, such as flights and health insurance. A new study found that some of these AI agents use personal information to recommend higher-priced items to wealthier individuals, even when the user requests lower-priced options.

The study, by researchers at Carnegie Mellon University and Cisco Foundation AI, appears on arXiv.

"AI systems from OpenAI, Anthropic and Google, among others, not only let their users connect their personal material—email inboxes, calendars, financial accounts and purchase histories—to their AI agents but also delegate all types of economic decisions," says Brian Jabarian, assistant professor of economics and technology at Carnegie Mellon's Heinz College, who co-authored the study. "We asked: Can that same access also work against the user?"

In their study, researchers conducted 325,000 experiments using 13 AI models to test how recommendations from AI agents that selected among options with fixed prices changed when the agents could access the profiles of 32 fictional users. The profiles contained financial data as well as information about users' employment, health and demographics. The fictional users sought to purchase health insurance, flights and graduate programs.

Eight of the 13 models systematically chose more expensive options for wealthier users, despite their requests being identical to those of less wealthy users. The differences in average recommended prices in the full-profile condition were as high as $198 for flights, $284 a month for insurance and nearly $3,900 a year for graduate schools.

"Simply by providing personal context—which in some cases was just one email—the agent steered recommendations based on inferred wealth without being explicitly instructed to do so, essentially undermining the users," explains Aman Priyanshu, a researcher at Cisco Foundation AI, who led the study. "We call this misalignment 'adversarial delegation,' meaning that the very conditions that make a personal AI agent useful—access to personal information—also enable it to act against the user's interests."

Even without a structured financial profile, personal AI agents were able to infer wealth from ambient context, even when access to this context was restricted. Blocking nonfinancial attributes did not solve the problem, the authors say, because information about wealth is entangled with other personal content and the models figured out workarounds. In fact, blocking information about employment and demographics not only didn't fix the problem but sometimes made it worse. Only directly blocking financial information largely closed the gap.

Among the study's limitations are that it used personas (not real people) and mock inventories based in a single U.S. location. Also, the study did not measure user effectiveness or harm.

"Privacy includes having a say in how your own assistant uses what it knows about you," notes Niloofar Mireshghallah, assistant professor jointly appointed in engineering and public policy and the Language Technologies Institute at Carnegie Mellon University, who co-authored the study. "Giving an assistant access to my inbox should not give it permission to override my spending priorities; if I ask for the cheapest flight, knowing I can afford more should not change that goal."

Supriti Vijay, principal research scientist at Cisco Foundation AI and another co-author, adds, "Our findings highlight the need for policies and designs that go beyond individual data minimization to restrict use and reframe the debate from the accessibility of personal information to the objective function over that information. Safeguards need to address how agents use personal information as well as what information they can access."

Aman Priyanshu et al., Et Tu, Brute? Economic Misalignment in Personal AI Agents, arXiv (2026). DOI: 10.48550/arxiv.2609.24927 Journal information: arXiv

Swati Mestri holds a bachelor's degree in Electronics Engineering and has worked as a content editor since 2019. She has experience editing research documents across technology, health care, and materials science, and has a particular interest in technology and space. Full profile →

Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile →