Neural network study sheds light on how AI chatbots 'reason' like humans

edited by Lisa Lock, reviewed by Robert Egan

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

Add to Preferred Sources “A Sunday on La Grande Jatte,” Georges Seurat, 1884. Credit: Public domain/courtesy Wikimedia Commons When humans write sentences or solve math problems, they are thinking symbolically—each number, variable or word is expressed by a symbol. While AI chatbots do not "think" that way, if prompted, they can offer cogent advice about writing, math and many other topics as if a person were offering it.

How artificial intelligence systems can do this is poorly understood because they are not directly programmed by humans. Instead, they are trained to develop their own internal strategies by analyzing massive amounts of data. And while the use of AI chatbots will become increasingly common, the fact that people do not fully understand how they operate internally makes them difficult to trust—especially in sensitive situations.

A new study led by Yale computational linguist Tom McCoy posted to the arXiv preprint server provides insight into how large language models (LLMs)—the AI systems trained on large datasets to generate human-like texts—can ably perform tasks, like solving math problems or writing computer code, that appear to require symbolic reasoning. The study provides evidence that the internal representations of neural networks implicitly realize symbolic structure.

At the heart of this knowledge gap, the researchers say, is that successful LLMs, like Claude or ChatGPT, don't operate on a symbolic model of intelligence, in which discrete units of text—words, for instance—are combined in structured ways, creating a logical formula.

Rather, the neural networks powering these chatbots are composed of vectors, which are long lists of numbers that enable LLMs to process information. While those vectors might seem inadequate for capturing the symbolic structure of language, logic, mathematics and other cognitive domains, advanced LLMs dramatically outperform prior AI systems that were built on symbolic structure, McCoy said.

"How do we reconcile their success with how we've always thought intelligence worked?" said McCoy, an assistant professor of linguistics in Yale's Faculty of Arts and Sciences and the study's lead author.

For the new study, McCoy and his co-authors analyzed the internal functions of several high-profile LLMs. They found that neural networks implicitly grasp symbolic structure within the numeric lists that drive them.

"Our analysis demonstrates that, despite appearances, the vectors powering LLMs are organized in a way that is equivalent to the symbolic structures that drive much of human cognitive function," McCoy said. "Think of a pointillist painting. At a low level, the individual dots that compose the painting have no interpretable human meaning, but the way they're arranged gives rise to a high-level structure that we can understand. Suddenly, you're seeing people gathered in a Parisian park on a beautiful day.

"With LLMs, we show that vectors are organized in a very particular way that gives rise to emergent symbolic structure—symbolic structure that is present at a high level despite not being apparent at a low level."

The study was co-authored by Paul Soulos of Microsoft, Tal Linzen of New York University and Paul Smolensky of Microsoft Research.

"LLMs have become an essential technology in AI," Linzen said. "It is disconcerting that we do not fully understand how they structure information because that might lead them to behave in surprising or unsafe ways. Our results significantly advance our understanding of these models and suggest a path toward controlling them more effectively."

Vector representations of a variety of neural networks can be closely approximated with symbolic structure, the study showed. To do this, the researchers replaced LLMs' vectors with tensor product representations (TPRs)—mathematical constructions that represent information as combinations of "fillers" and "roles." For example, in the sentence "cats chase dogs," the words are considered the fillers while the grammatical parts they play—subject, verb, object—are the roles. Likewise, in a fraction, the numbers are fillers while the numerator and denominator are the roles.

The researchers found that they could replace an LLM's entire representation-generating process with "role-filler" approximations embodying symbolic structures, and the AI system's behavior would remain largely unchanged. The finding held for small-scale neural networks trained to manipulate lists as well as for seven large-scale LLMs operating in four domains that have long been central in research on symbolic intelligence: language, arithmetic, logic and computer coding.

They also found that they could modify an LLM's behavior in targeted ways through precise interventions on the model's internal vector representations. For example, if the LLM's input was "the clever doctor helped a lawyer," the researchers could edit the component of the vector that encodes "clever" as an adjective describing the sentence's subject and change its position so that it modifies the object. The LLM would then behave as if the input had been "the doctor helped a clever lawyer," McCoy explained.

The ability to modify the LLM's behavior in this manner shows that the AI systems' internal operations are reliant on symbolic structure, he said.

"What we've found is that LLMs naturally converge to representations with role-filler structure," McCoy said. "This shows that LLMs utilize a particular type of symbolic structure, but our work only scratches the surface of what must be studied to truly understand how neural networks encode information."

R. Thomas McCoy et al, The Emergent Symbolic Structure of Artificial Neural Networks, arXiv (2026). DOI: 10.48550/arxiv.2608.29530 Journal information: arXiv

BA art history, MA material culture. Former museum editor, paramedic, and transplant coordinator. Editing for Science X since 2021. Full profile →

Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →