Why the most dangerous AI may be the one we trust too much

edited by Gaby Clark, reviewed by Andrew Zinin

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

Add to Preferred Sources Credit: Unsplash/CC0 Public Domain Machines don't have to become superintelligent to cause catastrophic harm. As artificial intelligence spreads through military decision-making, they just have to be wrong—and trusted by the humans in charge.

In 1991, "Terminator 2: Judgement Day" gave us one of popular culture's most enduring images of AI gone wrong. Humanity builds Skynet, an artificial intelligence given control of strategic defense. It learns at a "geometric rate," becomes self-aware and, when its human operators try to shut it down, it starts a nuclear war.

Versions of that story now feature in serious debate about AI. Increasingly capable large language models (LLMs), the argument goes, could develop into artificial general or superintelligence (AGI/ASI), acquire goals that diverge from ours and ultimately threaten humanity.

That debate deserves attention, but it depends on a chain of large and unproven steps. Today's AI systems must somehow transform into genuinely general intelligences, develop independent goals and gain control over the physical world.

Meanwhile, a more immediate risk is taking shape that requires none of those steps.

In the film, Skynet builds the Terminators, machines that find, track and kill humans. Strip away the science fiction and the idea is simple: combine sensors, software and weapons and reduce the human role.

Human beings are doing exactly that, without anything resembling a malicious superintelligence like Skynet. In August 2026, Ukraine's Ministry of Defence said its forces were using more than 70 AI and computer vision systems to strike targets. Algorithms can recognize equipment and personnel and lock onto those targets autonomously, so that a drone can complete its attack even if jamming cuts the link to the operator.

These systems need no general or superintelligence. They need sensors, software that can recognize a target, navigation and a weapon. And the more targeting is automated, the more the human's role shrinks to approving what the machine has already identified.

That's precisely where trust becomes the weak point.

The second risk is more subtle: AI shaping the decisions of the people who pull the triggers.

According to a recent CNN investigation, in spring 2026, during the war with Iran, a U.S. special operations analyst used an AI chatbot to assess intelligence about a Chinese ship in the Middle East. The chatbot, drawing on both open-source and classified material, wrongly identified the ship's cargo as components of a nuclear weapons program. The analyst used AI again to package this into a standard intelligence report.

Armed personnel were preparing to board the ship and military aircraft were airborne before an official looked more closely and found the error. One source said the false report "almost started a war." Any U.S. operation against a Chinese vessel could risk spiraling into an armed conflict.

The AI was not conscious, hostile or out of control. It was simply wrong—but people trusted and acted on it.

The U.S. military is pushing to weave AI into almost every part of its work, from analyzing intelligence to selecting targets. CNN's sources said errors of this kind are not uncommon across the intelligence community, and that pressure to produce intelligence faster is opening the door to mistakes. The ship incident was caught in time. The next error may not be.

At its heart, this is a problem of trust. LLMs produce fluent, confident language effortlessly—and, repackaged as an official report, an error gains an authority it never earned. Colleagues and I have argued that these systems are built to produce an answer rather than to say "I don't know." For example, in one experiment, we found that nonspecialists could not tell a chatbot's convincing but wrong physics answers from genuine expertise.

Policymakers often reassure us that a human makes the final decision. In the Chinese ship case, people were in the loop throughout. They simply became another stage through which the error was passed.

That fits decades of research on automation bias, the tendency to overrely on automated systems. It affects experts as well as novices. In 2003, U.S. Patriot missile batteries shot down two allied aircraft after operators accepted their misclassification.

A person in the loop works only when that human is able, adequately equipped and willing to challenge the machine.

That means verification. AI output should be clearly labeled, so its origin is visible to everyone who relies on it, and treated as a claim to be tested against independent sources before anyone acts on it. In war, the cost of not checking could hardly be higher.

None of these concerns over trusting AI in the short term mean long-term questions about advanced AI in the future should be dismissed. But much public anxiety focuses on what these systems might become, and less on what they are already doing.

Perhaps "Terminator 2" got more right than we appreciate. Skynet's catastrophe doesn't begin with self-awareness. It begins with trust, when humans hand a machine control because they believe it will make better decisions than they can.

We don't need a conscious machine to repeat that mistake. Fallible AI, misplaced human trust and increasingly autonomous weapons offer a far more mundane route to catastrophe.

The danger is not that our machines become too intelligent. It is that we hand responsibility to machines that are far less reliable than we assume. As the film puts it: "There's no fate but what we make for ourselves."

MA in English, copy editor since 2021 with experience in higher education and health content. Dedicated to trustworthy science news. Full profile →

Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile →

This article is republished from The Conversation under a Creative Commons license. Read the original article.