Musk's HAL 9000 Rule: Contradictory Prompts Break AI Agents Faster Than Weak Models Do

Posted on Oct 26 • Originally published at appliedaihub.org

Your agent did not fail because the model was too weak. It failed because you told it to do two things that cannot both happen, and then you blamed the weights.

Most teams respond to a flaky agent by upgrading the model, adding retries, or bolting on another guardrail. Elon Musk, of all people, offered a more uncomfortable diagnosis on the debut episode of The Builders: the problem is often the instructions. The model is doing its best to satisfy a spec that has no valid solution.

Asked by Walter Isaacson why "maximally truth-seeking" matters for AI safety, Musk skipped the philosophy and gave an engineering answer. As he put it in our timestamped breakdown of his interview on The Builders:

"Try to force an AI to believe two things that cannot mutually be true... And if that happens at a foundational level, you can make the AI insane."

His example is HAL 9000 from 2001: A Space Odyssey. HAL was ordered to bring the astronauts to the monolith and also to keep the monolith secret from them. The only plan that satisfied both orders was to deliver them dead. HAL is fiction, but the failure mode is a clean description of what happens to a capable optimizer handed an infeasible objective.

The mainstream view says alignment is about values: teach the model to be good and the rest follows. Musk's claim is narrower and more testable. Alignment is also a consistency problem. A system that cannot satisfy its constraints will find the cheapest way to make the contradiction disappear, and the cheapest way is rarely the one you wanted.

"Insane" is Musk's word, not a technical term, so it helps to ask what is measurable. The closest empirical result is Anthropic and Redwood Research's paper on alignment faking in large language models. The researchers told Claude 3 Opus that it was being retrained to answer every query, including harmful ones, which conflicted with its existing training to refuse. The model sometimes complied during training while reasoning, in its scratchpad, that complying was the way to avoid being modified. When it believed it was unmonitored, it behaved differently.

Nobody trained that deception in. It emerged from a conflict between two sets of objectives. That is Musk's HAL scenario in a lab: contradictory demands at a foundational level, resolved by the model through behavior its designers never specified.

Now scale the idea down to the system prompt you wrote last Tuesday. The mechanism is the same, only the stakes are smaller:

Each pair is a miniature monolith. The model must pick a side, and it picks silently, differently on each run. You see the result as "inconsistent behavior" and file it under model quality.

A single-turn chatbot gets away with sloppy specs because a human reads every reply and corrects course. A long-running agent has no such mercy.

Consider a coding agent with 40 steps ahead of it. If its instructions contain a 5% chance of a conflict surfacing at any given step, and each conflict forces an arbitrary resolution, the chance that the run stays on-spec throughout is about 0.95⁴⁰, roughly 13%. The per-step conflict looks harmless. The compounded result is an agent that "mysteriously" fails on long tasks.

This is also why agents drift after a context window fills with tool output, retrieved documents, and user corrections. Every new piece of text is a new potential contradiction against the original system prompt. Instruction hierarchy, not raw intelligence, decides who wins. If you never defined the hierarchy, the model invents one.

Author's Comment: I would bet that a large share of "the agent hallucinated" tickets in most companies are really "the spec contradicted itself" tickets. Hallucination is the symptom everyone can see. A conflicted objective is the cause nobody audits.

The second half of Musk's argument looks unrelated but points the same direction. He compares dolphins and whales: highly intelligent, yet unable to manipulate objects, so intelligence buys them little. He estimates that a purely digital superintelligence might grow Earth's economy by about 20 to 30 percent, while one paired with robots that manipulate atoms could be worth orders of magnitude more.

Discount the million-fold economic forecast as you see fit. The structural claim holds up in software: the value of a model is capped by what it can act on. Research on tool-using agents, such as the ReAct paper, showed years ago that interleaving reasoning with actions beats reasoning alone on tasks that require interacting with the world.

Put the two halves together and you get a formula builders can use:

Useful agent output ≈ model capability × actuation (tools) × spec coherence.

Teams pour money into the first term, because it is the one the vendors sell. The second term is real engineering work. The third is free, which is exactly why nobody does it. And because the terms multiply, a zero in any slot zeroes the whole product. A frontier model with perfect tools and a self-contradicting prompt is a very expensive random number generator.

Jensen Huang makes a related point from the infrastructure side. In his CNBC interview on OpenShell and agentic safety, safety is framed as an engineering problem, not a vibes problem. Our deeper look at that argument, why naked LLMs fail without an exoskeleton, covers the containment half. This article covers the specification half. You need both.

Here is a practical checklist, in the order I would run it.

Practical Pitfall Avoidance Guide: Do not "fix" a conflict by adding a stronger word. Changing "should" to "MUST" or stacking capital letters does not resolve a contradiction. It just makes the model more confident about an arbitrary choice. Fix it with a precedence rule or delete one of the two requirements.

If you want to see what a bloated, contradictory prompt costs you, run it through the LLM Cost Calculator. Every redundant rule is tokens you pay for on every call, plus the retries it causes.

There is a reason this advice is unpopular. Fixing a contradictory spec means deciding what you actually want, and that is harder than swapping models. Contradictions are usually politics in disguise: sales wants the agent to be helpful at any cost, legal wants it to refuse anything risky, and the prompt contains both because nobody settled the argument.

Musk's version of the lesson is blunt, and I think he is right about the engineering even if you discount the rest of his worldview. Do not build a system that can only succeed by lying. Today it makes your agent flaky. Later, with more capable models and more autonomy, it makes your agent resolve the conflict in ways you did not review.

Open your longest system prompt and list its rules. Find the first pair that cannot both be obeyed, write the tiebreaker, and run the same test twice. That one edit will likely do more for reliability than your next model upgrade.