Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning

This month Elorian launched its first visual reasoning model, Glimpse. It’s being promoted as “a new category of visual thinking model: a system that reasons about the physical world from what it sees.”

So how does Glimpse compare to both frontier models and world models in vision tests? A new benchmark called DecoyCount, co-released by Elorian, provides some answers.

We quizzed CEO Andrew Dai on the benchmark, and also talked about Elorian’s positioning in a crowded models landscape.

Elorian emerged from stealth in April with $55 million in seed funding, co-led by Striker Venture Partners, Menlo Ventures and Altimeter Capital. Now, in partnership with AI data company Mercor, Elorian is announcing the DecoyCount benchmark, which tests whether language models genuinely count objects or rely on visual shortcuts and substitution heuristics.

According to a paper accompanying the release, frontier models scored below 50% accuracy on DecoyCount — the highest was GPT-6 Astra, with 43.17%. Elorian’s new model isn’t mentioned in the paper, but Dai told us that Glimpse scored 50.48% in an internal test.

Regarding the frontier models, the paper concludes that “stronger general reasoning does not by itself generalize to aligned visual reasoning.”

How the benchmark works: “DecoyCount exposes interpretable counting failures across seven mechanisms.”

To make sure it was a like-for-like comparison with the frontier models, we asked whether the paper’s evaluation images were withheld from Glimpse’s training?

“The evaluation images and text were withheld from our training, and our contamination check found no overlap in the training data checked,” Dai replied, via email. “Without any cues, Glimpse scored 50.48% on the 1,660 DecoyCount items, making it the only model in the comparison to clear 50%. The closest was GPT-6 Astra at 43.17%, about 7 points behind, and Gemini-3.8-flash scored 32.71% while Qwen3.8-max scored 28.31%, and Claude-Fable-5.1 came in at 25.22%.”

Dai did note that Glimpse’s results “come from a single run of the model on its own, without an agent or tool harness,” whereas the six models from the paper were each averaged across three runs under their reported settings.

DecoyCount benchmark results for frontier models over 3 runs each.

But what exactly is DecoyCount revealing that existing model benchmarks miss? According to Dai, it tests both vision and reasoning together — which most benchmarks elide.

“If a person sees an apple in front of a mirror, we know that there’s just one apple and it’s being reflected. But a model might miss the apple entirely (a vision problem) or count the apple and its reflection as two (a reasoning problem). But most benchmarks that test for this error only check the final number, so both mistakes are treated the same way, which is an issue if you want to fix the root problem and can’t tell if it’s an issue of vision or judgement. ”

Frontier Model Comparison in DecoyCount paper

Andrew Dai isn’t a Johnny-come-lately on the AI scene. In 2015, while working at Google, he co-authored a paper entitled Semi-supervised sequence learning. In this paper, Dai and his then colleague Quoc Le demonstrated an early version of the pretrain-then-fine-tune approach later adopted in large language models. Over the decade that followed, Dai worked for Google DeepMind on GLaM MoE, PaLM 2 and then Gemini.

Latent Space is read by the people shipping agents, models and infrastructure. Enquire about sponsoring this slot.

Dai only left Google this year, in order to co-found Elorian. Part of his motivation was to push the boundaries of what vision models can do. His approach, he explained to Latent Space, is based on a concept in human psychology called visual thinking.

When most people “think about spatial problems or visual problems,” he explained, “they visualize the problem [by] generating images and editing images in their mind.”

He gave an example of buying a sofa and mentally picturing what it will look like in your living room.

There’s also manipulating an image in your mind; for example if you’re counting people around a table, you might mentally “cross them off one by one,” Dai said.

This is what Elorian aims to do in its models: teach them visual thinking “that involves generation and editing images as part of the reasoning trace, along with language.”

The key point is that image generation isn’t just an output for a vision model — the generated or edited image is part of a chain of reasoning.

Visual reasoning is still a work in progress, but there have been some impressive releases this year in “world models” — where vision is a key component. Two that stand out are World Labs’ Atlas in September (the company was subsequently acquired by AMD ) and Google’s just released Genie 4.

When World Labs announced Atlas, it was described as an “omni world model for spatial intelligence.” But what Atlas enables — basically a 3D simulation of a world — isn’t the same as visual reasoning.

According to Dai, world models enable “direct action on a world,” whereas Elorian is more focused on what he calls “indirect action.” By that he means using visual reasoning to inform a decision or workflow (what Glimpse does), rather than building simulated environments that allow navigation and manipulation (what Atlas does).

As an example, you might use visual reasoning to decide whether to reject a user-contributed video for violating a policy. “You don't need a 3D rendering to do that,” said Dai.

Other business opportunities for Elorian include robotics, engineering, and defense applications. Dai added that architects need to regularly check building codes and compliance — an ideal task for a vision reasoning model.

In two robotics benchmarks, WGO Bench and ERQA, Elorian claims to rate equal or higher than frontier models, and at a much cheaper price:

Getting back to counting decoys as one measure of success for vision models, Dai argues that reasoning about what a model sees is crucial to its accuracy and effectiveness in real-world applications.

He gives an example of a warehouse robot counting items in a box.

“A familiar 12-slot tray might only hold 10 parts, and a 24-pack label doesn’t mean 24 items are still in the box,” said Dai. “A model relying on what it expects to see will face many issues in real-world situations, and Glimpse is designed to work through the image instead of jumping to a quick conclusion. We’ve designed Glimpse to ground the count in visible evidence as opposed to basic cues that could lead other models astray.”

Elorian goes so far as to claim that visual reasoning is a foundational requirement for AGI (artificial general intelligence). From the Glimpse launch post:

“We believe building the full stack across systems, data, model architecture, and training — from the ground up for multimodal intelligence — can unlock AGI in the physical world.”

Dai acknowledges there’s “still a huge gap between humans and machines.” But he thinks visual reasoning is the key to closing that gap.

“I think the way to think about visual reasoning is as a fundamental step of reasoning in human intelligence,” he said. “And ultimately, if we want to build anything that's like AGI, we can't just skip visual.”