KI-Agentenschwarm für 98 Dollar getestet: Ein einziger Aufruf für 2,5 Cent lieferte dasselbe Ergebnis
Ein Test des Open-Source-Tools MiroFish untersuchte, ob ein Schwarm aus vielen KI-Personen bessere Prognosen liefert als ein einzelnes Modell. Das Ergebnis: Ein einzelner, günstiger Claude-Opus-Aufruf erreichte dieselbe Vorhersagequalität wie der teure Agentenschwarm.
Posted on Oct 26 • Originally published at orcalayer.com
Multi-agent swarms are a popular way to get "more than one model" out of an LLM: generate dozens of personas from your documents, let them post and argue on a simulated social network, then read off what the crowd thinks. We tested one, the open-source MiroFish engine, as a forecaster on four live Polymarket markets, with Claude Opus 5.5 behind every agent. The full results are on our blog, and every forecast, prompt, seed file and persona correction is on GitHub.
Short version: the swarm landed 0 to 3 points from one direct Opus answer on all four markets, at 500 to 900 times the cost. This post is about how to evaluate an agent swarm so you can see that, and four things we would check in any multi-agent setup.
The most important number in the whole test was not a swarm output. It was the same model asked the same question directly, with the same seed files and instructions, five times, median taken:
Without the baseline, a swarm result looks like an emergent crowd opinion. With it, you can see the crowd is the base model's view of the evidence, re-voiced 34 to 56 times. Sonnet showed the same pattern (50 vs 52, 49.5 vs 50).
In our first round, repeat runs of the same setup moved the median by up to 10 points, so a 0 to 3 point gap is noise. One run per configuration is a real limitation; we spent the budget on Opus instead of repeats.
Why it happens: every persona is written by the same model from the same documents. The agents share the same facts, framing and blind spots. Diversity of roles is not diversity of information.
MiroFish ends a run with a report agent that writes a summary and its own probability. On the Bank of Canada market, three report agents said 89 to 93, and the Opus report had a chapter titled "Agent Consensus: Banks and Economists Converge on 'No Move'".
We asked each agent directly instead. Median: 45. Only 13 of 34 agents put YES above 50.
The cause was mundane: the report agent tries to interview the agents, but by then the simulation had shut down, so it wrote from the knowledge graph built from the seed files and never saw the agents' answers. If a framework gives you a "consensus" summary, check that it actually read the outputs it claims to summarize.
Before each run we compared every generated persona with the seed files, first with a second model, then by hand. Opus 5.5 still produced about 130 factual errors across four markets. Several hit the exact fact a market resolves on:
It also liked to end the simulation setup with a verdict ("the narrative should drift toward cautious confidence that the pause holds"). That is the generator steering the outcome before any agent speaks. We removed those sentences and 11 opening posts with wrong facts or ready-made conclusions, and published originals and fixes.
One way to make this routine: for every persona, extract its factual claims as a list, check each against the seed files, and flag anything not supported. A cheaper model is fine for the first pass; a human for anything that touches the resolution criterion.
Every agent re-sends its persona, tool list and history on every turn, so cost grows fast with agents times rounds.
Prompt caching cut the Opus bill by about 40%; without it a run would have been $23 to $40. The whole round, with preparation and persona checks, was $98.56.
A forecasting test is only honest if the numbers are fixed before the outcome. We pushed forecasts.csv to GitHub before any market resolved, and GitHub's event log records the push time. The scorecard (Brier score and absolute error per source, including the market price) comes after the last market closes on October 31.
Four markets this round, eight across two rounds: an illustration, not a ranking. One run per configuration. Hand-corrected personas, so an uncorrected swarm might behave differently. Accuracy comes after resolution.
The takeaway so far: putting a frontier model behind every agent did not make the swarm a different forecaster from the model it runs on. If you want what the model thinks, ask it once.
Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.