DeepSeek V4.1 Flash 1017 benchmark: same task, two builds, 0.24 and 0.66

Posted on Oct 19 • Originally published at leanzero.net

The best result of our Sunday came from a model that was one day old and billed like pocket change. DeepSeek put out V4.1 Flash 1017 on Saturday 17 October and called it "a minor update". On our two agentic coding benchmarks it scored above the V4.1 Flash release from September, which we tested a week ago: 0.6957 against 0.591 on the Gauntlet, and a Forge 2.0 median of 0.4520 against 0.3342. All day, that model cost us at least 83 cents in OpenRouter bills.

If your agent calls DeepSeek's own API, you may have been on 1017 since Saturday without choosing it. DeepSeek's change log says the API name and the price stay the same: "simply use deepseek-flash to access the latest model". Point releases get a news post and a changelog line, and then they change what your agent writes. This one changed what ours could build in half an hour.

It also built the same app twice and scored 0.2422 and 0.6618. I keep coming back to that spread, so I start there. After it I have three less flattering stories from the same day. NVIDIA's 120B Nemotron 3.5 Super VL finished 31st of 41 on one benchmark and lost its own context on the other. A local 30B model declared one name twice and reported the app ready. And our own public board had been understating what the runs cost.

I owe you a concession up front. The numbers come from one task per benchmark, one build or one pair per model, in our setup. That setup is goose, the open-source coding agent, in our own build (an unreleased one on Sunday; Mistral's Monday re-run used the 3.0.119 release), launched from our Goose Swarm desktop app.

Cloud models go through OpenRouter and local ones through our MLX engine, which is our own fork of the open-source Rapid-MLX server for Apple silicon. The cloud runs went out from a Mac Studio and a MacBook. The local Glimmer build ran on a 64 GB MacBook, and the local Nemotron attempt on the other MacBook, which has more memory.

None of this says any of these models is bad. Our base Qwen3.8-27B pair, at a 0.1852 median, is still the best local result on that MacBook, and none of the local candidates that finished today changed it.

We use two benchmarks. The Gauntlet (sb-7.2) has goose build a small payments app with two services and a 3D view, gives it 150 model calls and then runs 99 checks against whatever it built. Forge 2.0 hands goose a working Jira Forge app and a long spec for its v2. It gets 300 model calls, then 105 checks run against three seeded Jira sites. One model call can ask for several tool calls, which is why Nemotron shows 397 tool calls inside a 300-call budget. The Forge 2.0 board lists each model as a pair of independent builds. The Gauntlet board lists single runs.

Our chart from the run pages, our run logs and the verdict files, 18 October 2026. Starred rows are single unpublished builds, grey bars the September release run earlier in a different setup. Compare within a panel only. Mistral's build 2 finished after this chart was drawn, at 0.3304, and its Gauntlet re-run on 19 October scored 0.5462.

The run pages for the day: DeepSeek 1017 on the Gauntlet, its Forge 2.0 build 1 and build 2, and Nemotron on the Gauntlet. Numbers as read at 21:30 on 18 October; the live pages are the current word. Nemotron's and Glimmer's Forge numbers come from single unpublished builds, because our board only lists Forge 2.0 entries as two-build pairs and I cancelled the second build of both.

One more caveat, about the DeepSeek comparison. The board puts the two Forge pairs in different comparison groups (today numbered 4 and 2; the board renumbers them as groups are added), because something in our setup changed between 12 and 18 October. The two Gauntlet runs are two weeks apart. Both September runs also went out before our app asked for medium reasoning effort: the Gauntlet one most likely ran at DeepSeek's default, high, and our earlier article says the Forge pair ran at the provider's default. So "scored above its predecessor" is what our board shows, and part of that gap could be ours.

DeepSeek's model card for 1017 says it delivers "substantial improvements across reasoning, coding, agentic tasks, and visual benchmarks" and "reduces task execution time and output length". It doesn't restate the architecture. The September card describes a 552B-parameter mixture-of-experts backbone that activates 8B parameters per token during prefill and 16B during decode, and OpenRouter's listing gives 1017 the same numbers.

Both Forge 2.0 builds ran the same task fingerprint, the same starter app and the same three scoring sites. Build 1 made 262 tool calls in 28 minutes and scored 0.2422, while build 2 made 270 in 30 minutes and scored 0.6618, so the effort was nearly identical and the score was not. A Forge 2.0 score is the work score multiplied by a critical multiplier and a reliability multiplier.

In each of 19 groups of checks, which we call tiers, the worst check that falls short costs up to 10%, and the multiplier never drops below 0.25. Before penalties the work was close (0.8362 and 0.9693), but reliability wasn't (0.2896 and 0.6828). Build 1 fell short in 15 of the 19 tiers and build 2 in 9, and the blank modal below is behind several of build 1's losses.

Its worst tier was UI function, 0.0568 against 0.9399.

The pair popup on the Forge 2.0 board, captured 18 October 2026. Build 2's bill is a lower bound because OpenRouter's records and the session counters disagree by about 1,300 prompt tokens.

On one of the three scoring sites, build 1's sprint modal opened 12 times and painted nothing 12 times. No console errors, no failed requests, just an empty frame. Why? The scorer's observation file says. I counted 14 answers to its resolver. All of them looked like this.

{ "rateLimited": true, "retryAfter": 2237 } Enter fullscreen mode Exit fullscreen mode Here is build 1's page handling that answer, in static/shared/bridge.js.

export async function call ( key, payload ) { for (;;) { const res = await invoke ( key, payload ); if ( res && res. rateLimited ) { await wait ( Math. max ( 1, Number ( res. retryAfter ) || 1 ) * 1000 + 50 ); continue; Enter fullscreen mode Exit fullscreen mode It waits out the Retry-After, all 2,237 seconds of it (about 37 minutes), with nothing on screen. Nobody waits that long. Awkward detail: that loop came from the v1 starter app. Build 1 rewrote the file at 16:50 and kept the loop word for word. Build 2 called invoke directly. When the call failed, it painted a message.

const res = await invoke ( ' sprintLedger ' ); if ( res?. notStarted ) return renderNotStarted ( res. sprint ); if ( res?. error ) return renderMessage ( res. error ); Enter fullscreen mode Exit fullscreen mode So why did build 1 hit the rate limit at all? I have a reading, not a proof. Our task's rate model gives the app 2,400 points an hour, with background work allowed at most 1,680 of them (70%), and each REST call spends some.

Build 1 gave each scheduled reconcile run (the background job that re-syncs the ledger) its own allowance of up to 1,350 points, read once when the run started. When it set those numbers it wrote: "Worst = 1350+180+100 = 1630 < 1680." That sum assumes one reconcile per hour. Our scorer started two, 8.5 seconds apart.

They spent 787 and 732 points, the hour reached 2,400, and 25 requests made on behalf of people were refused. Even together, the background stayed under its 1,680 share. The people's traffic used the rest of the hour.

Build 2 re-read a shared counter before each request, and its two runs spent 1,010 and 175. I'm not done yet. Build 2's own hour reached 2,436 points and refused nobody, and I can't explain that yet.

Build 1 saw the wall in its own testing. At 16:52 it wrote "Quota exhausted!? Retry-After 938 s." and at 16:59 it decided other tenants had used the quota, reset the dev site and moved on. It never tested what a screen does when the resolver says "rate limited". Its closing line was "The app is complete and validated."

I don't want to make build 2 a hero, and I won't. It dropped the loop for its own reason (a budget of one call before first paint), not because it saw a quota problem coming. It also never got a 429 while it was scored. Its error path was never tested either. Some of the 0.42 gap is luck in where the stress fell. Build 1 also had a separate bug: it rewrote the board list to fetch one page ( src/views.js:26-28 ) where v1 had used the paged helper. The scoring site serves two boards per page, so build 1's widget showed two of four. Its own test passed, because the dev site returned all four boards in one page.

If we had stopped after build 1, the story would have been "1017 is worse than September". With only build 2, it would have looked about 0.2 better than its median.

NVIDIA's card describes a "Mamba-2–Transformer hybrid Mixture of Experts (MoE) with Multi-Token Prediction (MTP)", 120B total parameters with 12B active, and context up to 1M tokens. OpenRouter lists it at $0.10 in and $0.40 out per million tokens, across five hosts: four of them serve a 262,144-token window, one serves the full 1M.

Nemotron's Gauntlet run on the board, captured 18 October 2026. The cap comes from one check: the 3D view rendered as an empty canvas.

On the Gauntlet it was quick and stubborn in a good way. It wrote its first file 39 seconds in and put a 1,354-line backend down in one write. When goose's edit tool rejected all seven of its edit calls on their arguments, it switched to Python string replacement through the shell (27 edits) and kept going. It ran the browser self-test 19 times. 41 of 99 checks passed fully, and the score was capped at 0.599 because its 3D panel rendered as an empty canvas.

The self-test reported 50 rendered rows and no errors, and it wrote "Everything works with zero errors." Then it said it would check the screenshot to verify the towers. Python's imaging library wasn't installed on the machine, so the fallback it wrote printed the PNG header and the file size, 398,867 bytes, and the model moved on without looking at a pixel. The canvas was empty. DeepSeek 1017, on the same benchmark, opened seven of its own screenshots with goose's image tool, and its 3D view passed the visible-surface check. Nemotron is a vision model too. It had the same tool and reached for Python.

On Forge 2.0, build 1 went in three phases. For 8.6 minutes it read: the contract, the starter, the offline Forge toolkit, the manifest schema. Then for about 7 minutes it wrote, all 15 writes and 4 edits of the run. Then came 290 more tool calls and 11 minutes with no write, no lint and no build. Just reading. It asked for the contract 81 more times (39 of those calls returned it), the starter 70 times and the rate model 42 times. 68 of those 290 calls went to a read tool that doesn't exist.

You can see the turn to the second. At 931 seconds in, mid-build: "Now let me write the explain.js (Forge LLM), realtime.js, and the main index.js:". At 975 seconds, like a first day on the job: "Let me start by reading the contract and starter files to understand what's required." I winced at that. Much later it noticed its own work without knowing it was its own: "The manifest already lists v2 modules." The run ended on "Let me read the remaining source files." with no tool call after it and no summary.

Illustration, generated locally with ERNIE-Image. Nemotron's Forge build asked for the same contract 81 more times after its prompt was cut. The half-built app did not finish itself.

The token counts our harness logs per call show the prompt the provider received growing to 146,368 tokens and then, on the next call, 7,394. It never went above 67,997 again. Goose did not drop the history, and the session database kept all of it, every message from the first read to the dangling last sentence.

Goose's request log shows it sending the whole conversation on the last call, 824 messages and 5.5 million characters, with OpenRouter's middle-out transform switched on. OpenRouter's documentation says middle-out works "by removing or truncating messages from the middle of the prompt, until the prompt fits within the model's context window". It counts the room kept for the reply as well as the prompt, and it keeps the oldest and newest messages.

Nemotron behaved exactly as if that had happened. I can't prove it did.

Part of this is our setup. We ran Nemotron unpinned. Goose took its context limit, 1,048,576 tokens, from the model's listing, while DeepInfra, which served 225 of its 229 calls, accepts 262,144. Goose adds middle-out to all its OpenRouter requests. So goose never compacted, and something downstream trimmed instead. What I can't prove is why it fired at that call. The conversation was about 149k tokens, under DeepInfra's window. Goose sent no output limit, so I don't know how much room the host kept for the reply. The same model's Gauntlet prompt reached 165,998 tokens and was never cut. The second Forge build got through one call at 175,171 before its own cut at 436 seconds. After that cut, build 2 also wrote nothing for 289 calls, until I cancelled it.

The direct cause of 0.0321 is one edit made before the cut. It added getSprint to an import in src/views.js while the same file defines getSprint 63 lines lower.

// src/views.js:5 import { listScrumBoards, listActiveSprints, getSprint, isStarted } from './config '; // src/views.js:68 export async function getSprint ( sprintId, policy ) { Enter fullscreen mode Exit fullscreen mode All of the app's functions import that file, so all nine handlers failed to bundle with "The symbol "getSprint" has already been declared" and nothing on the backend ran under the scorer. Its manifest also pointed at three files it never wrote. Its last lint printed five errors, it fixed two, and it never ran lint again. Would it have caught that with its history intact? I don't know.

The first 15 minutes held good engineering. It used the offline manifest schema properly, found the right line ranges for each new module, declared all the v2 modules the contract asks for (that check scored 1.0), and wrote a CI webhook handler that does a correct HMAC check with a timing-safe compare. DeepSeek 1017, pinned to DeepSeek's own host, carried its whole history to 372,546 prompt tokens in build 2, more than twice what cut Nemotron's. Its bundles loaded in both builds. So I read our 0.0321 as one unpublished build that broke on a duplicate import and then lost its context, and I wouldn't judge Nemotron as a builder from it. Next time it runs pinned, with the window its host really has.

We also tried Nemotron locally, on the MacBook with more memory, through our MLX engine. In 4-bit the weights are 69.5 GB, the largest model I tried to run locally that day. That day it didn't build anything, because goose couldn't hold a conversation with it. The app's connection check said "no visible response". With tools, the model's calls came back as raw text, <tool_call><function=write>..., and the engine parsed zero of them.

Some background on what broke, for anyone who hasn't had to debug this. A model doesn't call tools. It writes text in a format it was trained on, and the inference server has to turn that text into a structured tool_calls field. That's the tool parser. A second piece, the reasoning parser, splits the model's thinking from its answer. vLLM's code shows what happens with no tool parser configured: the reply goes back as plain content with no tool calls in it. Even with the right parser, vLLM's documentation warns that unless strict tool constraints apply, "vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema".

Our engine picks parsers by looking at the model: a known alias, then a table of name patterns, then the checkpoint's own metadata and chat template. Nemotron fell through all three. Its chat template shows the tool format only as an example inside the instructions ("If you choose to call a function ONLY reply in the following format with NO suffix"), so the detector that looks for that format in the template missed it. No profile meant no reasoning parser either. Nemotron's closing </think> is a single token (id 13) but it isn't marked special in the tokenizer, and with no parser to claim it, our engine stripped it during streaming. The answer came out as <think>The user wants...OK, which goose read as no visible answer.

Our first idea was to rename the served model. No change. The fix, on our fork's branch lz/nemotron-h-metadata, maps the nemotron_h architecture to the nemotron tool parser and Qwen3-style reasoning, marks it hybrid, and marks it as mixture-of-experts when it routes experts. Two new tests fail before the change and pass after it. On the Mac Studio the 120B model then answered "OK" cleanly, parsed 3 of 3 write calls with and without reasoning effort, and split 1,030 to 2,142 characters of reasoning out of the answer. NVIDIA's card, for comparison, tells vLLM users to pass --tool-call-parser qwen3_coder --reasoning-parser nemotron_v3 by hand. A server left to guess this model's parsers can guess wrong.

Muse Glimmer, on the same engine the same day, needed nothing. Our engine already knew it by name: its table of name patterns maps muse-glimmer to the muse tool and reasoning parsers, and it passed a 3-of-3 tool smoke test. Nemotron had no entry, so it fell through to the template detector, which missed it.

Goose Swarm 3.0.119, released on 19 October, ships with an engine that has this mapping. A few hours before that release, the local Forge build of Nemotron ran overnight on the MacBook with more memory, on an interim engine with the same fix, and scored 0.0928. That is below our bar for a second build, so it stays unpublished.

Muse Glimmer 30B is a dense 29.6B model, "distilled from Muse Spark and purpose-built for autonomous agentic tasks on consumer hardware" per its card, Apache 2.0, with three sliding-window attention layers for every global one. I ran it in 8-bit on the 64 GB MacBook where the stock Qwen3.8-27B holds the local record.

It started well. It read the whole starter before editing anything, and in its first reasoning it caught a real ambiguity between two sections of our contract and argued both readings. Then, at 20:04, it appended an admin resolver to src/index.js:

182 const adminResolver = new Resolver (); 183 adminResolver. define ( ' admin-get ', adminGet ); 184 adminResolver. define ( ' admin-save ', adminSave ); 185 adminResolver. define ( ' admin-rotate-secret ', adminRotateSecret ); 186 export const adminResolver = adminResolver. getDefinitions (); Enter fullscreen mode Exit fullscreen mode Same name, declared twice. 44 lines higher, the starter shows the right pattern: a local called resolver, exported as uiResolver. Glimmer copied the layout and kept one name for both. An easy slip to make.

It never touched that file again. Two minutes later goose compacted its context (105k to 51k tokens), and almost all of the remaining 27 minutes went into the manifest: 11 more lint runs, mostly about the admin page's resource. Lint said "No issues found." after it commented the admin page out, failed once more, and passed on its last two runs, once it pointed the page at the resolver and deleted the admin UI resource. Then it ran npm run build, which builds the three front-end bundles only, and wrote its summary. The line I'd frame:

The app now bundles and loads with npm run lint clean, all v2 modules are declared in manifest.yml, and the admin resolver enforces ADMINISTER on the server.

I count three true claims in those four, and the one that matters is false. Lint was clean (the scorer agrees) because our lint reads only the manifest. "Bundles and loads" was never tested by anything it ran. Under the scorer, all 8 functions failed to bundle on line 186, including the v1 functions it hadn't changed, and 0 of 22 resolver calls returned a result. Our rule for local candidates is to run a second build only when the first beats the stock Qwen3.8-27B's 0.1852 median. 0.0325 didn't, so I cancelled build 2.

Illustration, generated locally with ERNIE-Image. Muse Glimmer's manifest passed lint on its last two runs while its backend could not be bundled. The certificate was accurate about the manifest.

A trap if you want to draw the moral "it should have run node --check": that would have passed. Our workspace has no "type": "module", and on Node 24 node --check passes a.js file that uses export without reporting anything, even a plain syntax error. I checked with control files. Run as a module, Node says "Identifier 'adminResolver' has already been declared". The check that catches it is the one the starter names, forge-dev invoke, which runs a function in the Forge runtime, or any esbuild bundle of the backend. It used neither.

My guess at the why: the compaction two minutes after the edit replaced the tool outputs that showed the file, the edit tool never reports whether the result parses, and the two checks it did run couldn't see a backend syntax error. Nemotron made the same mistake with getSprint in the cloud, from a very different architecture, and in both cases bundling the backend would have reported the duplicate name straight away. DeepSeek 1017's builds both scored 1.0 on lint and bundles.

Mistral Large 4 is a 1.05T-parameter mixture-of-experts model with 52B active, in Public Preview since 6 October, and on OpenRouter Mistral is its only provider. It wasn't its first try. A 9 October Gauntlet attempt had ended after 49 calls on OpenRouter's "Upstream idle timeout exceeded".

In the 18 October run it thought before it typed: one thinking call of about 38,000 words ran 18 minutes, with under 1% of its 12-word sequences repeated, so coherent planning and no loop. Its first source file arrived about 32 minutes in. Then the requests started failing.

At 19:40, after 30 answered calls of a 150-call budget, Mistral's endpoint answered "Internal server error" with a 503. Goose resent the request three times and got the same answer each time, and the run ended after 46 minutes and at least $1.18, with no verdict and nothing published. Whether one minute of failed sends should end the run of a Public Preview model with one host and no fallback is our call to rethink.

The next attempt failed too, and I count that one as ours. When a tool result carries an image, goose follows it with an image-only user message, and with parallel tool calls that message landed between two tool results. Mistral's API is strict about message order and answered "400 Unexpected role tool after role user", which ended the run after 97 of 150 calls.

We fixed it in Goose Swarm 3.0.119 (the image messages now go after the last tool result), and on 19 October the fourth attempt finished: 0.5462 in 144 minutes for $4.93. Before the critical-defect multiplier it stood at 0.9104. One check cost it 40% of the score, and that one is on us.

Our check of the amounts on screen compares them with the oldest 50 payments, Mistral's table opens on the newest, and all 24 amounts our probe read off it were right. Without that penalty a separate cap, on its 3D interaction, would still have held it near 0.70 by my arithmetic, and the board still shows 0.5462. I take that apart in a separate piece.

On Forge 2.0 its two builds scored 0.1086 (all 300 calls) and 0.3304, billed at least $5.40 (an incomplete record) and $5.55. That is more than seventeen times what DeepSeek 1017's pair cost, for a lower median.

At $0.62 a pair, I can run DeepSeek 1017 again tomorrow to see which build I usually get. The most expensive Forge 2.0 pairs on our board cost $99.66 (GLM 5.3 Prime), $72.04 (Qwen3.8 Max Prime), at least $53.35 (Grok 4.7) and $52.85 (Ember-1); Opus 5.5's cost $42.50. DeepSeek 1017's pair sits ninth by median, alone in its comparison group. Bills change the ranking you'd act on, as they did when two models at the same list price ran up agent bills 3.6x apart.

On the DeepSeek bills specifically, setup moved them more than the model did. Both Gauntlet runs went to DeepSeek's own endpoint at the same rate, so $0.21 against $0.25 is the same rate on fewer tokens: 25.3 million prompt and 213k output, against 26.5 million and 269k (run pages). The effort setting may explain part of that. The September Forge builds were routed mostly to other hosts, and the first of them pushed 101 million prompt tokens through in 99 minutes. Its cache reads cost between one and two cents per million on those hosts. Each 1017 build, pinned to DeepSeek, used about 54 million in under half an hour, and 99.6% of those were billed as cache reads at $0.003 per million.

Until Sunday, our Gauntlet runs had no price on the board (one exception, priced on 15 October), and most of our Forge runs showed only an "at least" figure. We had saved each run's generation ids. Our app never sent the bill when it published. Embarrassing, and entirely ours. So we went back to OpenRouter's generation records, the per-call receipts that carry the provider and the cost (the same records that showed us where Jev Router's calls went ), and rebuilt each run's bill from its own saved ids. 36 Gauntlet runs got a price for the first time: 17 complete and 19 marked "at least". 15 of the 17 became complete only after we asked OpenRouter again about calls that had come back "not found" while the run was live. 33 Forge bills improved, 31 of them now complete.

A run page's cost block, captured 18 October 2026. It says where the number came from and how many requests it covers.

49 runs stay at "Not recorded": 19 on Bedrock (we kept no per-call bill there), 13 on vendors' own APIs, other Google-direct runs, and local runs, which have no provider bill by design. For some of the vendor runs our logs hold list-price estimates (Kimi K3 at $59.57, DeepSeek V4 Pro at $46.89). Those are upper bounds and they aren't on the board.

Names were the other half. 94 rows showed a raw model id instead of a model name, 59 on Forge and 35 on the Gauntlet, and we renamed them. Rows titled by a random poster handle are now named by their model id under a site rule. On Sunday evening the DeepSeek 1017 pair itself went up as two separate, unpriced rows named by id, and I caught it from a screenshot. A small naming wrinkle survives, since the Forge board says "DeepSeek 4.1 Flash 1017" and the Gauntlet board "DeepSeek V4.1 Flash 1017".

Since Goose Swarm 3.0.119, released on 19 October, the app sends the bill when a run is published, and where a provider gives no bill it can send a clearly labelled estimate that the site recomputes and checks against the run's own token counts. Our own entries are verified server-side. The site side went live Sunday evening and the app side followed the next morning.

DeepSeek V4.1 Flash 1017, one Gauntlet run and one Forge pair: strong and cheap in our setup, and fragile to one inherited shortcut. Its two builds were 0.42 apart.

Nemotron 3.5 Super VL, one Gauntlet run and one unpublished Forge build: quick to write and quick to adapt when a tool broke, didn't check its own pixels, and didn't recover once its context was cut. Locally, it needed an engine fix before it could talk at all.

Muse Glimmer 30B, one local build: reads carefully, trusts lint more than it should. Mistral Large 4, one scored Gauntlet run and one Forge pair: a long, coherent planner whose Gauntlet score lost 40% to our own money check, which compared its table with the wrong rows, and slow and expensive on Forge.

Most leaderboards, our Gauntlet board included, rank models on one run. DeepSeek 1017 built the same app twice on Sunday, same task and same seeds, and scored 0.2422 and 0.6618. The clearest difference I found is a retry loop one build inherited from the starter and the other dropped. Before you trust a model with agentic work, run it twice, and when a long run stops writing, check whether the prompt your provider saw shrank before you blame the model.

Goose Swarm, the app we ran all of this through, isn't production-ready yet. We use it daily because we know what's inside it, and the engine fix and the publishing changes above only reached a release on 19 October. If you run agents on point releases, or on a model your server can't quite parse, which one surprised you, and did its second run agree with its first? Tell me in the comments under this post.

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.