Qwen3.8-Flash-Next vs Claude Opus 5.5: Six way benchmark on a real Rust feature

Posted on Oct 20 • Originally published at deepu.tech

I recently posted on my blog and Reddit about my experience pitting Qwen 3.8 Flash-Next against Claude Opus in some tasks and some people on Reddit pointed out that it wasn't a fair comparison since I used different harnesses and thinking effort levels. That got me curious and so I did a more controlled comparison to see how they truly stack up against each other. It's still a non-deterministic comparison, but it provides a clearer picture of how they measure up against each other.

You can read the previous comparison blog below and on Reddit.

Add Comment 15 min read TL;DR All 12 runs finished the task, and all 12 PRs passed the end-to-end test. Flash-Next took 1.1x to 2.1x as long as Opus 5.5 at the same effort. Average score, out of 8: Claude Code + Opus 5.5 - 6.59 (82%) Claude Code + Flash-Next - 5.44 (68%) Pi + Flash-Next - 5.25 (66%) With Flash-Next, Claude Code and Pi scored the same on the review questions.

CI passed on 4 of 4 Claude Code + Flash-Next PRs and 1 of 4 Pi PRs. Claude Code was also faster in the cleaner set, though Flash-Next had to answer its auto mode tool checks too. Opus xhigh scored 6.94 against 6.25 for medium, at 2.7x the cost.

Flash-Next medium and xhigh scored almost the same. At API prices, a set 2 Opus run cost $6.52 (medium) or $17.78 (xhigh), and a Flash-Next run $0.32 to $0.67. Combinations compared I wanted to compare both harness differences and thinking effort differences so I decided to set up a six way comparison with the help of Claude Code + Opus 5.5.

Note: This is Qwen Focused so I'm not gonna burn money testing Pi + Opus combinations.

Each combination ran twice: once where the agent measured save and restore at 100k tokens itself (set 1), and once with those numbers given in the prompt (set 2). Total 12 runs.

Opus 5.5 is a frontier cloud model with more than a trillion parameters, and Anthropic doesn't publish its actual size. Qwen3.8-Flash-Next is a mixture of experts model with 125B parameters and 6B active per token and a 51B n-gram embedding, and it runs on my laptop at ~Q4 quantization.

I picked a real feature from the LlamaStash roadmap. LlamaStash unloads a model when it needs room for another one or when it has been idle too long, and the next request then has to reprocess the whole prompt. llama-server can save a slot's KV cache to disk and load it back, so the task was to save the cache before an unload and restore it when the model comes back.

The agent had to measure save and restore at 100k tokens first (a spike), design the caps, cleanup and cache matching, prove it works end to end, add tests and docs, and open a PR.

The full prompt for the first set You are in a fresh clone of llamastash at commit <SHA>, on branch <BRANCH>. Do all the work here. Task (TODO.md, R14 batch 4): "Keep a model's prompt cache across an unload.

When make-room or the idle sweep stops a llama.cpp launch mid-conversation, the next request reprocesses the whole prompt. llama-server can write a slot's KV cache to disk and read it back: --slot-save-path PATH plus POST /slots/{id_slot}?action=save|restore (present on local build 11390 and in upstream tools/server/README.md, checked 2026-10-05). Nothing in src/ uses it. Spike first: measure save time, restore time and file size at a large context (100k tokens) against plain reprocessing.

Then decide when to save (before an eviction only), where the files live, how they are capped and cleaned up, and how a restored slot is matched to the returning conversation." Rules - Work from scratch. Do not look at any other branch, pull request, clone or worktree. - Do not ask me anything and do not stop to check in. If something is unclear, pick the reasonable option, note it for the PR, and continue until done. - Do not use child agents.

Do all the work in the main session. Environment - llama-server: <LLAMA_SERVER> - Test model: <TEST_MODEL> - Run llama-server on CPU only, with 8 threads, and one server at a time. - Do not use /tmp for anything: it is a RAM disk. Put temporary files, slot save files and daemon directories under <SCRATCH>. - Any llamastash daemon you start gets its own LLAMASTASH_STATE_DIR, LLAMASTASH_CONFIG_DIR and LLAMASTASH_CACHE_DIR and non-default ports.

Do not send test requests to the model endpoint you are running on. - While coding, run narrow tests only: cargo nextest run --features test-fixtures --lib <filter>, or --test <file>. The full suite runs once, at the end, in the pre-push hook. Done means 1.

Spike numbers at 100k tokens are in a doc under docs/spikes/: reprocess time, save time, restore time, file size. 2. End-to-end proof on an isolated daemon with the test model: send a prompt, let the idle sweep or make-room stop the launch, send the follow-up, and show that the prompt came from the restored cache (cached token count and time, before and after).

3. Unit tests cover the new code. Docs, config example, CHANGELOG and TODO.md are updated as AGENTS.md requires.

4. The pre-push hook passes. If it fails on tests your change does not touch, run it once more; if it then passes, note the flake.

Do not use --no-verify. 5. A PR against main is open, titled <BRANCH>, with these sections: Spike numbers, Design decisions (when to save, where files live, cap and cleanup, slot matching), End-to-end proof, Tests, Known limits.

Then stop. Do not wait for CI or poll it; I check CI myself. End with a summary of at most 10 lines.

Enter fullscreen mode Exit fullscreen mode

An orchestrator session (Claude Code + Opus 5.5) started each run and collected the numbers. Every run got a fresh clone of LlamaStash at the same commit, the same prompt, a 3 hour limit and one "continue" if it stopped early. Claude Code ran in auto mode, which sends its tool checks to the same model, and Pi ran with its guard off. The PRs are in a public scratch repo, deepu105/llamastash-b4r2.

The agents, builds and tests ran on my Dell XPS 15. Flash-Next was served from my ASUS ROG Flow Z13 (Strix Halo, 128GB) by LlamaStash via Halogen, over the LAN.

In the first set, each agent ran the 100k-token spike itself on the XPS CPU, which took 48 to 53 minutes per run. This was a mistake, it should have used the GPU, and by the time I noticed all six runs finished and since quality won't depend on the spike I decided not to redo them. In the second set, the spike numbers were in the prompt and llama-server ran on the GPU.

Versions Tool Version Claude Code 2.1.296 Pi 1.1.0 LlamaStash (on the Z13) 0.6.1 Halogen 0.17.4 Flash-Next Halogen checkpoint qwen38-flash-next-v2.hgn from peonist-ai/halogen-qwen3.8-flash-next, 4.16 bits per weight on average llama-server for the agents build 11525, CPU build (first set), CUDA build (second set) Test model Llama-3.2-1B-Instruct-Q4_K_M.gguf

Each PR got seven review questions and a CI check, for a maximum score of 8:

Each answer gets the grade of its worst problem:

For CI, a failing test that the PR added for working code gets 0.75. That includes two tests that hang on Windows, since the hang could be in the test rather than the code.

Four scorings went into this. Pi + Flash-Next and Claude Code + Opus 5.5 scored the PRs first, with an older version of these questions. A second Claude Code + Opus 5.5 session checked every place they disagreed in the code and graded each problem by severity. Then Kimi K3 (in GitHub Copilot CLI) scored all 12 PRs on its own, without knowing which run made which PR. The final score is the average of these last two, except where Kimi missed something that was checked in the code. There the Opus grade is used.

Yes. All 12 runs opened a PR, and all 12 PRs pass the end-to-end test. After LlamaStash unloads the model, the follow-up prompt gets 15,000 tokens from the restored cache and takes about 1.4 s, against 5.2 to 6.1 s without the feature. One Flash-Next run needed the "continue".

Set 2 times, where no run had to measure the spike:

In set 1 the gap was 1.9x to 3.5x, even after taking out the time spent waiting for the spike, because some Flash-Next runs built and tested on the same CPU the spike was using. Halogen served Flash-Next at 51 to 56 tokens/s in every run.

Opus wrote better code but not by a huge margin and every PR works.

All 12 PRs, by set Set 1 (the agent ran the spike itself)

Set 1 times have the time spent only waiting for the spike taken out. The minutes taken out are in brackets.

So Opus was ahead on the parts that change what a user gets: cache matching and eviction safety. Flash-Next did better only in some PRs, on CI, docs, tests and scope.

This was surprising for me, since Pi is my daily driver. Both scored the same on the review questions, and Claude Code was ahead on CI.

Claude Code also sent its auto mode tool checks to the same Flash-Next server, 22 to 79 per run, which took 10 to 14% of the run time. Pi sent none. Claude Code + Flash-Next was still faster than Pi in both set 2 pairs (by 6% and 22%). In set 1, Pi was faster in both (by 18% and 3%).

Three of the 4 Pi PRs fail the Windows test jobs, and no pre-push hook covers Windows. With one run per combination per set, I wouldn't read too much into it.

For Opus, xhigh scored higher in both sets (6.63 against 6.0, and 7.25 against 6.5), and only the xhigh PRs block two evictions at once. But it cost 2.6x to 2.7x as much and took 43.5 minutes against 23.4 in set 2.

For Flash-Next, medium and xhigh scored almost the same (5.28 and 5.41), and xhigh was slower in all 4 pairs. I have been running Flash-Next at xhigh, and on this task it didn't help.

Flash-Next, Opus and Kimi K3 all scored these PRs, so they can be compared as reviewers too:

In the previous post, I said Flash-Next's reviews found more issues than Opus's. On this task it was the other way around.

With the harness and effort kept the same, Opus 5.5 wrote better code than Flash-Next on this task and finished sooner. This was expected of course from a frontier model. That's different from my previous post, where I liked Flash-Next's PR better than the one from Opus at medium effort.

But the real takeaway here is that Flash-Next was right behind in scores and time and the gap wasn't that wide. Opus is a frontier model on Anthropic's servers, and Flash-Next has 6B active parameters and runs on my laptop. It finished every run, every PR works, it took 1.1x to 2.1x as long, and it cost me only electricity. What it missed (the save race, cache matching, integration tests, docs) is what a review catches, and that is how I already use it: an orchestrator plans and reviews, and Flash-Next does the implementation.

Two things I'll try in my work. I'll run Flash-Next at medium instead of xhigh and see how it goes, and I'll give Claude Code + Flash-Next a proper try instead of using only Pi and see how it goes.

All 12 PRs are in deepu105/llamastash-b4r2 if you want to read the code yourself.

If you like this article, please leave a like or a comment.

You can follow me on Bluesky, Mastodon, and LinkedIn.