Benchmarking Quantized LLMs for Local Coding Agents Part 6: One Mystery Solved
In Part 1 and Part 2, I reported some peculiar behavior for quantized Qwen3.5-2B models. In Part 3 I moved to Qwen3.5-35B-A3B. Part 4 found that turning thinking off made the model better, and also turned up a llama.cpp tool-call parsing bug that was killing thinking runs outright. In Part 5 I ruled out sampling parameters as the explanation, and noticed that the thinking penalty was much worse on llama.cpp than on vllm. I blamed the parsing bug, but I didn't actually verify it.
This post has two parts. First, I upgraded llama.cpp and re-ran, which fixed the llama.cpp thinking penalty. Second, I tried to sweep across agents, which mostly didn't work.
Part 1: Upgrading llama.cpp
I re-ran the two presets that Qwen recommends, thinking-general and
nonthinking-reasoning, on a newer llama.cpp build. I also added a
Version column to the sweep summary, for reasons that are about to become
obvious.
| Run | Resolved | % | PPL | KL | Runtime | Version | Exceptions |
|---|---|---|---|---|---|---|---|
| thinking-general-BF16 | 254/500 | 50.8% | 6.62 | — | 1011m 32s | 0.1.2-dev (build 10540, commit 07822bddf) | Timeout(3), ExitCode(15), Reward(6), Verifier(1) |
| thinking-general-vllm | 250/500 | 50.0% | — | — | 1649m 0s | 0.20.2 | Timeout(43), ExitCode(26), Reward(2), Verifier(1) |
| nonthinking-reasoning-BF16 | 277/500 | 55.4% | 6.62 | 0.0000 | 1045m 22s | 0.1.2-dev (build 10540, commit 07822bddf) | Timeout(2), NetworkConnectionError(1), ExitCode(22), Reward(3), Verifier(1) |
| nonthinking-reasoning-vllm | 288/500 | 57.6% | — | — | 808m 25s | 0.20.2 | Timeout(9), NetworkConnectionError(1), ExitCode(22), Reward(3) |
The mystery is solved
Here is how these runs compare to the same four runs from Part 5:
| Run | Part 5 | Part 6 | Δ |
|---|---|---|---|
| thinking-general-BF16 | 187/500 (37.4%) | 254/500 (50.8%) | +13.4pp |
| thinking-general-vllm | 243/500 (48.6%) | 250/500 (50.0%) | +1.4pp |
| nonthinking-reasoning-BF16 | 264/500 (52.8%) | 277/500 (55.4%) | +2.6pp |
| nonthinking-reasoning-vllm | 276/500 (55.2%) | 288/500 (57.6%) | +2.4pp |
Only one number really moved: llama.cpp with thinking enabled, which gained 67 resolved instances. The other three deltas are small enough that I'd attribute them to ordinary run-to-run variation. So upgrading llama.cpp fixed the thinking parsing bug.
What this does to Part 5's findings
Part 5's Finding 2 was "the thinking penalty is worse on llama.cpp." With the newer build, that finding is gone:
| Backend | Non-thinking | Thinking | Penalty (pp) |
|---|---|---|---|
| llama.cpp BF16 | 277 (55.4%) | 254 (50.8%) | 4.6 |
| vllm | 288 (57.6%) | 250 (50.0%) | 7.6 |
Part 5's Finding 1, unfortunately, is untouched. Thinking still resolves fewer instances than non-thinking on both backends. That mystery is still open; the one I solved here is why the two backends disagreed about how much worse it was.
Part 2: Trying to sweep agents
Since Part
4
I've wanted to test agents other than openhands, which I originally picked
simply because it was the first one I could get working with
Harbor. So I ran several agents Harbor offers
against the same configuration: llama.cpp build 10540, BF16 GGUF, and the
nonthinking-reasoning preset.
| Run | Resolved | % | PPL | KL | Runtime | Version | Exceptions |
|---|---|---|---|---|---|---|---|
| openhands | 274/500 | 54.8% | 6.62 | — | 1096m 56s | 0.1.2-dev (build 10540, commit 07822bddf) | Timeout(4), ExitCode(19), Reward(1), Verifier(2) |
| opencode | 1/500 | 0.2% | 6.62 | 0.0000 | 199m 17s | 0.1.2-dev (build 10540, commit 07822bddf) | ExitCode(498) |
| mini-swe-agent | 308/500 | 61.6% | 6.62 | 0.0000 | 1228m 21s | 0.1.2-dev (build 10540, commit 07822bddf) | ExitCode(4), Reward(3) |
| hermes | 0/500 | 0.0% | 6.62 | 0.0000 | 197m 9s | 0.1.2-dev (build 10540, commit 07822bddf) | ApiRateLimitError(5), NetworkConnectionError(8), ExitCode(487) |
| pi | 0/500 | 0.0% | 6.62 | 0.0000 | 49m 6s | 0.1.2-dev (build 10540, commit 07822bddf) | ValueError(500) |
| claude-code | 1/500 | 0.2% | 6.62 | 0.0000 | 174m 47s | 0.1.2-dev (build 10540, commit 07822bddf) | AgentAuthenticationError(500) |
| codex | 1/500 | 0.2% | 6.62 | 0.0000 | 203m 47s | 0.1.2-dev (build 10540, commit 07822bddf) | ExitCode(500) |
Five of the seven agents failed on essentially every instance. The majority of these seem to be from Harbor's agent code bitrotting against the latest versions of those agents.
The agent matters more than anything else I've tested
The two agents that did work are interesting. mini-swe-agent resolved 308/500 (61.6%) against openhands' 274/500 (54.8%).
That is a 6.8pp swing from changing nothing but the scaffold, and it is the largest effect of any single knob I've tested in this entire series. For comparison, the difference between BF16, Q8_0, and Q5_K_M is usually a couple of points, and Part 5 showed that sampling presets barely register at all. 61.6% is also the best result I've gotten across all six posts. The previous best was the 57.6% from the vllm run above.
This is worth holding next to Qwen's claimed 70.0% on SWE-bench Verified. Their footnote says they used an "internal agent scaffold (bash + file-edit tools)," which is a much closer fit to mini-swe-agent's minimal design than of openhands.
Concluding Thoughts
The llama.cpp/vllm discrepancy from Part 5 was a llama.cpp bug, and upgrading fixes it. The thinking penalty itself is still there on both backends and I still don't know why.
Next Steps
I've now spent five posts on Qwen3.5-35B-A3B. It's time to point the harness at other models:
- Qwen3.5-27B, the dense model I've been meaning to test since Part 4.
- Newer Qwen3.6 and Qwen3.8 variants. Qwen3.6-35B-A3B is an obvious one. Qwen3.8 doesn't have any similarly sized MoE models, so Qwen3.8-27B is probably the next best option.
- Gemma 4. Part 1 opened with quantized Gemma 4 being completely unusable for coding agents, which is what motivated this whole project. It seems only fair to give it a proper measurement.
Powered with by Gatsby 5.0