Edward J. SchwartzComputer Security Researcher3 min. read

In Part 1 and Part 2, I reported some peculiar behavior for quantized Qwen3.5-2B models. In Part 3 I moved to Qwen3.5-35B-A3B, and in Part 4 I found that turning off thinking made the model better, which is surprising.

I also realized that I didn't use the same sampling presets that Qwen did. In this post, I re-ran the sweep with all four sampling presets: thinking-general (1.0/0.95), thinking-coding (0.6/0.95), nonthinking-general (0.7/0.8), and nonthinking-reasoning (1.0/0.95). Qwen's footnote parameters correspond to thinking-general and nonthinking-reasoning --- the two I hadn't used.

Sweep Summary

Run	Resolved	%	PPL	KL	Runtime	Exceptions
nonthinking-general-BF16	272/500	54.4%	6.62	-0.0000	1114m 18s	Timeout(2), ExitCode(22), Reward(4), Verifier(1)
nonthinking-general-Q5_K_M	250/500	50.0%	6.62	0.0083	1027m 24s	Setup(1), Timeout(4), NetworkConnectionError(1), ExitCode(44), Reward(2), Verifier(1)
nonthinking-general-Q8_0	275/500	55.0%	6.61	0.0068	976m 38s	Timeout(3), ExitCode(20), Reward(2), Verifier(2)
nonthinking-general-vllm	269/500	53.8%	—	—	928m 41s	Timeout(10), ExitCode(30), Reward(2), Verifier(1)
nonthinking-reasoning-BF16	264/500	52.8%	6.62	-0.0000	1114m 15s	Timeout(3), ExitCode(19), Reward(5)
nonthinking-reasoning-Q5_K_M	264/500	52.8%	6.62	0.0083	943m 24s	Timeout(2), ExitCode(19), Reward(5), Verifier(1)
nonthinking-reasoning-Q8_0	269/500	53.8%	6.61	0.0068	950m 9s	Timeout(1), ExitCode(21), Reward(3), Verifier(2)
nonthinking-reasoning-vllm	276/500	55.2%	—	—	869m 14s	Timeout(13), ExitCode(19), Reward(1), Verifier(1)
thinking-coding-BF16	183/500	36.6%	6.62	-0.0000	1428m 38s	Timeout(23), ExitCode(17), Reward(2), Verifier(1)
thinking-coding-Q5_K_M	198/500	39.6%	6.62	0.0083	1040m 1s	Timeout(12), ExitCode(24), Reward(4), Verifier(1)
thinking-coding-Q8_0	196/500	39.2%	6.61	0.0068	971m 55s	Timeout(10), ExitCode(20), Reward(2), Verifier(1)
thinking-coding-vllm	254/500	50.8%	—	—	1619m 34s	Timeout(40), ExitCode(21), Reward(3), Verifier(1)
thinking-general-BF16	187/500	37.4%	6.62	—	1049m 8s	Timeout(5), NetworkConnectionError(1), ExitCode(20), Reward(4), Verifier(1)
thinking-general-Q5_K_M	199/500	39.8%	6.62	0.0083	573m 14s	ExitCode(19), Reward(2), Verifier(1)
thinking-general-Q8_0	211/500	42.2%	6.61	0.0068	711m 51s	Timeout(2), ExitCode(19), Reward(4), Verifier(1)
thinking-general-vllm	243/500	48.6%	—	—	1694m 29s	Timeout(47), ExitCode(20), Verifier(1)

Finding 1: Sampling parameters are not the explanation

Summing each preset across all four backends:

Preset	Resolved	%
nonthinking-reasoning	1073/2000	53.6%
nonthinking-general	1066/2000	53.3%
thinking-general	840/2000	42.0%
thinking-coding	831/2000	41.5%

Non-thinking runs consistently perform significantly better than thinking runs.

In contrast, the sampling parameters don't seem to matter very much.

So the results from Part 4 do not seem to be a mistake caused by using the wrong sampling parameters.

Finding 2: The thinking penalty is worse on llama.cpp

Averaging each backend's two non-thinking runs against its two thinking runs:

Backend	Non-thinking avg	Thinking avg	Penalty (pp)
llama.cpp BF16	268.0	185.0	16.6
llama.cpp Q8_0	272.0	203.5	13.7
llama.cpp Q5_K_M	257.0	198.5	11.7
vllm	272.5	248.5	4.8

I would expect:

The non-quantized (vllm) run should out-perform the other runs
llama.cpp's BF16 run should be fairly close to vllm
More quantization should lead to lower accuracy, though there is evidence this is not always true

For non-thinking runs, this is spot on, considering a small amount of noise. But thinking runs are different: vllm does significantly better. This is probably because of the llama.cpp parsing bug discovered in Part 4.

Concluding Thoughts

Despite being significantly more expensive, thinking mode seems to be worse for coding, which still surprises me. This clearly isn't true for frontier models.

Sampling parameters don't seem to be that critical. Whether thinking is enabled is much more impactful.

The best run here is nonthinking-reasoning-vllm at 55.2%, compared to Qwen's claimed 70.0%. It's still unclear what the difference is.