In Part 1 and Part 2, I reported some peculiar behavior for quantized Qwen3.5-2B models. In Part 3, I reported results for a larger variant, Qwen3.5-35B-A3B, with thinking enabled.

In this post, we'll take a look at performance with thinking enabled vs. disabled.

Sweep Summary

Analysis

As expected, the thinking runs took longer to complete than the non-thinking runs. But unexpectedly, non-thinking runs outperformed thinking runs across the board. This is counterintuitive, since we would expect that thinking would improve performance. That is how it's supposed to work!

One concern I had with this experiment was whether timeouts were affecting the results, that is, if given enough time, the thinking runs would eventually outperform the non-thinking runs. I actually ended up running this several times, eventually with quite long timeouts. Most of the remaining timeouts were due to thinking loops rather than legitimate reasoning timeouts. So endless loops are a concern with this particular model on thinking runs. But we see more differences in performance --- exactly 100 more problems were resolved by nonthinking-BF16 than thinking-BF16 --- that can't be explained by the 15 observed timeouts in the thinking-BF16 run.

I asked AI to analyze the results and it found two problems.

Problem 1: Tool-call parsing

In thinking mode, llama.cpp is experiencing a malformed tool-call parse crash that is causing a large number of failures. These failures cause openhands to exit.

I can trigger this behavior using this script:

=== firing the same clean real prompt 10 times ===
POST http://172.17.0.2:8085/v1/chat/completions x 10

  probe 1: EXTRACTED (str_replace_editor)
  probe 2: EXTRACTED (str_replace_editor)
  probe 3: EMPTY (finish_reason=stop, no content, no reasoning)
  probe 4: EMPTY (finish_reason=stop, no content, no reasoning)
  probe 5: CRASH (500: Failed to parse input at pos 49: <tool_call>
<function=str_replace_editor>
<parameter=command>
view
)
  probe 6: EMPTY (finish_reason=stop, no content, no reasoning)
  probe 7: EMPTY (finish_reason=stop, no content, no reasoning)
  probe 8: CRASH (500: Failed to parse input at pos 49: <tool_call>
<function=str_replace_editor>
<parameter=command>
view
)
  probe 9: EMPTY (finish_reason=stop, no content, no reasoning)
  probe 10: EMPTY (finish_reason=stop, no content, no reasoning)

  -> extracted: 2/10, stuck: 0/10, crash: 2/10, empty: 6/10, other: 0/10

So out of 10 queries, we get the failure to parse twice, and then an empty response 6 times. Something is clearly going wrong!

With the help of AI, I was able to attribute many of the problems to a grammar problem. The model often emits a redundant <thinking> tag, which causes llama.cpp's parser to fail. As you can see in this AI-generated table, this happens very frequently:

¹ Timeout/ExitCode/Setup/Reward/Verifier/NetworkConnectionError — the only category Harbor's own exit-code-based detection actually catches. ² llama.cpp's PEG parser crash (Failed to parse input at pos N), trial ended fatally right there. Zero on vLLM. ³ OpenHands' AgentStuckInLoopError killed the attempt. ⁴ Other unrelated fatal errors (mostly the chardet/action-execution-server bug on vLLM). ⁵ Attempts that neither resolved, hit a Harbor exception, nor showed any of the fatal patterns checked — genuinely completed but produced a wrong patch, or a crash pattern not yet characterized.

Problem 2: Harbor error handling

The second problem is that Harbor does not detect the error. And it seems that this is because openhands returns exit code 0.

What a mess!

Concluding Thoughts

We can't detect the redundant <thinking> tag problem in vllm, but it could still be happening (update: it is happening). vllm has a different tool parsing mechanism than llama.cpp; it might be more forgiving (update: it is). For whatever reason, vllm-thinking is still underperforming vllm-nonthinking.

According to Qwen's blog post, Qwen3.5-35B-A3B scores 70.0% on SWE-bench Verified. There is a small footnote too:

SWE-Bench Series: Internal agent scaffold (bash + file-edit tools); temp=1.0, top_p=0.95, 200K context window. We correct some problematic tasks in the public set of SWE-bench Pro and evaluate all baselines on the refined benchmark.

I'm not going to comment on the "correct some problematic tasks" part.

That temperature of 1.0 and top_p of 0.95 are recommended by Qwen for both thinking "general tasks" (as opposed to "precise coding tasks") and non-thinking "reasoning tasks" (as opposed to "general tasks"). Unfortunately, as you can see here, for these results I used "precise coding tasks" and "general tasks", which both use different temperature and top_p settings. If you are wondering why SWE-Bench Verified is not considered a "precise coding task"... well, so am I. Ask Qwen! IMHO, models should have a single set of recommended parameters for all tasks, and the model should be able to handle the task type automatically.

Next Steps

1. I am going to try to confirm whether the redundant <thinking> tag is generated by vllm-thinking. If it is, then it suggests that the model is buggy. Which would be weird --- the model defaults to thinking. (Update: I confirmed that vllm-thinking does generate the redundant <thinking> tag, so its parser must be more forgiving than llama.cpp's parser.)

2. I am going to rerun the experiments with all four parameter settings that Qwen recommends, since I apparently chose different ones than Qwen did when they tested SWE-Bench Verified.

3. I'm will test the dense model Qwen3.5-27B.

4. I am also going to investigate agents other than openhands. I originally selected openhands because it is the first agent I was able to get to work with Harbor. But at the time, Harbor was only compatible with an older branch of openhands. Hopefully this has changed, or a more reasonable harness like opencode will work now.