Part 3 of the local LLM benchmarking series. Part 1 covers the harness, Part 2 the llama.cpp and speculative decoding runs. The interactive results page has the full dataset.

Reasoning models spend a lot of tokens thinking before they answer. Qwen3.6-27B routinely burns 2,000 thinking tokens on a grade-school math problem. On my M3 Max at ~18 t/s, that’s almost two minutes of watching <think> scroll by before the first word of the actual answer.

So when BottlecapAI released ThinkingCap-Qwen3.6-27B, a finetune that claims 45.8% fewer thinking tokens with accuracy within a point of the base model, I wanted it to be true. Which is exactly why I didn’t trust it.

Benchmark gaming is common enough that “trained to score well on the published eval set” is the default hypothesis rather than the cynical one. So I did two things:

1. Retested their claims independently: partly on benchmarks from their own tables (GSM8K, ARC-Challenge, MMLU-Pro), partly on one they didn’t publish numbers for (HumanEval).
2. Tested at 4-bit quantization instead of BF16. BottlecapAI only published BF16 results. But nobody runs a 27B dense model at BF16 on local hardware. And Qwen3.6-27B still has the best Artificial Analysis intelligence-index-to-parameter ratio of any open model, which makes the quant question the practically relevant one. heyneo’s case study benchmarked the base model across GGUF quants; I use their numbers as a second reference point below.

The short version: the token reduction claim checks out almost exactly. And at 4-bit, ThinkingCap beats the base model outright, by a margin that surprised me.

Setup

Everything ran locally on the M3 Max (64 GB), same machine as Parts 1 and 2.

• Models: ThinkingCap-Qwen3.6-27B-MLX-4bit vs. mlx-community/Qwen3.6-27B-4bit, ~15 GB each
• Inference: mlx_lm.server 0.31.3
• Sampling: temperature 1.0, top_p 0.95, matching BottlecapAI’s methodology so the token counts are comparable
• Samples: N=100 per benchmark (80 for HumanEval), seed 42, same prompts for both models
• Runtime: 15.4 hours of measured inference, sequential because both models don’t fit in RAM together. ThinkingCap finished its 380 samples in 5.5 hours; the base model needed 10.0 hours for the same prompts.

One methodological caveat up front: mlx_lm reports total completion tokens, not thinking tokens separately. I estimate the split via the character ratio between the reasoning field and the full generation. That’s an approximation (BottlecapAI counts exact <think> trace lengths), so treat my token numbers as comparable rather than identical.

Part 1: The token efficiency claim

Thinking enabled, both models, four benchmarks:

I measured 44.2% average reduction. BottlecapAI claims 45.8%. Different benchmarks, different precision, different token counting method, and the numbers still land within 1.6 points of each other. That includes HumanEval, which doesn’t appear in their published tables at all but shows the same ~37% reduction as their out-of-domain set. Whatever they trained, it generalizes past their eval suite. The benchmaxxing hypothesis is dead.

Part 2: The result they didn’t report

The accuracy column above deserves a second look. At BF16, BottlecapAI measured ThinkingCap at −0.8pp versus base: a small, honest regression, the expected price of shorter reasoning. At 4-bit, I measured +7.7pp in ThinkingCap’s favor, driven by GSM8K (+21pp) and MMLU-Pro (+13pp).

The base model held its ground on ARC and HumanEval; the collapse is specific to the long-reasoning benchmarks. GSM8K at 74% is far below the 93.3% BottlecapAI measured at BF16. ThinkingCap’s 95% at 4-bit, meanwhile, is within two points of its BF16 score.

My working hypothesis: long thinking traces compound quantization error. Base Qwen generates ~2,000-token reasoning chains where each step feeds the next; 4-bit noise gets more chances to derail the chain. ThinkingCap’s traces are half as long, so there’s less runway for drift. That would explain why the gap is largest exactly where the base model thinks longest.

Single seed, N=100, temperature 1.0: the individual deltas carry real variance (±3–5pp wouldn’t shock me). But +21pp on GSM8K is not noise, and the direction is consistent across both reasoning-heavy benchmarks.

Part 3: Quants, and where my numbers meet heyneo’s

For the second goal I turned thinking off (greedy decoding, temperature 0) and ran the three benchmarks from heyneo’s case study, which covered the base model at Q4_K_M, Q8_0, and BF16 in GGUF:

Three observations:

• HumanEval validates cleanly. My base model at 50.0% versus heyneo’s Q4_K_M at 50.6%: two independent harnesses, two different 4-bit formats, same answer. And heyneo’s ladder (50.6 → 62.8 → 67.7 going up to BF16) shows code generation is where 4-bit hurts most. If you’re doing serious code generation with this model without thinking mode, quantization is costing you real pass rate.
• HellaSwag barely moves across quants. 84 to 87 across every format, consistent with heyneo’s finding that commonsense reasoning tolerates quantization well.
• With thinking off, TC and base are near-identical. That’s what you’d want: the finetune changed how the model thinks, not what it knows. (On my extended set of IFEval, GSM8K 8-shot, and BigCodeBench Hard: same picture, except BBH CoT where TC leads by 12.6pp even without thinking.)

The BFCL discrepancy I can’t explain

My BFCL number is 87%. heyneo’s is 63%, for the same base model. A 24-point gap between two independent evaluations of the same checkpoint is a finding about the harnesses rather than the model.

BFCL is a leaderboard with multiple versions and categories rather than a single benchmark, and results depend heavily on prompting format, function-schema serialization, and which subset you sample. I ran prompting-based evaluation (model emits the call as text) on 100 samples; heyneo ran 400 samples on their harness, version unspecified. Both numbers are probably “correct” for what they measured. Neither is comparable without knowing the exact category mix. That’s a good reminder of why single-number benchmark comparisons between different harnesses should be read with suspicion. If someone from heyneo reads this: I’d genuinely like to reconcile these.

What this means in practice

Raw throughput is identical between the two models: same architecture, same quant, same ~18 t/s on my machine (see /bench for the full speed dataset). The speedup comes entirely from generating fewer tokens: 44% fewer thinking tokens means ~44% less wall-clock time to an answer. The benchmark run itself is the cleanest demonstration: the base model took 10.0 hours to work through the 380 prompts that ThinkingCap finished in 5.5, a 1.8× difference measured over an entire day of inference. The two-minute GSM8K wait becomes about 50 seconds, with a better answer at 4-bit.

My recommendation after 15 hours of benchmarks:

• Running Qwen3.6-27B locally at 4-bit? Switch to ThinkingCap. It’s faster to answer, more accurate on reasoning tasks at this precision, and identical everywhere else. I don’t see a downside on my data.
• Running at BF16 on datacenter hardware? The base model keeps a marginal accuracy edge (−0.8pp for TC per BottlecapAI’s own numbers). Whether that’s worth double the thinking latency is your call.
• Code generation as the main workload? Consider Q8_0 or higher for the base weights regardless of which variant you pick; the HumanEval quant ladder is steep.

Caveats

Single seed, N=100 per benchmark, token counts estimated via character ratio, 4-bit only. BottlecapAI’s numbers come from full test sets with 5 seeds and confidence intervals; mine are a spot check rather than a replication study. The headline findings (token reduction ~44%, base model’s GSM8K collapse at 4-bit) are large enough to survive that uncertainty. The small deltas (ARC −2pp, HumanEval −1.2pp) are not.

Raw results and the eval script live in the benchmark repo; the fact sheet with all tables is in results/quality/thinking_tokens/.