Situation
With Qwen 3.8 27B launching next week, the question is simple: what’s the best way to run a 27B dense model on a single Intel Arc Pro B70 32GB?
Not “can it run” — we know it can. The real questions are:
- Which quant fits, and at what context?
- Does Q5_K_M work at 200K, or are we stuck with Q4?
- Does Q6_K fit at all?
- How much does MTP-4 speculative decoding actually help?
- What KV cache config gives near-lossless quality without wasting VRAM?
I tested all three quants (Q4_K_M, Q5_K_M, Q6_K) at every context length from 128K to 512K, measured the exact VRAM boundary for each, and benchmarked with llama-bench on build b10222. This is the full recipe.
Hardware reference: Intel Arc Pro B70 on Amazon.
The VRAM problem (and why dense 27B is harder than MoE 35B)
The B70 has 32,656 MiB visible VRAM. A 27B dense model eats most of it in weights alone:
| Quant | Weights | VRAM left for KV + buffers |
|---|---|---|
| Q4_K_M | 15.6 GB | ~17 GB |
| Q5_K_M | 18.2 GB | ~14.5 GB |
| Q6_K | 20.9 GB | ~11.8 GB |
The remaining VRAM must hold the KV cache (which scales with context length) plus ~2 GB of fixed compute buffers (SYCL JIT, attention workspace, etc.).
Here’s the non-obvious insight: KV cache scales with the attention layer size, not total parameter count. Dense 27B runs full-size attention on all 27B params, consuming ~6,960 MiB KV per 128K context. MoE 35B-A3B has tiny attention (only 3B active), consuming just ~1,847 MiB per 128K — 3.8x less.
This means a 25 GB MoE model reaches 512K context, while a 16 GB dense model can’t pass 256K. The bigger model goes further because its attention is smaller.
KV cache: q8_0 K + q4_1 V (near-lossless, ~50% VRAM savings)
The fleet standard is asymmetric KV quantization:
--cache-type-k q8_0 --cache-type-v q4_1 --flash-attn onWhy asymmetric? In attention (softmax(Q·Kᵀ / √d) · V), the K matrix determines which tokens to attend to (routing), while V is averaged across attended tokens. K is sensitive to quantization noise; V tolerates it. So: spend bits on K (q8_0), save bits on V (q4_1).
Measured KL-divergence (from llama.cpp discussion #23470, cross-referenced with my own KL-divergence analysis):
| K cache | V cache | KL-divergence | Same top-p | Verdict |
|---|---|---|---|---|
| q8_0 | q4_1 | ~0.003 | ~97% | Fleet standard |
| q8_0 | q8_0 | ~0.003 | ~97% | Conservative (more VRAM) |
| q5_0 | q4_1 | ~0.006-0.008 | ~93% | Acceptable trade-off |
| q4_0 | any | ~5.5 | ~11.6% | Catastrophic — never use |
The q4_0 K cliff is not gradual — it’s catastrophic (KL-divergence ~5.5, only 11.6% of top-p tokens match). The model output is destroyed. This is why we use q8_0 for K: it’s the safest option that still saves ~50% VRAM vs FP16.
Context ceilings per quant (measured)
I loaded each quant at progressively higher context until VRAM ran out. These are measured values from llama-bench on build b10222, not estimates:
| Quant | 128K | 200K | 256K | 512K |
|---|---|---|---|---|
| Q4_K_M | 8.1 GB free ✅ | 4.2 GB free ✅ | 0.9 GB free ⚠️ | OVER ❌ |
| Q5_K_M | 3.3 GB free ✅ | 2.7 GB free ✅ | OVER ❌ | OVER ❌ |
| Q6_K | 0.7 GB free ⚠️ | OVER ❌ | OVER ❌ | OVER ❌ |
(Values = MiB free after model + KV + compute buffers. ✅ >1 GB, ⚠️ <1 GB tight.)
Key finding: Q5_K_M at 200K works with 2.7 GB headroom — better than linear VRAM models predicted. This means Q5 quality at 200K context is a viable production config, not just Q4.
Q6_K at 128K is the absolute quality ceiling: it fits with only 687 MiB free. Pure text only — no room for a vision projector or extra context.
Decode speed (llama-bench, build b10222)
Prefill throughput at multiple prompt sizes
| Config | pp512 | pp4096 | Δ vs Q4 |
|---|---|---|---|
| Q4_K_M @ 256K | 567 t/s | 700 t/s | — |
| Q5_K_M @ 128K | 614 t/s | 706 t/s | +0.9% |
| Q5_K_M @ 200K | 616 t/s | 706 t/s | +0.9% |
| Q6_K @ 128K | 585 t/s | 715 t/s | +2.1% |
Build b10222, q8_0-q4_1 KV, FA on, 150W, llama-bench 3 reps.
Prefill is flat across dense quants (~700-715 t/s at pp4096) — same bandwidth constraint as decode. The B70’s XMX engines (via oneDNN flash attention) push prefill well beyond what the raw 608 GB/s bandwidth suggests.
Token generation (decode) with MTP-4 delta
| Config | Base decode (tg128) | With MTP-4 (est.) | Δ vs base |
|---|---|---|---|
| Q4_K_M @ 256K | 18.4 t/s | ~29 t/s | +58% |
| Q5_K_M @ 200K | 16.2 t/s | ~24 t/s | +48% |
| Q6_K @ 128K | 16.0 t/s | ~24 t/s | +50% |
Base from llama-bench tg128 (no speculative). MTP-4 estimate from measured +35-50% gain (93-94% draft acceptance, build b10222, 165-180W).
Decode speed is quant-independent (~16-18 t/s base across all quants). The B70 is bandwidth-bound on weights — the quant only affects quality and VRAM, not decode speed. The choice between Q4/Q5/Q6 is purely about quality vs context ceiling.
Cross-hardware comparison: 27B dense class
| Hardware | VRAM | Model | Decode (base) | Prefill (pp4K) | Max ctx | Source |
|---|---|---|---|---|---|---|
| Arc Pro B70 32GB | 32 GB | ThinkingCap 27B Q5 | 16.2 t/s | 706 t/s | 200K | this post |
| Arc Pro B70 32GB | 32 GB | ThinkingCap 27B Q5 + MTP | ~24 t/s | 706 t/s | 200K | this post |
| RX 7800 XT 16GB | 16 GB | GLM-4.7-REAP-23B IQ4 | 59.8 t/s | 81.7 t/s | 32K | my bench |
| RTX 4090 24GB | 24 GB | Qwen 27B Q4 (est.) | ~35 t/s | ~1000 t/s | 128K | community |
The B70’s advantage is VRAM capacity (32 GB → 200K context) not raw speed. Consumer GPUs with less VRAM cap at 32-128K. The 7800 XT is faster per-token on smaller models but can’t fit a 19 GB 27B at 200K — it tops out at 32K for a 23B model.
MTP-4 speculative decoding: +35-50% decode
Multi-Token Prediction (MTP) is Qwen’s built-in speculative decoding. The model has a draft head that proposes multiple tokens per forward pass, which are then verified in parallel. On dense 27B:
| Config | Base decode | With MTP-4 | Gain |
|---|---|---|---|
| Q4_K_M @ 128K, 180W | ~18 t/s | ~29 t/s | +50% |
| Q5_K_M @ 200K, 165W | ~16 t/s | ~24 t/s | +35% |
Draft acceptance rate: 93-94% (measured 0.933-0.938). This is on the high end for self-draft speculative decoding — MTP shares the model’s own distribution, so acceptance is naturally high.
Power matters for dense MTP. At 165W you get ~24 t/s with MTP-4; at 180W it rises to ~29 t/s. The B70’s power scaling helps dense models because higher GPU frequency increases effective memory bandwidth utilization. MoE models don’t benefit from power scaling — they self-limit to ~130-140W draw regardless of the cap.
# MTP-4 flags (add to the server launch below)
--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75The full recipe: Q5_K_M @ 200K with MTP-4
This is the production config running daily through the Telegram bot bridge:
source /opt/intel/oneapi/setvars.sh --force > /dev/null 2>&1
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=0
export SYCL_CACHE_PERSISTENT=0
export SYCL_DEVICE_FILTER=level_zero
export ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE
export ONEAPI_DEVICE_SELECTOR=level_zero:0
export ZE_AFFINITY_MASK=0
# Set power cap (165W = efficiency sweet spot for dense MTP)
echo 165000000 | sudo tee /sys/class/hwmon/hwmon4/power1_cap
# Launch
~/llama.cpp/build-sycl-0801/bin/llama-server \
-m ~/models2/ThinkingCap-Qwen3.6-27B-Q5_K_M-MTP.gguf \
--host 0.0.0.0 --port 8765 \
-ngl 99 -ncmoe 0 \
-fa on -ctk q8_0 -ctv q4_1 \
-c 204800 -b 8192 -ub 4096 \
--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75 \
-t 8 --no-mmap \
--mmproj ~/models/gguf/mmproj-ThinkingCap-Qwen3.6-27B-f16.ggufExpected: ~24 t/s decode with MTP-4, 706 t/s prefill, 200K context, 2.7 GB VRAM free.
For pure text (no vision projector), add 889 MiB headroom — use this for the Q4_K_M @ 256K config.
Runtime flags handbook
| Flag | Value | Why |
|---|---|---|
-ngl 99 | All layers on GPU | Full GPU offload, no CPU fallback |
-ncmoe 0 | All MoE experts on GPU | (For dense models, this is a no-op but harmless) |
-fa on | Flash attention | Required for quantized KV cache to work |
-ctk q8_0 | K cache = 8-bit | Near-lossless (KL ~0.003). Never use q4_0 for K — catastrophic (KL ~5.5) |
-ctv q4_1 | V cache = 4-bit | V tolerates aggressive quantization (averaged in attention) |
-c 204800 | 200K context | Max safe context for Q5_K_M. Q4 can reach 256K |
-b 8192 | Batch size | Optimal for SYCL prefill throughput |
-ub 4096 | Micro-batch | Do NOT reduce — smaller values hurt prefill significantly |
-t 8 | 8 CPU threads | For prompt processing and CPU-side ops |
--no-mmap | Keep in VRAM | Don’t page model to disk |
--spec-type draft-mtp | MTP speculative decoding | +35-50% decode on dense. 93-94% acceptance |
--spec-draft-n-max 4 | Max 4 draft tokens | Best acceptance/speed trade-off |
--spec-draft-p-min 0.75 | Min acceptance probability | Below this, fall back to standard decode |
MoE comparison: when to use 35B instead of 27B
For reference, the Qwen3.6-35B-A3B MoE on the same B70:
| Config | Decode | Prefill | Max context | VRAM free |
|---|---|---|---|---|
| MoE Q4_K_XL @ 512K | 70.2 t/s | 1659 t/s | 512K | 1.6 GB |
| MoE Q5_K_M @ 256K | ~68 t/s | ~1621 t/s | 256K | 2.2 GB |
| Dense Q5_K_M @ 200K (MTP) | ~24 t/s | 706 t/s | 200K | 2.7 GB |
The MoE is 3x faster in decode and 2.3x faster in prefill, with 2.5x more context. Use MoE for throughput and long context; use dense for reasoning quality and tool use. Dense 27B with MTP is better for agentic workloads where reasoning chains matter more than raw t/s.
What changed since the original B70 posts
If you read the earlier B70 posts, three things have changed:
-
KV cache upgraded from q5_0-q4_1 to q8_0-q4_1. The old config is still acceptable (KL ~0.006-0.008), but q8_0 K halves the KL-divergence to ~0.003 at only ~6% more VRAM. This is now the fleet standard, validated against llama.cpp #23470.
-
Build advanced from b9853 to b10222 (+6-13% improvement). The upstream SYCL commits (oneDNN XMX flash attention #25222, fused top-k MoE #25217, RMS_NORM fusion #26015) are real and measurable. Prefill improved up to 12.8% at long context; decode improved 12.5%.
-
Model switched to ThinkingCap-Qwen3.6-27B. This is a token-efficient finetune of Qwen3.6-27B that keeps full accuracy with ~46% fewer thinking tokens. The MTP draft head is built into the weights — no separate draft model needed.
Benchmark methodology
All numbers from llama-bench (synthetic prompts, immune to HTTP API quoting bugs):
llama-bench \
-m ThinkingCap-Qwen3.6-27B-Q5_K_M-MTP.gguf \
-ngl 99 -ncmoe 0 -fa 1 -ctk q8_0 -ctv q4_1 \
-p 512,4096 -n 128 -r 5 -d 0 \
-b 8192 -ub 4096 -t 8 -dev SYCL0- pp4096 = prefill throughput at 4096-token synthetic prompt
- tg128 = decode throughput generating 128 tokens
- 5 repetitions per test, coefficient of variation < 0.5%
- 150W power cap, 54-66°C temperature range
- Build b10222 (
a7a6d0d26,build-sycl-0801, IntelLLVM 2026.0.0)
MTP-4 numbers measured via llama-server HTTP API (llama-bench doesn’t support speculative decoding). Decode rate from timings.predicted_per_second in the JSON response.
What to expect with Qwen 3.8 27B
Qwen 3.8 27B should drop into this recipe with minimal changes — same architecture class, same VRAM profile, same MTP support. The Q5_K_M quant will likely be ~19 GB (same as 3.6), fitting at 200K with q8_0-q4_1 KV.
If the 3.8 model includes the MTP draft head (Qwen has been shipping MTP since 3.0), the speculative decoding flags will work unchanged. If not, you’ll need a separate draft model or can use ngram speculation as a fallback.
The B70’s XMX engines (via oneDNN flash attention) are the key enabler — they push prefill to 700+ t/s on dense and 1600+ t/s on MoE, which is what makes long-context agentic workflows practical on a single 32 GB card.