Community threads claim vLLM XPU gives 10x the speed of llama.cpp on the B70 (11,000 t/s prefill, 150 t/s decode on Qwen3.6-35B-A3B). I tested the full path — image, FP8, MXFP4, self-built native checkpoints, seven engine patches — and the model did run. The measured truth: llama.cpp wins single-stream by 6x, and the '150 t/s' claim turns out to be vLLM at 16 concurrent users (measured 153.4 t/s aggregate on our card).
Issue
The vLLM path has three gates, and each one turned out to be real: (1) the FP8 checkpoint for this model is 37.5 GB — it does not fit 32 GB of VRAM, so FP8 is off the table for 35B; (2) the MXFP4 (Intel 4-bit) checkpoints that exist publicly are in the compressed-tensors layout, which vLLM's XPU build rejects; (3) the prebuilt Intel images have an MXFP4 MoE path written for gpt-oss — the Qwen loader is broken in seven separate places, from tensor shapes to activation gating to hybrid-model page sizes.
Solution
I built the native-format MXFP4 checkpoint myself (fused 256 experts per layer into the w13/w2 layout, verified to 8.8e-5 MSE against BF16 ground truth), moved to the newer intel/vllm:0.17.0-xpu image, and patched the seven engine bugs in-container — 2D per-expert loader support, missing scale-key mapping, the silu-vs-swiglu_oai activation gate (the XPU kernel supports silu; the Python gate doesn't), CUDA-only device contexts in the linear-attention path, a contiguity check, and the hybrid block-size alignment that produced a page size the XPU flash-attention kernel rejects. The model then served and generated correct output.
A single llama.cpp upgrade — oneDNN SDPA extended to quantized KV (#25874) — takes the B70's long-context prefill from ~780 to 1871 t/s at 32K, +26-29% at 4-8K, and fixes a multi-turn corruption bug (#25880). Measured A/B on Qwen3.6-35B MoE and dense 27B, then promoted to production.
Issue
Long-context prefill was 2.4× slower than it should have been, and b10222 carried a multi-turn corruption bug (SDPA scale use-after-return, #25880). Two upstream PRs that fix exactly this — #25874 (quantized-KV XMX FA) and #25880 — shipped after our pinned release, so the fixes were not in the production build.
Solution
Rebuilt llama.cpp SYCL from upstream master (commit 071327508, b10255+) into build-sycl-0804 with the same config (-O3 -DNDEBUG, DNN=ON, F16=ON, GRAPH=ON, oneAPI 2026.0). Ran a same-day A/B against b10222 on both MoE 35B and dense 27B, then promoted the strictly-better build to production via a single config value (profiles.json runtime.binary), backed up and health-verified.
Everything you need to run production LLM inference on Intel Arc Pro B70 with llama.cpp SYCL — the build, runtime flags, all 5 model configs with measured VRAM boundaries, power tiers, KV cache KL-divergence analysis, and the b9853→b10222→master 0804 improvement data. 72.6 t/s MoE decode, 2128 t/s prefill, 512K context.
Issue
Running LLMs on Intel Arc requires SYCL-specific knowledge that doesn't exist in one place: which cmake flags, which env vars, which KV cache config, which power cap, which context length per model. Getting any of these wrong means either crashes, bad quality, or leaving performance on the table.
Solution
Fact-checked every claim against llama.cpp PRs and external benchmarks, ran a full boundary sweep measuring VRAM at every quant/context combo, A/B tested two SYCL builds, and consolidated everything into one recipe with the exact commands and measured numbers.
Every config that works for running Qwen3.6 27B dense on Intel Arc Pro B70 with llama.cpp SYCL — Q4/Q5/Q6 quant comparison, MTP-4 speculative decoding (+35-50%), q8_0 K + q4_1 V KV cache with measured KL-divergence, and the exact VRAM context ceiling for each quant. Validated on build b10222 with llama-bench.
Issue
The 27B dense model is VRAM-hungry (16-21 GB weights + 7-14 GB KV at high context), making the quant/context/KV trade-off non-obvious. The previous fleet used q5_0-q4_1 KV cache, but KL-divergence analysis showed q8_0 K + q4_1 V is near-lossless. The context ceiling per quant was unknown.
Solution
Tested all three quants (Q4_K_M, Q5_K_M, Q6_K) at every context length (128K-512K) with q8_0 K + q4_1 V KV cache. Measured VRAM boundaries, decode speed, and prefill with llama-bench. MTP-4 speculative decoding adds +35-50% decode. The KV insight: dense models use 3.8x more KV cache than MoE — so a 16 GB dense model can't pass 256K while a 25 GB MoE reaches 512K.
Running Poolside's 118B Laguna S 2.1 (256 experts, 8B active) on one Intel Arc Pro B70 32GB. 3.2x speedup from partial expert offload instead of dumping all experts to CPU. DFlash spec decode tested and dismissed.
Real single-stream timings from a 2026-07-16 B70 suite: long-prompt prefill near 1.7k t/s on MoE, dense Gemma 31B +51% decode with MTP-4, and Grok Build CLI with tools enabled.
Issue
Short-prompt prefill looked terrible, dense Gemma needed a path that was not stuck at ~16 t/s, and agent CLI numbers were being confused with engine throughput.
Solution
Re-measure with engine timings at long prompt sizes, replace Gemma 26B with Gemma 4 31B + Unsloth MTP draft + vision mmproj, keep MoE profiles for speed, and run Grok Build with tools on.
Practical setup for xAI Grok Build against a local OpenAI-compatible llama.cpp server: install, config.toml models, XAI_API_KEY, tools-on usage, profile switching, and real single-stream timings.
Issue
Grok Build defaults to cloud models; local routing needs correct base_url, model IDs, auth env, and an honest split between engine tok/s and agent wall time.
Solution
Install Grok Build, point custom models at http://127.0.0.1:8765/v1 with model id active (or matching served id), set XAI_API_KEY to the server API key, keep tools enabled for real agent work, and switch llama-server profiles underneath.
An analysis of KL divergence when quantizing the Key/Value cache in llama.cpp, and why the K-cache requires significantly higher precision than the V-cache.
Issue
I was using an aggressive `Q5_0` quantization for the Key (K) cache and `Q4_1` for the Value (V) cache. Over long contexts, the model's reasoning capabilities began to degrade, showing signs of hallucination and logic loops.
Solution
Researched the Kullback-Leibler (KL) divergence of various KV cache quantization formats in llama.cpp. Discovered that the Key cache is highly sensitive to precision loss due to dot-product attention mechanics. Shifted to a hybrid `K=Q8_0 / V=Q4_1` profile.
A post-mortem on attempting to compile and run experimental TurboQuant (WHT rotation) KV Cache compression on Intel Arc B70 GPUs using a custom SYCL fork.
Issue
We attempted to compile and run TurboQuant (TQ) — a promising new WHT-based quantization method — on Intel Arc hardware using an experimental SYCL fork, but hit hard driver and kernel limitations.
Solution
Documented the failure modes (specifically `SET_ROWS` view tensor crashes) and fell back to stable asymmetric block quantization (`K=Q8_0 / V=Q4_1`) for production use until upstream support matures.
How to tune llama.cpp on Intel Arc Pro B70 SYCL to double aggregate throughput under massive parallel loads with Mixture of Experts (MoE) models — and the 2026-08 head-to-head that settles the vLLM debate: continuous batching makes vLLM win at 16+ concurrent users (153 t/s) while llama.cpp wins every single-stream test (6.3x).
Issue
Baseline sequential generation hard-capped at ~67 tokens/second due to memory bandwidth starvation. Furthermore, naive scaling with large contexts (131K) and unoptimized batching caused immediate VRAM exhaustion (OOM) and server timeouts under load. Later (2026-08): the community insisted vLLM was strictly better — a claim that needed a head-to-head on the same card.
Solution
Diagnosed hardware bottlenecks and optimized the llama.cpp SYCL stack. Disabled heavy DNN operations (`GGML_SYCL_DISABLE_DNN=1`), quantized the KV Cache (`Q5_0`/`Q4_1`), and saturated the GPU using deep micro-batching (`-b 8192 -ub 4096`) under a 32-parallel request load. In 2026-08, stood up the full vLLM XPU path (self-built native MXFP4 checkpoint + 7 engine patches) and benchmarked both engines at 1/4/8/16 concurrent users on the same hardware.
Switching from symmetric q8_0 to asymmetric q5_0-q4_1 KV cache quantization freed 6.2 GB of VRAM per 128K context, pushed context ceilings to 256K on a 35B model, and was 3.3% faster in engine decode rate. Hardware-verified on Intel Arc Pro B70 32GB.
Issue
Standard q8_0 KV cache quantization used a 0.531 VRAM multiplier, capping a 35B Q5 model at 128K context on 32GB. The question was whether asymmetric K/V quantization (q5_0 for K, q4_1 for V) could unlock higher context lengths without hitting the quality cliff or degrading throughput.
Solution
Ran a 5-test hardware-verified benchmark suite on llama.cpp b9851 comparing q8_0-q8_0 against q5_0-q4_1 across control baseline, target comparison, flagship configs, and dense model validation. Calculated per-model VRAM budgets using measured multipliers from the Anbeeld 2026 KV cache benchmark methodology.
Corrected power scaling data for Qwen 27B MTP-4 speculative decoding after discovering single-prompt caching was inflating baselines by 4-5%. True MTP-4 gain at 180W is +35%, not +41%. Includes vision benchmark results after ffmpeg dependency fix.
Issue
The initial power sweep data was inflated. Single-prompt prefix caching was active during testing, which inflated the baseline measurements by approximately 4-5%. This made the speculative decoding gains appear larger than they actually were.
Solution
Rewrote the benchmark script to enforce warmup discards, isolate engine decode rate from wall-clock time, and maintain strict thermal cooldowns between test rounds. Re-ran the entire power sweep with the corrected methodology.
A reproducible case study for running Qwen3.6-35B-A3B on Intel Arc Pro B70 with llama.cpp SYCL on Ubuntu 26.04, including the exact build, runtime flags, benchmark data, and the persistent SYCL cache issue that caused model-load crashes.
Issue
The obvious checks all passed: the GPU was visible through Level Zero, ReBAR exposed the full 32GB BAR, the model fit in VRAM, and Vulkan could load it. SYCL still failed during model load, first with xe bcs engine resets and then with SIGSEGV crashes even when GPU offload was disabled.
Solution
I rebuilt llama.cpp from current master with Level Zero development headers installed, disabled Intel SYCL persistent kernel cache, pinned the Level Zero device explicitly, and reduced the environment to the smallest set of variables required for stable SYCL inference.