r/IntelArc • u/Strict-Elevator9352 • 12h ago
News inferzero: an LLM engine written from scratch for Intel Arc, from laptop iGPUs to 4× B60. Still far from what the cards can do, looking for contributors
I’ve been building inferzero, an open-source (Apache 2.0) inference engine only for Intel GPUs: C++ on Level Zero, kernels in OpenCL C, GGUF files as they are. It started on a Core Ultra 5 125H laptop (no XMX, DP4a only) and now runs on Arc B-series cards, Alchemist, and several GPUs at once.
Qwen3.8 27B (IQ3_S), measured by users on their own hardware:
| GPU | 4K prompt | Generation |
|---|---|---|
| Arc Pro B70 (Linux) | 2,897 tok/s | 35.2 tok/s, 70.6 with MTP |
| Arc B580 12 GB (PCIe 3.0) | 1,877 tok/s | 7.1 tok/s (the model barely fits: 0.5 GB of it in RAM, with ARC_HOST_KEEP_GB=0.75) |
| Arc Pro B60 | 1,726 tok/s | 22.7 tok/s |
| Arc A770 16 GB | 1,223 tok/s | 12.8 tok/s |
| Arc B390 (Panther Lake iGPU) | 592 tok/s | – |
| Arc 140T (Core Ultra 9 285H iGPU) | 390 tok/s | 6.0, 16.9 with MTP + DFlash2 |
| 4× Arc Pro B60 (one model split, older release) | 1,888 tok/s | 53 tok/s |
What’s in it:
- Qwen3.5 / 3.6 / 3.8 (dense, MoE, Flash Next with 512 experts), Gemma 4, DeepSeek-V4.1-Flash over several GPUs.
- MoE experts streamed from the SSD when the model is bigger than the GPU memory: the 35B-A3B and Flash Next run on a 32 GB laptop.
- Speculative decoding (MTP heads, DFlash2) that gives exactly the text of plain decoding.
- Every model checked against an independent f32 CPU reference (KL ≤ 0.015). Same prompt, same logits, bit for bit.
- An OpenAI / Anthropic / Ollama-compatible server with a web UI (web search, voice, transcripts), Windows and Linux, no Python needed.
Why I’m posting: it’s still far from the hardware’s limits, on every card. The 140T’s prompt GEMM runs at ~33 % of its XMX peak, while a probe shows ~85 % is reachable. The B70’s prompt GEMMs sit at ~62 % of the int8 peak, and generation at ~62 % of memory bandwidth. Alchemist and most iGPUs are barely tuned, and on Windows multi-GPU sums still go through host memory.
Two ways to help:
- Run it on your Arc and post the numbers. One script writes a zip for the issue. Every card and setup we haven’t measured is the most useful thing.
- Code: kernels (GEMM/GEMV on XMX and DP4a), multi-GPU, Linux packaging, the web UI. CONTRIBUTING.md has where the speed is missing, how to build, and the rules a change is checked against (KL, bit-identical logits, before/after numbers).
Repo: https://github.com/rizagp/inferzero · Releases: https://github.com/rizagp/inferzero/releases/latest · How to help: https://github.com/rizagp/inferzero/blob/main/CONTRIBUTING.md

