So this started because I bought two of the base Snapdragon X boxes for some software and build testing at work (the cheap ones, about £300 each, not the Elite) and then they were sitting unused. With fast LPDDR5x (8448 MT/s) in them I thought I'd have a play and see if I could run proper models on them. I tried the usual engines and couldn't get anywhere near the speed I thought the hardware should do... So then my ADHD took over and I got carried away, and this is the result 🤣
ToRc is an inference engine written from scratch in Rust. Weeks of days and nights measuring, rewriting and measuring again went into it, and the numbers below are all measured on my own kit, different platforms etc... So I hope this proves useful to others 🙏🙃
ToRc on GitHub
Measured performance
If you can unzip a file, you can run your own model on your own machine. The download is a zip of about 13 MB (Windows x64, Windows on ARM, Linux x64 and Linux ARM, plus an experimental Mac one): unzip it, open ToRc, pick a model (a GGUF converts once, and it tells you how much space that takes first), press Auto-tune, and talk. A few minutes from download to chatting, nothing installed, and the app keeps everything (settings, chats, the lot) in that one folder. The only thing outside it is what the engine learns about your machine (Auto-tune's results and the like), in %LOCALAPPDATA%\torc or ~/.cache/torc.
What it can do:
- Uses the CPU, the NPU and the integrated GPU at the same time where it's present or faster, per model. Or just NPU + iGPU if you want to leave the machine usable while it runs.
- Tunes itself to your machine, and every setting can be overridden if you want to fiddle / let an agent try to wring every last token of performance for your particular setup.
- Speaks the OpenAI and Anthropic APIs, so Claude Code, OpenCode and anything else that talks to those just works. Full tool calling and MCP is also built in, and the desktop app has its own chat, tools and MCP servers if you just want to talk to it.
- Runs on Windows and Linux, x86 and ARM. (Linux: one user runs it on bare metal on a Snapdragon box under Ubuntu, and I've run it under WSL2, so tell me how it goes on yours.) There's also an EXPERIMENTAL Mac zip (Apple Silicon): one person has built and run it from source, and GitHub builds the zip, but I don't have a Mac to test it myself.
- It converts to its own file format for efficient SSD streaming where the model won't fit into memory and can split one model across multiple machines. I've tested two boxes over USB4; in principle it's N boxes. You pool their RAM, memory bandwidth and compute for enhanced performance.
Headline numbers, engine timed, read (prompt) / write (reply) tok/s:
- Qwen3.8 Flash Next, a 107 GB model, on two Snapdragon boxes with 64 GB of RAM between them (the experts that don't fit stream off the SSDs): ~67 / ~17.5 on a 16k prompt, ~64 / ~21 on a short one
- Qwen3.6 35B A3B, 4k prompt, two Snapdragon boxes: ~406 / ~54 (one box: ~283 / ~56)
- Same 35B on an i9-13900K (8 threads, CPU only), 4k prompt: ~243 / ~44, where llama.cpp manages ~143 / ~24
Against llama.cpp on the same hardware (CPU against CPU) it's quicker on most of what I've tried, and the gap grows with the prompt: on a 32k prompt it writes 2 to 10x faster. It isn't ahead everywhere, though. On my laptop llama.cpp reads long prompts slightly faster on GLM-4.7-Flash, and writes short replies a little faster on Moonlight and gpt-oss-20b. On the box, it reads short prompts a bit faster on the small dense Qwen3.5 models, until ToRc's NPU is on (then ToRc is ~2x). Full matrices, llama.cpp comparisons and every machine tested are in PERFORMANCE.md (link below).
Tested on: 2x Snapdragon X (base X1, 32 GB) boxes (1 box and 2), a Ryzen 7 7840HS laptop (Radeon 780M + RTX 4070), an i5-12600K desktop (RTX 3060) and an i9-13900K workstation (96 GB, Radeon AI PRO R9700 used for attention only). Everyday Windows installs with all my apps on them, not streamlined benchmark rigs on minimal Linux.
What I'm looking for:
- Testers, especially anyone with an X Elite or X2 Elite. This was all done on the base chip, so I'm curious what performance the bigger chips can manage.
- Mac people. As above, it builds and runs, but I can't test it myself, so numbers (and fixes) from more Macs would be brilliant.
- Linux on more real hardware, x86 especially.
- Holes. Find them and tell me what I can add or improve.
One question for you all: what is one feature you want from a local inference engine that nobody ships yet? I would genuinely like to try building it.
Basically I'm looking for collaboration with other people and projects, I've thoroughly enjoyed this project and I'd really like to see how far it can fly! 🕊️
Discrete GPU expert offload is on the list of my to-dos, but it was never the point of what I was trying to achieve, I use Strata on my big GPU workstation at work, and it's great, but that's a £5k machine, and out of reach of a lot of people. The point was making cheap hardware people are more likely to already have, actually useful - and use these NPUs that we now get with bigger models than the piddly ones we usually see.
Every model and machine I have measured is in PERFORMANCE.md 🙏
ToRc on GitHub