Processing img j9s5oa7dem2i1...
What this is (so you can weight the numbers): a native inference engine + compiler we built from scratch β no Python and no CUDA toolkit at inference time β that runs your existing checkpoint. It's an early / experimental release, the engine itself is sealed (not open source), and the benchmark below is our own, first-party. I'm one of the people who built it (founder/team), so apply the usual discount for a vendor posting their own numbers. I've put the operating point on every row and the places where we're worse in bold.
The result: on Qwen3.6-35B-A3B (35B-param
MoE, ~3B active), our decode throughput beats llama.cpp Q4_K_M on the same GPU, same pod, at a comparable ~4-bit quant (ours = VQ-balanced lut4, theirs = Q4_K_M).
Operating point for every row below: autoregressive decode, 512 ctx, single-stream (batch=1), our measurement on the same machine.
| GPU |
veizik (tok/s) |
llama.cpp Q4_K_M (tok/s) |
speedup |
| RTX 4090 |
272.5 |
184.5 |
1.48Γ |
| A100 80GB |
192.1 |
140.9 |
1.36Γ |
| RTX 5090 |
345.0 |
258.9 |
1.33Γ |
| H100 80GB |
268.8 |
215.2 |
1.25Γ |
Not speculative decoding (the thing this s
ub asks first): both sides are plain batch-1 autoregressive decode β no draft model, no MTP on either side. The gap is the engine, not a speculation trick.
The tradeoff (quant quality): at the same ~4-bit, our fidelity check puts us a touch behind Q4_K_M β kld_vs_q4km = 1.28 vs a BF16 reference, i.e. slightly less faithful. So: a throughput win at comparable bitrate with a small fidelity cost β want the most faithful 4-bit, Q4_K_M is still it. (llama.cpp also wins on hardware breadth and being open source.)
Other boundaries:
- Single-stream decode only. For batched / concurrent serving, vLLM and TensorRT-LLM win β not claiming that regime.
- Our own benchmark, same-pod β run it yourself (below).
- NInfer: on the 5090 we're on par (~1%), not ahead.
- SGLang exl3 u/3**.0bpw** is cited faster β but a much lower bitrate / quality point, not apples-to-apples with ~4-bit.
Fits on a 3090 too (supporting point, not the headline): the 35B-A3B is 21.44 GiB resident (balanced) on a single 24GB 3090 and decodes 194β2
21 tok/s (our measurement). Fitting a 35B on a 3090 is table-stakes β llama.cpp does it too β so I'm calling this a supporting point, not a brag. The llama.cpp 3090 figure I have is cited, not same-pod, so I'm deliberately not putting a multiplier on the 3090.
If you want to sanity-check the engine at sizes you can actually run locally, small dense Qwen3.5 on the 3090, same-pod vs llama.cpp (AR decode, single-stream):
| model |
speedup vs llama.cpp (3090, same-pod) |
| 0.8B |
1.93Γ |
| 2B |
1.47Γ |
| 4B |
1.23Γ |
| 9B |
~par |
Other measured points (own numbers, different workloads β deliberately NOT paired with the llama.cpp table):
- 5090 fast-mode decode: 544.
- 7 tok/s (a different config from the AR@512 row above β don't compare it to the 258.9).
- H100 long-context: scales to 131,072 tokens at 172 tok/s.
Why we built it: one engine, no Python glue and no CUDA toolkit to install at inference, that takes a checkpoint you already have and runs it β and the same engine also covers image, video and robotics, down to a Raspberry Pi CPU. That breadth under one runtime is the thing we couldn't get out of any single-model engine (llama.cpp / vLLM / NInfer / ik_llama), so we wrote our own.
Repro: our full own-number board β all these GPUs, every mode β is live at veizik.com/benchmarks, and the kit at github.com/veizikhq/veizik-cuda (cuda
_deploy/bench/, method in REPRODUCIBILITY.md) runs veizik run on your own GPU so you can clock the engine yourself. The llama.cpp rows are our own same-pod Q4_K_M measurement β run llama.cpp Q4_K_M on the same box and check the gap. Got a 4090 / 5090 / A100 / H100? Post your numbers β especially if the gap doesn't hold on your setup.
License: the engine is sealed / proprietary (not open source); it runs your open checkpoints.
Happy to get into kernel-level, quant-fairness or determinism questions in the comments.