So
I've been messing with EXL3 quants of Qwen3.6-35B-A3B on my two mismatched cards, a 3060 12GB and a 5060 Ti 16GB. The thing that kept bugging me was that EXL3 only does whole-number bit rates per tensor. A tensor that wants about 4.5 bits gets either 4 or 5, and on a 35B MoE that rounding adds up fast.
So I forked it and added 1.5 / 2.5 / ... / 7.5 bpw, with the kernels to go with them. Then I measured where this model actually loses accuracy when quantized and placed the bits from that.
The weirdest finding: most of the damage on hard text (around 87% of the KLD) wasn't the experts being imprecise. It was tokens getting routed to different experts than the full model would pick. So the attention/DeltaNet layers that feed the routers get 8 bits, and the experts in the middle layers can drop to 4.5 without much harm.
Against BF16, a build with whole-number rates only came out at 24.6 GB with a KLD of 0.0166. The half-bit builds are smaller and closer to BF16:
T5 (24.5 GB, KLD 0.0156) is the one to grab if you want it as close to BF16 as possible. About 177 tok/s on my setup.
T5 FV (23.9 GB, KLD 0.0157) is the faster version: 0.6 GB smaller, about 204 tok/s.
Both keep the vision tower and the MTP head, so speculative decoding just works. I ran both through 13 benchmark suites (MMLU-Pro, AIME, GPQA Diamond, LiveCodeBench, HumanEval, IFEval and a few others, 1,910 questions total) with thinking on and the full thinking budget. They landed within a handful of questions of each other: 1,546 vs 1,557, which is basically noise for a single run. Per-suite numbers are on the model cards.
You do need the fork, since stock exllamav3 can't load the half-bit tensors. I tried to make that painless: clone it and run `MODEL=/path/to/model bash install.sh`. It sorts out PyTorch for your driver, compiles, tests every bit rate, then figures out how to split the model across whatever GPUs you have. No CUDA toolkit needed. Fair warning: the compile takes a while (15 to 40 minutes).
This also leads into my EXL3 quant project which I’ll be uploading some more models any recommendations please let me know as my hardware is meh bigger model may not be possible or will take weeks with my setup so will be aiming for under 40b models if yall have any recommendations
- Fork: https://github.com/hunterxma/exllamav3-half-bit
- T5: https://huggingface.co/HydraAiM/Qwen3.6-35B-A3B-EXL3-HalfRate-T5
- T5 FV: https://huggingface.co/HydraAiM/Qwen3.6-35B-A3B-EXL3-HalfRate-T5-FV
Big thanks to turboderp for EXL3 and to the Qwen team for the model. If anyone runs these on other cards, I'd love to see your speeds, especially single 24GB setups with some experts on the CPU
THIS HAS ONLY BEEN TESTED ON Ampree and Blackwell so any other cards have not been tested if EXL3 works on them me too broke for multi card brand setups lol sorry