PART 2: 2x RX 7900 XTX + Qwen3.8-Flash-Next — EXL3, hipfire, Avalon, the rdna3 fork's NaN found, and every config quality-tested (spoiler: above ~3 bits it's a plateau)
A week ago I posted engine speeds for Flash-Next on my 2x 7900 XTX box. I got several interesting responses for more things to try, and software moves so fast in this space that several updates also came out, so here are the updates.
(Thanks especially to u/nasone32 (EXL3 fork), u/zeno_g2 (hipfire and others), u/sergitto11 (the #879/ROCm lead), u/xyzzing (the optimization ledger), and everyone who posted their own numbers.)
Same box: Ryzen 7 5800X, 64 GB DDR4, 2x RX 7900 XTX at PCIe 4.0 x8, now on ROCm 10.1. The speed bench is unchanged: 112 / 3.5k / 34k-token prompts, 512 out, temp 1.0 / top-p 0.95 / top-k 20, thinking on, MTP on, medians.
Critical note: quality is always checked, not just speed Every engine and quant below also ran my private coding-agent benchmark: real bugs mined from ~10 years of my own git history, with hidden tests (results public at lab.kylebaker.io). I compare configs problem by problem on the same problems, not leaderboard averages, and use a sign test on the problems where they disagree. Since late September that's ~8,000 scored local runs and ~2,100 agent-hours.
Also critical: I'm not benchmaxing. My speeds use the model's recommended sampling (temp 1.0, thinking on), not greedy, which flatters speed. The table is a fixed-prompt bench; the wall-time comparisons come from real multi-turn agent runs on hard bugs.
What that shows:
- Above ~2.5–3 bits, Flash-Next quants are a plateau on agentic coding.
- GSQ-RCO IQ3_XXS (76 GB), IQ3_S, UD-IQ4_XS, UD-Q4_K_XL (111 GB) and EXL3 4.05 are all within noise of each other.
- Two runs of the same config disagree on about as many problems as two different quants do.
- The only quant with a measurable loss is IQ1_M.
- I thought about running q8 on the cloud to verify this, but it would cost me somewhere between $30 and $120... not worth it to me, unfortunately, as it's purely a theoretical question. (A full run on my bench suite tends to require roughly 5M generated tokens.)
- "Does Strata trade quality for speed?": I see no sign of it. Strata's IQ3_XXS is my strongest Flash-Next config, ahead of every llama.cpp Flash-Next config I've run.
- The two quality-trading flags u/nasone32 mentioned (
STRATA_ROUTE_PRIOR,STRATA_ROUTE_TAIL_SKIP) are off on gfx1100. ROUTE_PRIOR is opt-in, and the rdna3 fork explicitly sets TAIL_SKIP=0. - But TAIL_SKIP is on by default on CUDA and on RDNA4 (gfx1200/1201) builds, so check yours if you compare quality on those.
- Note that the RDNA3 fork of strata DOES seem to trade some quality for speed in my testing! but strata's iq3xxs and iq3s have both performed within noise ("slightly better", but within noise) of UD q4 k xl flash next on llama.
- The two quality-trading flags u/nasone32 mentioned (
- On 97 problems both ran, Strata (GSQ IQ3_XXS) solved 82 and llama.cpp (UD-Q2_K_XL) 79, with 10 vs 7 problems solved only by one side: noise.
- On the one file I ran on both engines (UD-IQ4_XS), Strata solved 6 of 6 and llama.cpp 5 of 6. Small, but pointing the same way.
Results
| engine (2x XTX unless noted) | quant | decode t/s (112 / 3.5k / 34k) | prompt t/s (3.5k / 34k) | quality vs upstream (paired) |
|---|---|---|---|---|
| Strata 0.2.13.1 | GSQ IQ3_XXS | 79 / 100 / 84 | 1,440 / 2,496 | baseline |
| Strata-rdna3 fork, NaN-free variant | GSQ IQ3_XXS | ~95–125 (119–127 in agent runs) | 2,418 / 3,346 | probably 2–5 pts weaker on the hardest problems (still testing) |
| EXL3 fork, 4.05 bpw | EXL3 | 26 / 31 / 29 | 686 / 1,122 | tie (188 vs 193 of 222; fork 183) |
| EXL3 fork, 3.05 bpw | EXL3 | 49 / 50 / 50 | 876 / 1,349 | 19 vs 21 of 26 (noise) |
| Avalon 1200934a7, 1 card | Avalon IQ3_KS | 87 / 90 / 90 | 1,051 / 2,385 | not run at scale |
Avalon 1200934a7, 2 cards, LLAMA_STAGES=0 + split | Avalon IQ3_KS | 64 / 75 / 64 | 1,741 / 2,434 | not run at scale |
Notes
- rdna3 fork NaN (#879), found. The lead that ROCm's runtime caused it didn't hold on my box: same NaN rate on ROCm 7.2.4 and 10.1. Bisecting with a replay of the requests that NaN'd pinned it to the fork's refill settings (Willing-Cover2914's reinstall fixing it is consistent with that).
- With the 10 refill settings removed: zero NaNs across ~300 real agent runs.
- It matches the the same refill machinery as the fork's issues #3 and #14.
- I'm narrowing it to the exact key(s) now and will send the fork a PR soon.
- Claude's write-up: lab.kylebaker.io/blog/hunting-a-nan-in-strata-rdna3
- The fork vs upstream on quality.
- Over 222 paired problems, upstream solved 193 and the fork 183.
- I re-ran the 30 problems where they disagreed, fresh on both engines in the same week: upstream 22, fork 16.
- Combined, about a 2% chance that's luck. So the rdna3 fork looks a few points weaker on the hardest problems.
- While it looks dramatically faster, a weaker model writes more tokens... net: it's still ~16% less wall time per run. More paired runs are queued to pin down the size of the gap. But, again, at this point it's within 98% certainty that it's a few points weaker than upstream strata.
- EXL3 (u/nasone32).
- While in theory the 4.05 bpw EXL3 flash next quant should be even better than a traditional q4 quant... in my bench, it within-noise-ties IQ3_XXS on quality. IT does write ~28% fewer tokens, which suggests it may be stronger, but on 64 GB it can't keep 4.05 fully resident, so it runs at ~2.3x the wall time
- This one was particularly interesting to me, as I was wondering if this would 'break the plateau', in theory this should be the 'strongest' flash next quant I can realistically run on my machine. Alas--given that I didn't actually see any real-world improvement, I can't say the 2.3x slower runtime (on my box) is worth it.
- Likewise, ran the 3~bpw EXL3, still significantly slower than strata (though much faster than the 4.05bpw). Still all within noise on strength/solves on my benchmark.
- Tip for agent use: set
--conversation-cache-mib. It defaults to 0, and with sub-agents alternating conversations the engine re-read the whole ~150k-token prompt on every switch: ~30 s each, versus ~2 s with a 16 GB cache. That fixed a multi-hour time-out. - hipfire (u/zeno_g2). On the dense 27B it's shockingly good:
- on one XTX, about 2x llama.cpp's speed: decode 94–98 vs 45–63 t/s, prompt reads ~2.1k vs ~0.9k;
- no quality loss I can detect: 15 vs 17 of 23 paired, in about half the wall time; 60 more problems are running now.
- Flash-Next on hipfire didn't work out on 64 GB: its 2-bit build only runs on one card, at ~15 t/s. Sad.
- Haven't tried radiance or caminus yet.
- Avalon.
- Current main fixed the prompt-ring regression and the 32k out-of-memory I reported.
- Two cards with
LLAMA_STAGES=0+ manual split is the setup that works for me. - I'm re-checking default pipeline stages on the latest main, since issue #55 reports it working well on the same hardware.
The rdna3 fork's kernel switches, patched onto upstream Strata. The rdna3 fork's speed has two parts: tensor parallel across both cards (most of its decode gain) and a table of gfx1100 kernel switches, mostly for reading prompts. I turned on 15 of those switches on upstream Strata 0.2.13.1:
- Bench prompt reads roughly doubled: 2,549 / 4,886 t/s at 3.5k / 34k tokens, vs 1,440 / 2,499 plain. Decode unchanged.
- Real agent runs: no wall-time gain. Prompt reading is a small share of an agent run, and cached prefixes mean only the new tokens get read.
- Quality: 18 vs 21 solved on the same problems. Slightly worse, but within noise.
- Strata 0.2.14+ now ships its own prompt kernels by default, so this is mostly moot; I'm testing 0.2.16 now (speed plus a quality pairing).
7900 XTX vs 3090?
I think dual 7900xtx is comparable to dual 3090 right now, which is kind of wild. Can anyone sanity check me on this? I bought my xtx's at about half the price of what 3090s were going for on facebook marketplace a few months ago (about $700~ vs. the about $1300~ 3090s were going for)
- Against the best public dual-3090 Flash-Next recipe, on its own terms (greedy, thinking off): 2x XTX with the rdna3 fork matches or beats it on code and file edits (up to +31%), is 2–9% behind on prose, and reads long prompts at the same speed.
- Watch out for greedy numbers: greedy flatters code by ~1.6x on this setup. Converted to normal sampling, I estimate the dual 3090 at ~90 t/s code and ~105–112 prose, against the fork's 104–111 and 95–109. Roughly a tie, with the XTX ahead on code.
How fast this moved
Two months ago, before strata, Flash-Next on this box meant llama.cpp on one card with experts streaming from RAM: ~15–20 t/s decode. Upstream Strata does 80–100 t/s (about 5x, no quality cost); the fork 105–125 (6–8x, a few points weaker on the hardest problems). For reference, llama.cpp on both cards does 31–36 t/s on the same bench today.
At this point RAM is my bottleneck; I'd be more likely to spend $700 floating around on more ram than another gpu.
Still testing more things, always: Strata 0.2.16 (all-own kernels, speed plus a quality pairing), the fork PR, more fork-vs-upstream paired runs, and Ternary-Bonsai-27B (5.9 GB, 1.58-bit) as a reference. If you're running radiance or caminus on RDNA3, I'd love numbers.
edit:
Strata 0.2.16 passed its speed and sanity check. Both fresh starts were clean against a same-session 0.2.13.1 baseline:
┌─────────────────────────┬───────────────────────┬─────────────┐
│ 2 XTX, IQ3_XXS, 262k │ prompt read │ decode │
├─────────────────────────┼───────────────────────┼─────────────┤
│ 112-token prompts │ +14% (282 vs 248 t/s) │ +4% │
├─────────────────────────┼───────────────────────┼─────────────┤
│ 3.5k │ +20% (1,724 vs 1,432) │ +8% │
├─────────────────────────┼───────────────────────┼─────────────┤
│ 34k │ +11% (2,777 vs 2,498) │ −3% (noise) │
├─────────────────────────┼───────────────────────┼─────────────┤
│ 256 / 512-token prompts │ +22% / +32% │ ±, noisy │
└─────────────────────────┴───────────────────────┴─────────────┘
- Correctness: both known-answer checks were right on every start, with no garbage.
- Memory: it fits about 1,000 more experts in VRAM than 0.2.13.1 (18.8 vs 17.3 GiB of expert cache).
- Quality pairing: its 26-problem pairing (three items) started automatically and is running now, with its 5-minute check armed.
Avalon stuff is more complicated, I still consider that project a WIP for my machine but am happily contributing my findings upstream as I find them and have high hopes for it going forward.
