r/ROCm • • 5h ago

PART 2: 2x RX 7900 XTX + Qwen3.8-Flash-Next — EXL3, hipfire, Avalon, the rdna3 fork's NaN found, and every config quality-tested (spoiler: above ~3 bits it's a plateau)

13 Upvotes

A week ago I posted engine speeds for Flash-Next on my 2x 7900 XTX box. I got several interesting responses for more things to try, and software moves so fast in this space that several updates also came out, so here are the updates.

(Thanks especially to u/nasone32 (EXL3 fork), u/zeno_g2 (hipfire and others), u/sergitto11 (the #879/ROCm lead), u/xyzzing (the optimization ledger), and everyone who posted their own numbers.)

Same box: Ryzen 7 5800X, 64 GB DDR4, 2x RX 7900 XTX at PCIe 4.0 x8, now on ROCm 10.1. The speed bench is unchanged: 112 / 3.5k / 34k-token prompts, 512 out, temp 1.0 / top-p 0.95 / top-k 20, thinking on, MTP on, medians.

Critical note: quality is always checked, not just speed Every engine and quant below also ran my private coding-agent benchmark: real bugs mined from ~10 years of my own git history, with hidden tests (results public at lab.kylebaker.io). I compare configs problem by problem on the same problems, not leaderboard averages, and use a sign test on the problems where they disagree. Since late September that's ~8,000 scored local runs and ~2,100 agent-hours.

Also critical: I'm not benchmaxing. My speeds use the model's recommended sampling (temp 1.0, thinking on), not greedy, which flatters speed. The table is a fixed-prompt bench; the wall-time comparisons come from real multi-turn agent runs on hard bugs.

What that shows:

  • Above ~2.5–3 bits, Flash-Next quants are a plateau on agentic coding.
    • GSQ-RCO IQ3_XXS (76 GB), IQ3_S, UD-IQ4_XS, UD-Q4_K_XL (111 GB) and EXL3 4.05 are all within noise of each other.
    • Two runs of the same config disagree on about as many problems as two different quants do.
    • The only quant with a measurable loss is IQ1_M.
    • I thought about running q8 on the cloud to verify this, but it would cost me somewhere between $30 and $120... not worth it to me, unfortunately, as it's purely a theoretical question. (A full run on my bench suite tends to require roughly 5M generated tokens.)
  • "Does Strata trade quality for speed?": I see no sign of it. Strata's IQ3_XXS is my strongest Flash-Next config, ahead of every llama.cpp Flash-Next config I've run.
    • The two quality-trading flags u/nasone32 mentioned (STRATA_ROUTE_PRIOR, STRATA_ROUTE_TAIL_SKIP) are off on gfx1100. ROUTE_PRIOR is opt-in, and the rdna3 fork explicitly sets TAIL_SKIP=0.
    • But TAIL_SKIP is on by default on CUDA and on RDNA4 (gfx1200/1201) builds, so check yours if you compare quality on those.
    • Note that the RDNA3 fork of strata DOES seem to trade some quality for speed in my testing! but strata's iq3xxs and iq3s have both performed within noise ("slightly better", but within noise) of UD q4 k xl flash next on llama.
  • On 97 problems both ran, Strata (GSQ IQ3_XXS) solved 82 and llama.cpp (UD-Q2_K_XL) 79, with 10 vs 7 problems solved only by one side: noise.
  • On the one file I ran on both engines (UD-IQ4_XS), Strata solved 6 of 6 and llama.cpp 5 of 6. Small, but pointing the same way.

Results

engine (2x XTX unless noted)quantdecode t/s (112 / 3.5k / 34k)prompt t/s (3.5k / 34k)quality vs upstream (paired)
Strata 0.2.13.1GSQ IQ3_XXS79 / 100 / 841,440 / 2,496baseline
Strata-rdna3 fork, NaN-free variantGSQ IQ3_XXS~95–125 (119–127 in agent runs)2,418 / 3,346probably 2–5 pts weaker on the hardest problems (still testing)
EXL3 fork, 4.05 bpwEXL326 / 31 / 29686 / 1,122tie (188 vs 193 of 222; fork 183)
EXL3 fork, 3.05 bpwEXL349 / 50 / 50876 / 1,34919 vs 21 of 26 (noise)
Avalon 1200934a7, 1 cardAvalon IQ3_KS87 / 90 / 901,051 / 2,385not run at scale
Avalon 1200934a7, 2 cards, LLAMA_STAGES=0 + splitAvalon IQ3_KS64 / 75 / 641,741 / 2,434not run at scale

Notes

  • rdna3 fork NaN (#879), found. The lead that ROCm's runtime caused it didn't hold on my box: same NaN rate on ROCm 7.2.4 and 10.1. Bisecting with a replay of the requests that NaN'd pinned it to the fork's refill settings (Willing-Cover2914's reinstall fixing it is consistent with that).
    • With the 10 refill settings removed: zero NaNs across ~300 real agent runs.
    • It matches the the same refill machinery as the fork's issues #3 and #14.
    • I'm narrowing it to the exact key(s) now and will send the fork a PR soon.
    • Claude's write-up: lab.kylebaker.io/blog/hunting-a-nan-in-strata-rdna3
  • The fork vs upstream on quality.
    • Over 222 paired problems, upstream solved 193 and the fork 183.
    • I re-ran the 30 problems where they disagreed, fresh on both engines in the same week: upstream 22, fork 16.
    • Combined, about a 2% chance that's luck. So the rdna3 fork looks a few points weaker on the hardest problems.
    • While it looks dramatically faster, a weaker model writes more tokens... net: it's still ~16% less wall time per run. More paired runs are queued to pin down the size of the gap. But, again, at this point it's within 98% certainty that it's a few points weaker than upstream strata.
  • EXL3 (u/nasone32).
    • While in theory the 4.05 bpw EXL3 flash next quant should be even better than a traditional q4 quant... in my bench, it within-noise-ties IQ3_XXS on quality. IT does write ~28% fewer tokens, which suggests it may be stronger, but on 64 GB it can't keep 4.05 fully resident, so it runs at ~2.3x the wall time
  • This one was particularly interesting to me, as I was wondering if this would 'break the plateau', in theory this should be the 'strongest' flash next quant I can realistically run on my machine. Alas--given that I didn't actually see any real-world improvement, I can't say the 2.3x slower runtime (on my box) is worth it.
  • Likewise, ran the 3~bpw EXL3, still significantly slower than strata (though much faster than the 4.05bpw). Still all within noise on strength/solves on my benchmark. - Tip for agent use: set --conversation-cache-mib. It defaults to 0, and with sub-agents alternating conversations the engine re-read the whole ~150k-token prompt on every switch: ~30 s each, versus ~2 s with a 16 GB cache. That fixed a multi-hour time-out.
  • hipfire (u/zeno_g2). On the dense 27B it's shockingly good:
    • on one XTX, about 2x llama.cpp's speed: decode 94–98 vs 45–63 t/s, prompt reads ~2.1k vs ~0.9k;
    • no quality loss I can detect: 15 vs 17 of 23 paired, in about half the wall time; 60 more problems are running now.
    • Flash-Next on hipfire didn't work out on 64 GB: its 2-bit build only runs on one card, at ~15 t/s. Sad.
  • Haven't tried radiance or caminus yet.
  • Avalon.
    • Current main fixed the prompt-ring regression and the 32k out-of-memory I reported.
    • Two cards with LLAMA_STAGES=0 + manual split is the setup that works for me.
    • I'm re-checking default pipeline stages on the latest main, since issue #55 reports it working well on the same hardware.

The rdna3 fork's kernel switches, patched onto upstream Strata. The rdna3 fork's speed has two parts: tensor parallel across both cards (most of its decode gain) and a table of gfx1100 kernel switches, mostly for reading prompts. I turned on 15 of those switches on upstream Strata 0.2.13.1:

  • Bench prompt reads roughly doubled: 2,549 / 4,886 t/s at 3.5k / 34k tokens, vs 1,440 / 2,499 plain. Decode unchanged.
  • Real agent runs: no wall-time gain. Prompt reading is a small share of an agent run, and cached prefixes mean only the new tokens get read.
  • Quality: 18 vs 21 solved on the same problems. Slightly worse, but within noise.
  • Strata 0.2.14+ now ships its own prompt kernels by default, so this is mostly moot; I'm testing 0.2.16 now (speed plus a quality pairing).

7900 XTX vs 3090?

I think dual 7900xtx is comparable to dual 3090 right now, which is kind of wild. Can anyone sanity check me on this? I bought my xtx's at about half the price of what 3090s were going for on facebook marketplace a few months ago (about $700~ vs. the about $1300~ 3090s were going for)

  • Against the best public dual-3090 Flash-Next recipe, on its own terms (greedy, thinking off): 2x XTX with the rdna3 fork matches or beats it on code and file edits (up to +31%), is 2–9% behind on prose, and reads long prompts at the same speed.
  • Watch out for greedy numbers: greedy flatters code by ~1.6x on this setup. Converted to normal sampling, I estimate the dual 3090 at ~90 t/s code and ~105–112 prose, against the fork's 104–111 and 95–109. Roughly a tie, with the XTX ahead on code.

How fast this moved

Two months ago, before strata, Flash-Next on this box meant llama.cpp on one card with experts streaming from RAM: ~15–20 t/s decode. Upstream Strata does 80–100 t/s (about 5x, no quality cost); the fork 105–125 (6–8x, a few points weaker on the hardest problems). For reference, llama.cpp on both cards does 31–36 t/s on the same bench today.

At this point RAM is my bottleneck; I'd be more likely to spend $700 floating around on more ram than another gpu.

Still testing more things, always: Strata 0.2.16 (all-own kernels, speed plus a quality pairing), the fork PR, more fork-vs-upstream paired runs, and Ternary-Bonsai-27B (5.9 GB, 1.58-bit) as a reference. If you're running radiance or caminus on RDNA3, I'd love numbers.


edit:

Strata 0.2.16 passed its speed and sanity check. Both fresh starts were clean against a same-session 0.2.13.1 baseline:

  ┌─────────────────────────┬───────────────────────┬─────────────┐
  │  2 XTX, IQ3_XXS, 262k   │      prompt read      │   decode    │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 112-token prompts       │ +14% (282 vs 248 t/s) │ +4%         │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 3.5k                    │ +20% (1,724 vs 1,432) │ +8%         │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 34k                     │ +11% (2,777 vs 2,498) │ −3% (noise) │
  ├─────────────────────────┼───────────────────────┼─────────────┤
  │ 256 / 512-token prompts │ +22% / +32%           │ ±, noisy    │
  └─────────────────────────┴───────────────────────┴─────────────┘
  • Correctness: both known-answer checks were right on every start, with no garbage.
  • Memory: it fits about 1,000 more experts in VRAM than 0.2.13.1 (18.8 vs 17.3 GiB of expert cache).
  • Quality pairing: its 26-problem pairing (three items) started automatically and is running now, with its 5-minute check armed.

Avalon stuff is more complicated, I still consider that project a WIP for my machine but am happily contributing my findings upstream as I find them and have high hopes for it going forward.


r/ROCm • • 21h ago

AMD ROCm 10.2 Released With Latest Optimizations, Expanded Ryzen Laptop Support

Thumbnail
phoronix.com
43 Upvotes

r/ROCm • • 10h ago

Just Purchased RX 7900 XTX 24 GB

Thumbnail
6 Upvotes

Any suggestions related to models are welcome.


r/ROCm • • 1h ago

Strix Halo 2x Nightlies build

• Upvotes

so i was benchmarking my own inference code for strix halo https://github.com/Legendofzaday/halodiffuse against comfyui and found out that rocm nightlies v2 wheels are twice as fast as the stable wheels on sd3.5 large turbo this was the case:

halodiffuse: 4.6s
comfyui nightly v2 wheels: 5.6s
comfyui stable: 11.8s

cold start times have also improved a lot:

halodiffuse: 9.1s
comfyui nightly v2 wheels: 22.6s
comfyui stable wheels: 31.2s


r/ROCm • • 8h ago

Four engines, seven models, one Radeon AI PRO R9700: local LLM benchmarks

Thumbnail
1 Upvotes

r/ROCm • • 13h ago

DL research on AI 395/495 in PyTorch?

2 Upvotes

I'm not sure if this is the right place to ask this, since it seems like a lot of people are trying to run local LLMs, but can't hurt to ask.

Has anyone had any experience doing DL research using PyTorch on one of these AI 395/495 dev boxes (any brand)? AMD puts out their own version and they're are a number of vendors with their own versions. I'm wondering how stable it is. These are dramatically cheaper than the DGX Sparks, but I don't want to pull out what little hair I have left on top of my head trying to get it to run.

I would develop in Linux or WSL (don't care which). But I think I read you can't even do training under windows (not that I would do any serious research in Windows) so that suggests the drivers are still pretty immature and I should probably just pony up the $2000-3000 extra for a spark.

Thanks. If there's a better place to ask, please let me know.


r/ROCm • • 20h ago

3 x AI 9700 build - 96gb VRAM

Thumbnail gallery
5 Upvotes

r/ROCm • • 17h ago

My AMD V620 32GB is running solid after the power mod you guys recommended! Really appreciate the help! 🙌

Post image
2 Upvotes

r/ROCm • • 18h ago

I built ROBJØRN, a free one-click overclocker for Radeon Pro V620 / V540 / V520 / V320 server GPUs.

Post image
2 Upvotes

The heist for more power local LLMs 🪓

Datacenter GPUs ship tuned for racks of servers running 24/7. That means safe power limits, capped clocks, and settings you can't touch. AMD's standard tuning tools don't support these cards, so that extra performance is locked away.

So I built a Viking to go get it.

Robjørn (Rob + Bjørn, a Viking Robin Hood) is a one-click overclocker for:

Radeon Pro V620 · V540 · V520 · V320

Load your card's firmware. Drag a slider. Hit Apply. Done.

Why you'll like it

  • 🛡 Never flashes your card. Your firmware is only read, never written. One click puts everything back to stock.
  • 🎯 Built from YOUR card. It reads your card's own power table, so every slider starts at your card's real stock values.
  • 🔒 SHA-256 locked. Every table is hashed, and the apply script refuses to write anything that doesn't match.
  • 🚦 Safety rails. A layout check runs on every firmware load, and Apply is blocked if anything looks off.
  • ⚡ Presets or full control. Stock · Efficient · Mild OC · 2350 MHz (V620), or tune every value yourself.
  • 🪶 One tiny .exe. About 185 KB. No installer, no Python. It runs on the .NET Framework built into Windows 10 and 11.

3 steps. That's it.

1. Grab your firmware. GPU-Z → click the save icon next to BIOS Version, or run amdvbflash -s 0 card.rom. ⚠ It has to be the ROM from the exact card you're overclocking.

2. Load it and tune. Pick a preset or drag the sliders. The Pending Changes box shows exactly what will change before anything happens.

3. Click "Apply on this PC." Approve the admin prompt. Your card restarts with its new limits. (Prefer doing it by hand? "Export files" gives you a .reg, a PowerShell script, and a Linux pp_table.)

🔥 Read this before you chase 2350 MHz

These are passive server cards. No fan. They're built for server chassis that blast air straight through the heatsink.

2350 MHz on a V620 is NOT possible without proper cooling. In a normal case, the card hits its hotspot limit and throttles or crashes long before it gets there.

You need:

  • A blower or fan shroud forcing air through the card
  • Hotspot and memory temps on screen while you test (GPU-Z / HWiNFO)
  • Patience. Not every card hits the same clock. If it crashes, drop 25–50 MHz.

Memory overclockers: unstable VRAM doesn't always crash. It can quietly give wrong AI and compute results. Go up in small steps and test each one.

Robjørn warns you when you push into this territory. Listen to the Viking. 🪓

Compatibility (honest version)

Tested on (v1.0): test firmware built to AMD's own table layouts (the same definitions the Linux driver uses) for all three chip families. Loading, editing, export and SHA-256 checks all pass.

Real card dumps: ⏳ none verified yet. This is a preview release, which is where you come in. 👇

Format Built for Same format, untested
Navi 21 V620 RX 6800 / 6800 XT / 6900 XT, Pro W6800
Navi 12 V520, V540 (tune each GPU) RX 5500 to 5700 XT, Pro W5700
Vega 10 V320 V340, MI25, Vega 56/64, WX 9100

Not supported: Vega 20 (MI50 / Radeon VII), Navi 24, RDNA 3 (V710, RX 7000), RDNA 4, Polaris and older.

🙋 Help make it verified

If Robjørn runs well on your card, comment with:

  • Card model
  • BIOS part number
  • Table SHA-256 (from the app's Compatibility tab)

Every confirmed firmware gets added to the built-in verified list and shows ✓ Verified for everyone after you.

Download

-------> DOWNLOAD HERE

Verify it before you run it: certutil -hashfile Robjorn.exe SHA256 It should match: a2b914f628faafc85b5ca194f1b4da915b09594cb44a127a5cefdb3cd3ee0292

Windows SmartScreen may warn you the first time because the exe isn't code-signed. Source code is included in the zip.

Undo anytime

Click Remove overclock, or run apply_windows.ps1 -Remove as admin. On Linux, a reboot brings back stock.

Overclocking can make systems unstable and shorten hardware life. Robjørn never flashes your card. It only changes the power table the driver loads, but you use it at your own risk. Built-in limits (1250 mV, 1.5× stock power, 110 °C) stay on unless you unlock extended ranges.

Take from the datacenter. Give to the homelab. 🪓


r/ROCm • • 21h ago

Stay tuned for my new overclocker for GPUs, Windows, this card, and others.

Thumbnail
1 Upvotes

r/ROCm • • 1d ago

How to Underclock or Undervolt AMD V620 32GB?

Post image
3 Upvotes

r/ROCm • • 1d ago

[Help Wanted] Testing GPU detection & TFLOPS benchmark paths (NVIDIA/AMD) for my from-scratch SLM trainer

0 Upvotes

Hi guys,

I’m a 15-year-old developer building SLM-CREATOR—a from-scratch, hardware-aware trainer for small, hyperspecialized text models (capped at 100M parameters) in Python and Rust. Instead of using high-level wrappers or fine-tuning existing models, I’m hand-coding the entire pipeline (detection, sizing, BPE tokenizer, and training loop) to learn how everything works under the hood.

Project Link: github.com
Tracking Issue (#1): github.com

Where I need help:
I’m developing this on a CPU-only machine right now, so all of my GPU detection and benchmarking code has only been tested against mock CLI outputs. Since r/llmdevs folks run all sorts of local setups—multi-GPU rigs, hybrid laptop GPUs, and ROCm environments—I’d hugely appreciate it if anyone with an NVIDIA or AMD GPU could run architecture.py and see if it breaks or misidentifies anything on real hardware.

What the script is doing under the hood:

  • Vendor CLI Parsing: Detects NVIDIA (nvidia-smi), modern AMD (amd-smi), or legacy AMD (rocm-smi/rocminfo) to extract architecture, total VRAM (MiB), device index, and GPU name.
  • PyTorch Index Mapping: Sets CUDA_DEVICE_ORDER=PCI_BUS_ID before importing torch and matches vendor tool names against torch.cuda.get_device_name() so multi-GPU systems don't target the wrong device.
  • Matmul TFLOPS Benchmark: Runs a 12-second thermal soak (to get past laptop boost clocks), then takes the median of 10 runs of 20 4096x4096 matmuls. It checks torch.cuda.is_bf16_supported(including_emulation=False) (or compute capability >= 8.0) so it benchmarks native bf16 and falls back to fp32 rather than getting fooled by slow software-emulated bf16.
  • Deterministic Sizing: Pipes VRAM, training hours, TFLOPS, and token counts into a Rust binary (sizer_core) to compute the max parameter count across memory, Chinchilla compute-time, and data limits.

How to test it (takes ~2 mins):

  1. git clone https://github.com
  2. Set up a venv and install requirements.txt (with your GPU's PyTorch build).
  3. Build the Rust sizer: cd sizer_core && cargo build --release
  4. Run python architecture.py from the root folder.
  5. Paste your GPU model, OS, and the terminal output (or any tracebacks) in Issue #1 or in the comments here.

I'd love any feedback on CLI parsing edge cases, ROCm quirks, or how the thermal soak behaves on your card!


r/ROCm • • 1d ago

Used RX 6800 XT vs 6900 XT (XFX Merc) for local LLMs – is the ~$63 premium worth it?

Thumbnail
0 Upvotes

r/ROCm • • 2d ago

Calling all RX7900(gfx1100) class card owners!!!

Thumbnail
10 Upvotes

r/ROCm • • 1d ago

Stay tuned for my new overclocker for GPUs, Windows, this card, and others.

Thumbnail
0 Upvotes

r/ROCm • • 2d ago

Handing two models at once in Radiance

Thumbnail
3 Upvotes

r/ROCm • • 2d ago

MoE Offloading update Exllamav3 on 9070XT

Post image
8 Upvotes

r/ROCm • • 2d ago

Two video models for the R9700, pick yours: LTX-2.5 or MiniMax H3

13 Upvotes

We have two new video packages out for the R9700.

Both run natively from the prompt to the mp4, and both make the sound along with the picture. They're good at different things, so here's the quick version to help you choose.

LTX-2.5 MiniMax H3 v1.2
You give it A prompt, or a prompt plus a first frame A prompt plus up to 9 reference images, or a reference video with its soundtrack
Sizes 1280x704, 1920x1088 864x480, 1344x768
5 s clip on one R9700 18.1 s at 704p, 48.8 s at 1080p 30.4 s with 1 image, 44.3 s with 9 (480p)
Against stock, same card 7.1x and 8.6x faster 2.2x to 3.8x faster
Peak VRAM 27.4 GiB under 27 GiB
First download 38 GB 51 GB

LTX-2.5 if you start from a prompt

Lightricks' 22B audio and video model. Type a prompt, wait 18 seconds, get 5 seconds of 704p video with sound. 1080p takes 48.8 s, and starting from your own first frame adds about 3 s.

The stock pipeline takes 128.6 s and 420.5 s for the same two clips. To be fair to it: it runs BF16, which doesn't fit in 32 GB, so it offloads to the CPU, and that's a good part of the gap.

Quality-wise we're as close to the BF16 output as Lightricks' own FP8 mode at 704p, and 1.1 dB PSNR further away at 1080p.

What this first build can't do yet: portrait sizes, sizes other than the two above, and it needs the clip length up front.

MiniMax H3 if you want your own subjects in it

Give it a photo of your dog, your product or yourself, write "the dog from <Picture 1> runs through a sunlit forest", and that's the dog you get. Up to nine images, or a video whose soundtrack it picks up. Audio is 32 kHz stereo.

v1.2 replaces the ComfyUI package we shipped in October. Everything runs on our HIP kernels now, with no PyTorch, Triton or ComfyUI in the container. The more references you give it, the bigger the win over the stock engine:

5 s clip, Turbo4 Stock v1.2
864x480, 1 image 67.4 s 30.4 s
864x480, 3 images 91.2 s 33.4 s
864x480, 9 images 167.5 s 44.3 s
1344x768, 9 images 278.7 s 114.1 s

Stock here is MiniMax's ComfyUI workflow with its own engine, same prompts, images and seed, running the W4A8 checkpoint from our old package. With nine images it spends over half its time just encoding them, and that's where v1.2 gains the most.

Limits: up to about 9.4 s of video at 768p with nine images, no audio-only references, and the 50-step Base preset is still on v1.1.0.

Try one (or both, one at a time)

They don't fit on the card together, so stop one before you start the other. One R9700, Docker and a ROCm driver:

git clone --depth 1 https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin


# LTX-2.5 (accept the license on the HF page first, then use a token from that account)
export HF_TOKEN=<your token>
./models/LTX-2.5/launch.sh
# second terminal, after READY:
./models/LTX-2.5/run.sh generate "A street drummer plays a fast rhythm on a city sidewalk at dusk while the crowd claps and cheers" --out drummer.mp4


# MiniMax H3 v1.2
PAITON_DETACH=1 ./models/MiniMax-H3/native/launch.sh http
docker logs -f paiton-minimax-h3-v12   # wait for READY
curl -o panda.mp4 -F image=@panda.png \
  -F prompt='The red panda from <Picture 1> walks through a sunlit forest. Audio: birds and a stream.' \
  http://127.0.0.1:30000/v1/videos/generations

Guides, with all the numbers and options:

Licenses: LTX-2.5 is under the LTX-2.x Community License (companies with US$10M+ yearly revenue need a paid license for anything past non-commercial testing and research), MiniMax H3 under the MiniMax H3 Community License.


r/ROCm • • 2d ago

Hierarchos-Native v0.2.0: RDNA3 WMMA, native multimodal training, and a lot of FP32 Vulkan optimization!

4 Upvotes

Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.

I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.

The new v0.2.0 release is now available!

GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

1. Native RDNA3 WMMA / cooperative matrix acceleration

One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.

The new Mixed WMMA execution arm uses:

  • FP16 matrix operands.
  • FP32 matrix accumulation.
  • FP32 master weights and gradients.
  • FP32 AdamW optimizer moments and updates.
  • FP32 normalization, softmax, and loss calculations.

The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.

I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.

Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:

Workload Measured kernel speedup
QKV projection 2.17×
FFN up projection 2.12×
FFN down projection 2.31×
Vocabulary projection 1.83×

These are individual GEMM measurements, not whole-model speedups.

There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.

In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.

WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.

Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.

Documentation: RDNA3 WMMA guide

2. Native multimodal image and text training

Another major development area has been the image-text pipeline.

The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.

Some of the implemented work includes:

  • Native RGB image input and preprocessing.
  • CLIP image encoders and LLaVA projector integration.
  • SigLIP / Gemma 3 vision training.
  • Gemma 4 vision integration.
  • Joint image/text training and encoder-decoder integration.
  • A dedicated DeepSeek V4.1 image-training path.
  • Backpropagation through the image tower and projector.
  • Export, reload, and checkpoint-related validation.

This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.

For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.

The qualification work has also expanded considerably:

  • 44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
  • The Gen9 qualification retains 95 image cases and 166 text regression cases.
  • A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
  • Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.

The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.

These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.

Documentation:

3. A lot of additional FP32 optimization

I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.

This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.

Some examples:

RDNA3 FP32 work

  • Specialized attention forward/backward paths.
  • Image-text attention and projector optimizations.
  • RMSNorm, RoPE, GELU, and other arithmetic specializations.
  • Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
  • More careful matching of CPU reference arithmetic and GPU execution order.

Intel Gen9 work

  • Additional native linear and gradient kernels.
  • Attention and activation specializations.
  • More efficient parameter upload and staging behavior.
  • Continued optimization of DeepSeek V4.1 training and execution.

One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.

On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.

On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.

These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.

4. Preserving the original FP32 path

An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.

I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:

  • 12 alternating ABBA rounds per workload.
  • Five warm-up runs and 30 timed forwards per arm.
  • Separate, hashed baseline and candidate binaries.
  • Exact FP32 logit digest comparisons.

All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.

The worst paired performance ratio was 1.0007.

This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.

5. Getting started

The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.

You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.

Model weights are downloaded separately or supplied through a local model directory.

After extracting the archive, you can run:

.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-info

The final command is especially useful for checking which precision capabilities your GPU actually exposes.

Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

What's next?

There is still a lot I'd like to improve.

In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.

I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.

If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.

Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.

Repository: https://github.com/necat101/Hierarchos-Native

Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.


r/ROCm • • 2d ago

AMD 9070 GPU under volting gains for models ( moss image) and ( minimax h3 int8 )

3 Upvotes

hi i am total GPU noob here this is the first GPU i have ever owned in my life got it just for doing AI... asked deepseek flash and opencode to find the right combo for me to get max generation speed

spec:
AMD 9070 (non XT),
Intel i3 12100
32 GB DDR4

models: moss image, minimax h3 int8 (sol decoder)

best performance stable setting: -110 mv, stock fan speed


r/ROCm • • 3d ago

Initial VuPy release: CuPy-like GPU array library backed by vulkan. any and all feedback is preferred.

Thumbnail
github.com
1 Upvotes

r/ROCm • • 3d ago

Pushed Strata with QFN 3.8 Q4 K_M to 5,600 tps prefill @100k - 100 tps decode w/ 181k context on 2x 7900xtx & 64gb DDR5 in layer split (no tensor parallel) - config inside

Thumbnail
6 Upvotes

r/ROCm • • 2d ago

Radeon AI PRO R9700 (gfx1201) is an overpriced Ferrari shell that doesn't even roll — Stop misleading creators with "32GB VRAM"

0 Upvotes

Radeon AI PRO R9700 (gfx1201) is an overpriced Ferrari shell that doesn't even roll — Stop misleading creators with "32GB VRAM"

I bought the Radeon AI PRO R9700 (32GB) specifically for my YouTube video production workflow, expecting a reliable workstation GPU for ComfyUI image and video generation. Instead, I have spent 6 full days and nights just trying to make it run without crashing—zero actual production work done.
Here is the reality of using the R9700 on Ubuntu right now:

  • Terrible Ubuntu compatibility: The card wouldn't even load properly on the latest Ubuntu release, forcing me to downgrade Ubuntu itself just to get basic detection working.
  • Endless environment hell: After tweaking Python environments, Docker containers, and following community-recommended ROCm versions, the system still crashes 1 to 2 times every single day with Failed to pin framebuffer: -ENOMEM and desktop session freezes.
  • Severe CPU fallback & memory leaks: Instead of utilizing the GPU at 100% like NVIDIA cards do, PyTorch/ROCm constantly leaks memory and dumps heavy workloads onto the CPU (spiking CPU usage to 400%) until the system locks up.
  • Good only for LLM and Audio, terrible for Image & Video: While text generation (LLM) and music generation run fast, image and video workflows are a complete disaster. Especially FP8 models, which produce severe ghosting/artifacts and cause immediate crashes.

Even worse, this issue remains completely unresolved. For real video production, it performs worse and is far less stable than an 8GB NVIDIA GPU. Sure, it can spit out a 1-megapixel clip once, but sustained, long-hour production work is impossible. How can AMD dare call this a "PRO" card for people making a living?
With NVIDIA, at least creators know it's expensive and save up for something that actually works 24/7. AMD should stop baiting buyers with an "attractive price" and "generous 32GB VRAM" when the software is only half-baked. Be honest and tell customers upfront that it's an incomplete product for specific workloads only—so fewer creators suffer the same loss and frustration I did.
Right now, owning this card feels like buying an expensive Ferrari shell that doesn't even roll—nothing more than an overpriced display piece. Who is going to compensate creators for all this lost time and stress?


r/ROCm • • 3d ago

R9700 in a Thunderbolt 5 dock on Strix Halo (MS-S1 MAX + DEG2): now stable on the rear USB4 v2 port – two separate bugs, two fixes, logs included

Thumbnail
7 Upvotes

r/ROCm • • 4d ago

Blender Cycles Optix engine working natively on AMD RX 7800XT via Project GHOST

Thumbnail
gallery
8 Upvotes

Yeah the title pretty much says it its optix working on an amd card natively via software rt planned is vulkan rt addition but hey its optix 8 working on amd even tho render times are a biiiiit slow discord if anyone is intrested: https://discord.gg/zPQucyR8tU