r/ROCm • • 11h ago

Used RX 6800 XT vs 6900 XT (XFX Merc) for local LLMs – is the ~$63 premium worth it?

Thumbnail reddit.com
0 Upvotes

r/ROCm • • 1d ago

Calling all RX7900(gfx1100) class card owners!!!

Thumbnail
reddit.com
10 Upvotes

r/ROCm • • 14h ago

Stay tuned for my new overclocker for GPUs, Windows, this card, and others.

Thumbnail reddit.com
0 Upvotes

r/ROCm • • 22h ago

Handing two models at once in Radiance

Thumbnail
reddit.com
3 Upvotes

r/ROCm • • 1d ago

MoE Offloading update Exllamav3 on 9070XT

Post image
7 Upvotes

r/ROCm • • 1d ago

Two video models for the R9700, pick yours: LTX-2.5 or MiniMax H3

14 Upvotes

We have two new video packages out for the R9700.

Both run natively from the prompt to the mp4, and both make the sound along with the picture. They're good at different things, so here's the quick version to help you choose.

LTX-2.5MiniMax H3 v1.2
You give itA prompt, or a prompt plus a first frameA prompt plus up to 9 reference images, or a reference video with its soundtrack
Sizes1280x704, 1920x1088864x480, 1344x768
5 s clip on one R970018.1 s at 704p, 48.8 s at 1080p30.4 s with 1 image, 44.3 s with 9 (480p)
Against stock, same card7.1x and 8.6x faster2.2x to 3.8x faster
Peak VRAM27.4 GiBunder 27 GiB
First download38 GB51 GB

LTX-2.5 if you start from a prompt

Lightricks' 22B audio and video model. Type a prompt, wait 18 seconds, get 5 seconds of 704p video with sound. 1080p takes 48.8 s, and starting from your own first frame adds about 3 s.

The stock pipeline takes 128.6 s and 420.5 s for the same two clips. To be fair to it: it runs BF16, which doesn't fit in 32 GB, so it offloads to the CPU, and that's a good part of the gap.

Quality-wise we're as close to the BF16 output as Lightricks' own FP8 mode at 704p, and 1.1 dB PSNR further away at 1080p.

What this first build can't do yet: portrait sizes, sizes other than the two above, and it needs the clip length up front.

MiniMax H3 if you want your own subjects in it

Give it a photo of your dog, your product or yourself, write "the dog from <Picture 1> runs through a sunlit forest", and that's the dog you get. Up to nine images, or a video whose soundtrack it picks up. Audio is 32 kHz stereo.

v1.2 replaces the ComfyUI package we shipped in October. Everything runs on our HIP kernels now, with no PyTorch, Triton or ComfyUI in the container. The more references you give it, the bigger the win over the stock engine:

5 s clip, Turbo4Stockv1.2
864x480, 1 image67.4 s30.4 s
864x480, 3 images91.2 s33.4 s
864x480, 9 images167.5 s44.3 s
1344x768, 9 images278.7 s114.1 s

Stock here is MiniMax's ComfyUI workflow with its own engine, same prompts, images and seed, running the W4A8 checkpoint from our old package. With nine images it spends over half its time just encoding them, and that's where v1.2 gains the most.

Limits: up to about 9.4 s of video at 768p with nine images, no audio-only references, and the 50-step Base preset is still on v1.1.0.

Try one (or both, one at a time)

They don't fit on the card together, so stop one before you start the other. One R9700, Docker and a ROCm driver:

git clone --depth 1 https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin


# LTX-2.5 (accept the license on the HF page first, then use a token from that account)
export HF_TOKEN=<your token>
./models/LTX-2.5/launch.sh
# second terminal, after READY:
./models/LTX-2.5/run.sh generate "A street drummer plays a fast rhythm on a city sidewalk at dusk while the crowd claps and cheers" --out drummer.mp4


# MiniMax H3 v1.2
PAITON_DETACH=1 ./models/MiniMax-H3/native/launch.sh http
docker logs -f paiton-minimax-h3-v12   # wait for READY
curl -o panda.mp4 -F image=@panda.png \
  -F prompt='The red panda from <Picture 1> walks through a sunlit forest. Audio: birds and a stream.' \
  http://127.0.0.1:30000/v1/videos/generations

Guides, with all the numbers and options:

Licenses: LTX-2.5 is under the LTX-2.x Community License (companies with US$10M+ yearly revenue need a paid license for anything past non-commercial testing and research), MiniMax H3 under the MiniMax H3 Community License.


r/ROCm • • 1d ago

Hierarchos-Native v0.2.0: RDNA3 WMMA, native multimodal training, and a lot of FP32 Vulkan optimization!

4 Upvotes

Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.

I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.

The new v0.2.0 release is now available!

GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

1. Native RDNA3 WMMA / cooperative matrix acceleration

One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.

The new Mixed WMMA execution arm uses:

  • FP16 matrix operands.
  • FP32 matrix accumulation.
  • FP32 master weights and gradients.
  • FP32 AdamW optimizer moments and updates.
  • FP32 normalization, softmax, and loss calculations.

The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.

I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.

Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:

WorkloadMeasured kernel speedup
QKV projection2.17×
FFN up projection2.12×
FFN down projection2.31×
Vocabulary projection1.83×

These are individual GEMM measurements, not whole-model speedups.

There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.

In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.

WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.

Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.

Documentation: RDNA3 WMMA guide

2. Native multimodal image and text training

Another major development area has been the image-text pipeline.

The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.

Some of the implemented work includes:

  • Native RGB image input and preprocessing.
  • CLIP image encoders and LLaVA projector integration.
  • SigLIP / Gemma 3 vision training.
  • Gemma 4 vision integration.
  • Joint image/text training and encoder-decoder integration.
  • A dedicated DeepSeek V4.1 image-training path.
  • Backpropagation through the image tower and projector.
  • Export, reload, and checkpoint-related validation.

This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.

For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.

The qualification work has also expanded considerably:

  • 44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
  • The Gen9 qualification retains 95 image cases and 166 text regression cases.
  • A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
  • Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.

The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.

These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.

Documentation:

3. A lot of additional FP32 optimization

I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.

This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.

Some examples:

RDNA3 FP32 work

  • Specialized attention forward/backward paths.
  • Image-text attention and projector optimizations.
  • RMSNorm, RoPE, GELU, and other arithmetic specializations.
  • Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
  • More careful matching of CPU reference arithmetic and GPU execution order.

Intel Gen9 work

  • Additional native linear and gradient kernels.
  • Attention and activation specializations.
  • More efficient parameter upload and staging behavior.
  • Continued optimization of DeepSeek V4.1 training and execution.

One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.

On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.

On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.

These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.

4. Preserving the original FP32 path

An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.

I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:

  • 12 alternating ABBA rounds per workload.
  • Five warm-up runs and 30 timed forwards per arm.
  • Separate, hashed baseline and candidate binaries.
  • Exact FP32 logit digest comparisons.

All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.

The worst paired performance ratio was 1.0007.

This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.

5. Getting started

The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.

You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.

Model weights are downloaded separately or supplied through a local model directory.

After extracting the archive, you can run:

.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-info

The final command is especially useful for checking which precision capabilities your GPU actually exposes.

Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

What's next?

There is still a lot I'd like to improve.

In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.

I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.

If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.

Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.

Repository: https://github.com/necat101/Hierarchos-Native

Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.


r/ROCm • • 1d ago

AMD 9070 GPU under volting gains for models ( moss image) and ( minimax h3 int8 )

3 Upvotes

hi i am total GPU noob here this is the first GPU i have ever owned in my life got it just for doing AI... asked deepseek flash and opencode to find the right combo for me to get max generation speed

spec:
AMD 9070 (non XT),
Intel i3 12100
32 GB DDR4

models: moss image, minimax h3 int8 (sol decoder)

best performance stable setting: -110 mv, stock fan speed


r/ROCm • • 1d ago

Initial VuPy release: CuPy-like GPU array library backed by vulkan. any and all feedback is preferred.

Thumbnail
github.com
1 Upvotes

r/ROCm • • 2d ago

Pushed Strata with QFN 3.8 Q4 K_M to 5,600 tps prefill @100k - 100 tps decode w/ 181k context on 2x 7900xtx & 64gb DDR5 in layer split (no tensor parallel) - config inside

Thumbnail
reddit.com
6 Upvotes

r/ROCm • • 1d ago

Radeon AI PRO R9700 (gfx1201) is an overpriced Ferrari shell that doesn't even roll — Stop misleading creators with "32GB VRAM"

0 Upvotes

Radeon AI PRO R9700 (gfx1201) is an overpriced Ferrari shell that doesn't even roll — Stop misleading creators with "32GB VRAM"

I bought the Radeon AI PRO R9700 (32GB) specifically for my YouTube video production workflow, expecting a reliable workstation GPU for ComfyUI image and video generation. Instead, I have spent 6 full days and nights just trying to make it run without crashing—zero actual production work done.
Here is the reality of using the R9700 on Ubuntu right now:

  • Terrible Ubuntu compatibility: The card wouldn't even load properly on the latest Ubuntu release, forcing me to downgrade Ubuntu itself just to get basic detection working.
  • Endless environment hell: After tweaking Python environments, Docker containers, and following community-recommended ROCm versions, the system still crashes 1 to 2 times every single day with Failed to pin framebuffer: -ENOMEM and desktop session freezes.
  • Severe CPU fallback & memory leaks: Instead of utilizing the GPU at 100% like NVIDIA cards do, PyTorch/ROCm constantly leaks memory and dumps heavy workloads onto the CPU (spiking CPU usage to 400%) until the system locks up.
  • Good only for LLM and Audio, terrible for Image & Video: While text generation (LLM) and music generation run fast, image and video workflows are a complete disaster. Especially FP8 models, which produce severe ghosting/artifacts and cause immediate crashes.

Even worse, this issue remains completely unresolved. For real video production, it performs worse and is far less stable than an 8GB NVIDIA GPU. Sure, it can spit out a 1-megapixel clip once, but sustained, long-hour production work is impossible. How can AMD dare call this a "PRO" card for people making a living?
With NVIDIA, at least creators know it's expensive and save up for something that actually works 24/7. AMD should stop baiting buyers with an "attractive price" and "generous 32GB VRAM" when the software is only half-baked. Be honest and tell customers upfront that it's an incomplete product for specific workloads only—so fewer creators suffer the same loss and frustration I did.
Right now, owning this card feels like buying an expensive Ferrari shell that doesn't even roll—nothing more than an overpriced display piece. Who is going to compensate creators for all this lost time and stress?


r/ROCm • • 2d ago

R9700 in a Thunderbolt 5 dock on Strix Halo (MS-S1 MAX + DEG2): now stable on the rear USB4 v2 port – two separate bugs, two fixes, logs included

Thumbnail
reddit.com
6 Upvotes

r/ROCm • • 2d ago

Blender Cycles Optix engine working natively on AMD RX 7800XT via Project GHOST

Thumbnail
gallery
8 Upvotes

Yeah the title pretty much says it its optix working on an amd card natively via software rt planned is vulkan rt addition but hey its optix 8 working on amd even tho render times are a biiiiit slow discord if anyone is intrested: https://discord.gg/zPQucyR8tU


r/ROCm • • 3d ago

Strix halo 2.4x faster sd3.5 large turbo (halodiffuse v1.1)

6 Upvotes

in my last post i got tons of reviews about difficultly in instructions etc so i released a newer version with significantly simpler installation process and it no longer requires a separate python script as well as seeding and persistent weight support and detailed cli instructions https://github.com/Legendofzaday/halodiffuse i plan on adding batching support and would like reviews on what batch size do you all commonly use such as 1,2,3,4,5,6,7,8 etc.


r/ROCm • • 2d ago

vulkan-py easymode. all the boilerplate is in easy.py. vulkan-py fork https://github.com/billyblackburn/vulkan-py-easy/tree/main

Thumbnail reddit.com
1 Upvotes

r/ROCm • • 3d ago

RX 9070 XT + ROCm 7.2.1: torch.cuda.is_available() crashes and prevents OpenFill/ComfyUI from starting

Thumbnail reddit.com
2 Upvotes

r/ROCm • • 4d ago

My benchmark for Qwen3.8-flash-next-q4 on RT 7900 XTX

26 Upvotes

r/ROCm • • 4d ago

Update: elastic KV cache on 2× R9700, now running unattended (graphs + code)

Post image
8 Upvotes

Since my last update, the cache resizing runs on its own. After 10 quiet minutes, the picture/video service unloads its models and asks SGLang to grow its KV cache. SGLang picks the largest size that fits on both cards, about 380k extra tokens. When anyone asks for a picture or clip, it gives the memory back first.

One-hour soak, live service, untouched: 5 automatic cycles, each a real picture request through the normal endpoint.

  • Shrink: 1.02–1.07 s every time, almost all of it waiting for the kernel to actually return the VRAM.
  • Pictures: 63.9–64.9 s, the same as before the elastic cache existed (64–65 s warm).
  • Chat: 85.9–86.2 tok/s.
  • 0 GPU faults, 0 restarts.

Other numbers (graphs attached):

  • A shrink while chats are running copies their data out of the borrowed space first: 0.06–0.11 s, with a worst case of 170k tokens moved in 19 ms.
  • Exact outputs across shrinks mid-reply, mid-prefill and with one of two parallel replies just finished. Cached data intact after 70 grow/shrink cycles.
  • Decode speed unchanged at 86–88 tok/s.

One fix the GPU tests forced: my first version grew to the full reserved size, which never fits once the image models only partly unload. It now sizes itself to the free memory on the tightest card.

Code and design notes: Drgr33nSenior/Spry.ai-rdna4-serving PR #38 (https://github.com/Drgr33nSenior/Spry.ai-rdna4-serving/pull/38). It's a patch series (0031–0033) on top of my other gfx1201 patches for a pinned SGLang commit, so it won't apply to stock SGLang yet. The design record (https://github.com/Drgr33nSenior/Spry.ai-rdna4-serving/blob/feat/elastic-kv-tail-20261114/docs/ELASTIC-KV-TAIL-20261114.md) covers how it works, the control API, an independent review's findings and all the GPU results.

If you try this on RDNA, same-address remapping needs the amdgpu "fix TLB invalidation fence handling" (https://lore.kernel.org/amd-gfx/20261008124232.23825-1-christian.koenig@amd.com/) patch. It's queued for 7.4-rc3; without it, remaps corrupt other live memory. My server runs a 20 ms remap self-test at startup and won't enable the feature if it fails.

Next: a week of real family use, then probably an RFC to SGLang. Its VMM arena is CUDA-only and grows once at startup, and the closest open PR (#48106 (https://github.com/sgl-project/sglang/pull/48106)) only trades memory between two internal pools. Credit to kvcached (https://github.com/ovg-project/kvcached) for the idea.

Still keen to hear from anyone doing HIP VMM work on RDNA3/4.


r/ROCm • • 4d ago

Elastic KV cache on RDNA4 (2× R9700): anyone else tried kvcached or HIP VMM ballooning on ROCm?

13 Upvotes

TL;DR: It works :)

FOLLOW UP HERE: https://www.reddit.com/r/ROCm/comments/1xw8ttw/update_elastic_kv_cache_on_2_r9700_now_running/

Hey ROCm! As I post here daily atm :) You all probably know that I run a home AI server on two Radeon AI PRO R9700s. SGLang serves Qwen3.8-27B, and two ComfyUI instances on the same cards handle pictures (Qwen-Image) and short videos (Wan 2.2).

The goal: pictures and video are occasional, but their VRAM share sits idle most of the day. I'd like the LLM's KV cache to borrow that memory when the generators are idle and hand it back before a job starts, without ever cutting off a chat. The idea is the same as kvcached (https://github.com/ovg-project/kvcached).

Why not kvcached itself:

  • Version gates: its SGLang patches stop below 0.5.21, and it needs Python below 3.14, so on my stack it would silently do nothing.
  • Tensor parallelism: it doesn't support shrinking with TP>1.
  • Layout: its ROCm default layout conflicts with speculative decoding, and the alternative layout breaks my contiguous verify-attention kernel.
  • Shared zero page: it maps one zero page behind every unbacked slot, and AMD's HIP docs warn that RDNA can return wrong results when two virtual addresses map the same physical page.

What I'm building instead: a small "elastic tail" inside SGLang's own KV arena, using HIP virtual memory.

  • A fixed base pool stays exactly as today.
  • Each KV buffer reserves address space up to a large-mode size and maps or unmaps one tail extent at the end.
  • Both TP ranks resize at the same scheduler step.
  • Before a shrink, any running request's KV is migrated from the tail into the base.
  • A shrink only reports done once the card actually shows the memory free.
  • The tail is extra prefix-cache room only; per-request context doesn't change.

Finding so far: on kernel 7.3.3, unmapping and remapping at the same address silently corrupted other live memory (TheRock#8744, around 15 billion bad words per run). The amdgpu patch "drm/amdgpu: fix TLB invalidation fence handling" fixes it. On 7.3.4 with that patch, my probe was clean: 1,000 rounds per card of 256 MiB–4 GiB extents remapped at one address beside an 8 GiB live base, with zero corrupt words. One quirk: in about a third of rounds, mem_info_vram_used still showed the freed extent right after hipMemRelease plus sync. It always came back, so it's freed lazily rather than leaked, but "released" can't be assumed immediately.

Next is a GPU qualification run, then a one-week trial behind a flag.

I have some questions for the community:

  1. Has anyone run kvcached, or any HIP VMM map/unmap pattern, on RDNA4?
  2. Did you see the delayed VRAM release, or anything like the corruption bug?
  3. Any other gotchas with torch tensors over HIP-mapped memory, or with Triton kernels and HIP graphs reading it?

Once again, thanks for any assistance in advance. I'll come share my work as always once complete.


r/ROCm • • 4d ago

Running MiniMax H3 on an RX 9070 XT + 32GB DDR5 RAM

3 Upvotes

I offloaded the extra layers to system memory so the model fits across 16GB of VRAM and 32GB of DDR5 RAM (⁠[BF16 / quant]⁠).

Everything needed to reproduce the run on RDNA 4, including the launch flags and configs, is in this repo:
https://github.com/0c33/minimax-h3-rdna4

Let me know if you test it on another RDNA 4 card or get better token speeds with different offload settings.


r/ROCm • • 4d ago

Radeon Pro V620 on AMD Pro/Adrenalin drivers (Windows) with all 72 CUs, no permanent VBIOS flash + free helper kit

Thumbnail
reddit.com
3 Upvotes

r/ROCm • • 5d ago

Best model (and way) to run on a RX 9070 XT?

11 Upvotes

Hello!

I've been using LLMs for quite a while now, I've used Gemma 4, Qwen 3(.5), and similar smaller-sized models as I didn't have a GPU for a long time.

Until now! I recently got my hands on a 16GB RX 9070 XT. I've been running Qwen 3.8 27B (Q3) at somewhat comfortable speeds, getting around 60 t/s (output) when using a fork of llama.cpp made for the RX 9070 XT. Smaller models cap at 50t/s, no matter their size.

However, I want to know if you've got any tips! Should I check out Strata? (I've got 32GBs of DDR4 ECC RAM) Should I use another model? Is Unsloth Studio (my daily driver) the wrong choice?

Thank you.


r/ROCm • • 5d ago

Strata EXL3 - UPDATED +15-20% tk/s - run Q4 at Q3 speeds - MiMo v2.6 flash support (exp)

40 Upvotes

A quick update on my Strata EXL3 fork for AMD.

I've been working on the HIP kernels and memory management, and decode performance is now roughly 10-20% higher than the previous version, depending on the quant and workload. Some cases improved by almost 24%.

The other news is that I managed to get MiMo V2.6 Flash running at around 25-30 tokens/s with 2 cards.. More on that below.

First, why EXL3?

I think this is worth repeating because people tend to compare quantizations just by looking at the number of bits.

EXL3 is NOT equivalent to regular GGUF quants at the same bpw.

For example, 3.05 bpw EXL3 is roughly in the same quality range as a regular 4-bit quant (like IQ4_XS).
3.05 bpw quant: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/tree/3.05bpw_h5_ng5
4.05 bpw quant: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/tree/4.05bpw_h6_ng6

So the interesting part is that you can get something close to 4-bit quality, while running at speeds closer to what you'd expect from a 3-bit quant. And with the latest optimizations, I think this fork is becoming VERY competitive. The 4.05 bpw version is also pretty fast now, even on a single GPU.

WHAT'S NEW:

  • Faster HIP kernels and better memory management: around 10-20% faster decode on average compared to the previous version.
  • Optional local expert repacking: you can now repack the quantized expert weights into a layout that is more efficient for CPU access from system RAM. This helps when using CPU assistance for inference, since the CPU can process experts more efficiently without having to deal with the original GPU-oriented layout. It doesn't change the quantization itself.
  • The repacking is optional, and CPU assistance is not always beneficial. It depends on your hardware and GPU configuration. With two GPUs, for example, you're generally better off without it. Please read the README before enabling it.
  • Experimental MiMo V2.6 Flash support: currently running at around 25-30 tok/s using the 2.20 bpw MOPD quant. This is still experimental, so don't expect everything to work perfectly yet.

NOTES ON REPACKING:

Repacking can be done in two ways: by creating a file on disk once, that will be used to load the model, or directly to RAM everytime. directly to RAM adds a couple of minutes to the launch but uses no extra space. your choice.

BENCHMARKS:

My hardware is the same as in the previous post:

Ryzen 9 7950X, 96GB RAM, 2x Radeon RX 7900 XTX 24GB
Linux, HIP/ROCm core-7.14
Qwen3.8-Flash-Next, MTP drafter, spec 4, 64K context

BENCHMARKS:

Results with one or two RX 7900 XTX GPUs, using warm decode runs:

EXL3 quantGGUF equivalent qualityGPUsCPU assistanceTestPreviousUpdatedImprovement
3.05 bpw≈ IQ4_K_XS2NoCODE78.2787.33+11.6%
3.05 bpw≈ IQ4_K_XS2NoPROSE66.0170.35+6.6%
4.05 bpw≈ Q5_K_M2NoCODE61.8876.67+23.9%
4.05 bpw≈ Q5_K_M2NoPROSE57.7867.57+16.9%
4.05 bpw≈ Q5_K_M1No → YesCODE54.8059.30+8.2%
4.05 bpw≈ Q5_K_M1No → YesPROSE44.8051.60+15.2%

All speeds are in tokens/s, GGUF quality equivalents are approximate... dont' hate me :)

32K prefill is basically unchanged: around 1,500 tok/s with two GPUs and 1,360 tok/s with one GPU.

I think the single-GPU results are particularly interesting. We're getting 55-60 tok/s in CODE with 4.05 bpw EXL3, at roughly Q5_K_M quality, on a single 7900 XTX with CPU assistance.

And the 3.05 bpw quant is now reaching 87 tok/s with two GPUs, while offering considerably better quality than you'd normally expect from a 3-bit quant.

ABOUT MiMo:

In my previous post I mentioned that I wanted to support larger models, particularly the 300B class models.

Well, MiMo V2.6 Flash is now running :) I'm waiting for that ERNIE 5.5 to drop.I'm getting around 25-30 tok/s with the EXL3 2.20 bpw MOPD quant.It's still experimental and currently only tested with that specific quant. CPU assistance is recommended for MiMo, even with two GPUs.I'll probably spend some more time optimizing it, but the fact that it's already running at usable speeds on consumer AMD hardware is pretty interesting.

this is the quant: https://huggingface.co/vcruz305/MiMo-V2.6-Flash-MOPD-EXL3/tree/main/2.20bpw

REPO:

https://github.com/nasone32/Strata-EXL3-fork

If you're setting it up with an AI agent, point it to docs/AI_SETUP_EXL3_AMD.md.

The README has the benchmarks, configuration options and instructions for CPU repacking.As before, this is developed and tested on RDNA3 / Linux / ROCm. Other AMD architectures need community testing. And the long-term goal is still to get this merged upstream into Strata, if the maintainers are interested.

Credits to the original Strata developers, Ryutaku for the EXL3 integration, and Turboderp/ExLlamaV3 for the quantization and code I've been porting.

If anyone tests this on different AMD hardware, please report your results. Especially interested in seeing how it behaves on RDNA4.


r/ROCm • • 5d ago

Video gen speed 7900xtx vs 5060ti?

5 Upvotes

Hey guys, i am thinking of buying new gpu 7900xtx vs 5060ti. But i did not find more convincing reviews of ai video gen. I found some where users said amd is far slower then nvidia and does not support windows as well.

Id appreciate if anyone using amd for video gen and guide me.