r/LocalLLM • • 1d ago

Other Qwen 4 is not out

165 Upvotes

Please stop posting about Qwen4's endless PR drip until it's actually available.


r/LocalLLM • • 3h ago

Discussion Why is above 35B and below 70B still such a dead zone for local models?

85 Upvotes

Title basically.

One could argue the “dead zone” goes up to 120B now. After Qwen3.8 27B all the way up to 120, there’s very little advancement in this area unless I’m missing something.

Nemotron has some models in between, some of the older Qwens had stuff in the 30-50s I think, but other than that, it doesn’t even seem like there’s any real progress either.

I do understand most people either have under 32GB VRAM or they just jump to the 128 and beyond tier but more choice is better isn’t it?

If we could get solid 50-70B models, they could be quantized down to 5-6 bit and be better than 3.8 27B in theory and still run on 36-64GB setups?

Am I just way off and incorrect in my train of thought?


r/LocalLLM • • 1h ago

Project SSD MoE at high speed - The wait is over!

37 Upvotes

The wait is over!

Introducing Run My AI & TokenWing.

An inference engine that lets you offload over 60% of routed experts of mixture of experts models into SSD, yet achieve very low streaming latency...

And a GUI wrapper aimed at providing the best local AI experience.

Benchmarks: Ling 3.0 Flash 120B MXFP4 running on 12Gb VRAM + 32GB DDR5 RAM laptop at 45-50 tok/s decode and 900 tok/s prefill

Qwen 3.8 Flash Next 3.5bpw running on 8GB VRAM + 32GB DDR5 RAM laptop at 35-40 tok/s and 600-700 tok/s prefill

Flint 40B UD-Q4_K_XL running on a laptop with 24GB DDR4 RAM + 4GB VRAM at 30 tok/s decode and 400 tok/s prefill

All without MTP! And benchmarks are by users*

check the website out: https://runmyai.org

Also, the autotuner is still work in progress...

A common question, does it degrade SSD?

Nope! it doesn't... There's a thing called TBW rating. Well, your SSD degrades only when it writes. And the engine only reads....


r/LocalLLM • • 8h ago

LoRA Absolutely night and day for Qwen Coding

Thumbnail
huggingface.co
102 Upvotes

Basically works like a harness fine tune…. And does what it says on the tin.

Seriously, for quantized qwen 3.8 makes it much much more usable.


r/LocalLLM • • 35m ago

Model Stuck on 12GB VRAM but need 27B-level agent reasoning? SenseNova-Loop-9B loops its layers to match Qwen3.8-27B, official GGUF & AWQ out now

Thumbnail
gallery
12 Upvotes

SenseNova released the full version of Loop 9B on November 17 and posted the AWQ and GGUF download links today
https://www.reddit.com/r/LocalLLM/comments/1xzf343/sensenovaloop9b_matches_qwen3827b_on/
Loop 9B went from 14.33 in preview to 36.67 on AutomationBench, tying Qwen3.8-27B on that benchmark in the team’s evaluation.
What caught my attention was how the model uses its 9B parameters. Each token goes through four layers, then a block of 24, then the same 24 layers again, and finally the last four. On the second pass, the block works on the output of the first pass. That gives you 4 + 24 + 24 + 4 = 56 layer computations while storing just 32 sets of layer weights. You pay for the second pass in compute, but you only have to load those weights once
Inside the loop, the model carries two residual streams. Each layer learns how to mix information from the two streams and how to write its result back. The paper calls this two-stream mHC. There’s a separate design choice that matters for long prompts: in each group of four layers, three use linear attention and one uses full attention. Only that fourth layer needs a full attention KV cache that grows with the prompt.

Architecture details: https://github.com/OpenSenseNova/SenseNova-Loop

For local inference, a 12 GB card looks like a reasonable place to try 64K context (for example RTX5070). With 4-bit weights, the team estimates about 9 GiB at 64K with a 4-bit KV cache, or 10 GiB with an 8-bit KV cache. Keeping the cache at 8-bit brings the estimate to 12 GiB at 128K and 15 GiB at 256K. That last configuration leaves little headroom on a 16 GB card. Actual peak usage depends on the engine and how much VRAM is free.


r/LocalLLM • • 6h ago

Model SenseNova-Loop-9B matches Qwen3.8-27B on AutomationBench, and it now has official GGUFs (Q4_K_M is 5.8 GB)

Post image
34 Upvotes

SenseNova-Loop-9B came out this week, and the official GGUFs are now up.

It's only about 9B parameters but gets more depth out of them. Each token gets 56 layers of processing while the weights only hold 32, because the 24 in the middle are used twice. According to the chart in their README (attached), it sits at 36.67 on AutomationBench, the exact score listed for Qwen3.8-27B. Those are their own numbers, so let's see what independent tests say. Their repo with the full README is https://github.com/OpenSenseNova/SenseNova-Loop

As for the GGUFs, Q4_K_M is 5.8 GB, Q5_K_M 6.7 GB and Q8_0 9.8 GB. It's a new architecture, so LM Studio and stock llama.cpp can't load them at the moment. Instead you build OpenSenseNova/llama.cpp, the official fork. The exact commit and the command to run are on the GGUF model card: https://huggingface.co/sensenova/SenseNova-Loop-9B-GGUF

It takes text and images, and the license is Apache 2.0. If you serve from a GPU box there's a 4-bit AWQ too: https://huggingface.co/sensenova/SenseNova-Loop-9B-AWQ


r/LocalLLM • • 6h ago

Question Could uncensored open weight models in 2027-2029 force us to build something like the cyberpunk "Blackwall"?

31 Upvotes

It might sound like pure scifi, but just imagine local models reaching the reasoning level of something like Fable 5.2 (which will probably happen in about a year or two [it's hard to predict]). Now picture 100,000 people downloading an uncensored, Fable 5.2 level model onto their home pcs and running it in a 24/7 agentic loop with a prompt like "go make money for me." It’s pretty easy to imagine the kind of chaos that could unleash on the internet.


r/LocalLLM • • 3h ago

Question Hy4 flash is now open weight: what local setup makes it usable?

Post image
17 Upvotes

The Instruct, FP8 and base checkpoints are now publicly available.

Hy4 Flash has 224B total parameters (12B active), a 1M context window, and native text and image input. GGUF versions are also available under the Tencent Hy License.

The FP8 weights alone require well over 128GB, before runtime and KV cache overhead. Would a quantized GGUF setup with CPU/SSD offloading be practical at a useful context length, or is this still multi GPU territory?


r/LocalLLM • • 1h ago

Project Nihil IDE, free alpha, a hardware simulation lab that supports Local LLMs and 100% offline mode out of the box

Thumbnail
youtube.com
8 Upvotes

r/LocalLLM • • 9h ago

News A Danish AI lab has just launched one of the fastest LLMs for phones to date

Thumbnail
sevren.ai
41 Upvotes

Just came across a LinkedIn post mentioning this company, Sevren which launched a very fast and intelligent LLM for phones, they call it Embla. Apparently they have decided to trade a few benchmark points for speed. I'm linking their blog post about the model here and I think it’s impressive.

1,77s speed-to-intelligence is amazing. Imagine how efficient in terms of battery/energy consumption it should be. Of course there are limitations and they are clearly stated by the company and the articles about it, but seems like they can achieve much more with a couple of iterations on top of it.

I believe that soon this kind of model will be good enough to answer >90% of questions a normal user would ask daily to a LLM like chatGPT.


r/LocalLLM • • 4h ago

Project Pocket LLM 1.7: offline Gemma 4 on Android, now with long chats that don't break

11 Upvotes

Pocket LLM 1.7 is out, the update to my open-source Android app for running LLMs fully on your phone.

New in 1.7:

- Long chats are auto-compacted: older messages get summarized instead of dropped

- Context size fits your phone's memory, so big chats stop crashing

- Fresh design with light/dark themes and accent colors

- LAN server fixes

What it does:

- Offline chat with Gemma 4, Qwen3, Qwen2.5, DeepSeek R1, or your own .litertlm model

- PDFs (with OCR), images, audio and offline voice dictation

- LAN server: browser chat and an OpenAI-compatible API for your PC

No account, no tracking. On 1.6? Install the new APK over it, and your chats stay.

GitHub (APK + source, MIT): https://github.com/dineshsoudagar/local-llms-on-android

Google Play (GGUF models + optional Pro tools): https://play.google.com/store/apps/details?id=io.github.dineshsoudagar.pocketllm

Feedback welcome!


r/LocalLLM • • 3h ago

Discussion 3 x AI 9700 build - 96gb VRAM

Thumbnail gallery
7 Upvotes

Any recommendation on what else I can try on this set up?
Love this as a hobby! Just fun building!


r/LocalLLM • • 47m ago

Discussion Thread for recommended LLM setups for common hardware

4 Upvotes

As a beginner, it seems like there is a lot of decentralized information on this sub for recommendations for model/engine/harness for different common hardware setups.

Would there be a possibility to have a pinned thread on this sub where people can post these recommendations in one place? Or even better have an ongoing poll for each common consumer system (M5 Max, M5U, Spark, etc), for people’s favorite configs.

It seems like something like this would also cut down a lot on new posts where people buy a new computer and want to know what to run.

Thanks in advance!


r/LocalLLM • • 1d ago

Discussion Flash-Next is ridiculous

277 Upvotes

I've been playing around with Flash-Next on my little RTX 5060 laptop. Astra and Fable had been working on a test setup, so I figured I'd let Flash-Next loose on it overnight and see what happens.

Late at night I told Astra to turn the screen off. Now, I had actually prepared a little PS1 script for exactly that. Did Astra use it? Of course not. It came up with its own solution and promptly froze Windows. Well done. 😂

Next morning I booted the laptop again, reopened OpenCode and eventually just said "okay, continue". Honestly, I wasn't expecting much at that point.

Well, the thing just carried on working. For hours. Went through something like four context compactions and then casually informed me it had found a bug.

I was sitting there going "hang on... WHAT? You're still working on this?"

I'd pretty much written off the whole session the night before.

Still not entirely sure who was testing whom here. But honestly, I'm impressed by this model. Now I'm wondering what Qwen4 will do to my poor laptop.


r/LocalLLM • • 5h ago

Model Gemma 4 twice as fast on Ryzen AI Max+ 395

7 Upvotes

Hi! Just want to share that we have optimized Gemma 4 to run more than twice as fast as llama.cpp on the Ryzen AI Max+ 395!

To maintain accuracy we use a combination of rotation and GPTQ, along with selective mixed precision.

https://embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395


r/LocalLLM • • 9h ago

Discussion hmmm. New open model. Anyone tried it??

Post image
14 Upvotes

r/LocalLLM • • 3h ago

Project 464M music model trained from scratch for €30 of rent on consumer GPUs (FP8, RTX 5090). Writes multi-instrument MIDI from English/Ukrainian prompts (477M version)

Post image
4 Upvotes

I have been recently independently working on a musicAR - a family of domain specific language models for symbolic music generation. It is best at continuation of songs from a few seconds of opening. I have validated it to other open sourced models, and got such results:

ModelParametersContinuation MRR (CLaMP 3)
musicAR464M0.81
SkyTNT tv2o-medium234M0.666
AMT large780M0.640
Music-Llama457M0.537

musicAR-L is ahead of AMT and SkyTNT in all 10,000 paired bootstrap draws. On a density/instrument-statistics distance, AMT is closer over 1,000 notes (0.097 vs 0.144). musicAR-L is closer over the first 200.

And similar comparison on the captions following. My metric is F - how good the captions are followed. Prompts come in pairs (opposite, fast/slow etc and instruments prompts like a solo piano vs the empty prompt). For each pair I measure that attribute in the generated notes, measure the difference and scale to [-1, 1.5]. F = 1 - the captions change the generation by notable amount (I have settled it myself). 0 - the captions are ignored.

Below is the table of comparisons with the other models on a short captions

ModelParametersF, English captionF, Ukrainian caption
musicAR-L-caption477M (+ frozen LaBSE encoder)1.070.95
MIDI-LLM1.72B0.60.09
Text2Midi250M0.540.06

These 2 learned from english captions only, so this comparison isnt fair. It mostly shows the good property of my model.

Running locally: Real time (24 notes/s for a band and 8 notes/s for slow music genree like solo piano)

ModelMinimum requirementsLaptop RTX 5060 (8 GB VRAM GDDR7), 1 stream
musicAR-L-caption4 GB of VRAM (used 3.04 GB on peak including LaBSE)90 notes/s
musicAR-L2 GB of VRAM (used 1.09 GB on peak) or around 3 GB RAM127 notes/s and 14.3 notes/s on laptop cpu (ryzen 9 ai hx 370, 32 GB DDR5)

Training:
musicAR-L: 8.3B notes seen (4.5B unique notes) in 41h on one rented RTX 5090, about 30 euros including generation parameters search. FP8 + torch.compile, Muon for the transformer blocks. AdamW for the rest, WSD scheduler at the end.
Captions fine tuning (musicAR-L-caption) +2.04B notes, 14h on rented 5090, plus a 20-min caption phase.
Everything else including fine tunes: Laptop GPU (rtx 5060).
Ablations: chained heads −0.41 nats/note vs independent heads (MLP + residual connection); Muon −0.54 vs AdamW in pretraining. (for mo results look at the model card)

Code: https://github.com/DmytroHilei/Music_generative_model
Weights: https://huggingface.co/Dimitri-206265/musicar-L-caption musicAR-L-caption
Weights: https://huggingface.co/Dimitri-206265/musicar-L musicAR-L
Try samples: https://musicar-ivory.vercel.app/samples

I would like to hear any feedback on the quality of the music from people close to music


r/LocalLLM • • 2h ago

Question Amd Threadripper - 4x 3060 12gb - 128gb ddr4 2666

3 Upvotes

Anyone with a similar build who can share some real tips and bench?


r/LocalLLM • • 59m ago

Question Latest llama.cpp build feel 10-15% faster with no config changes?

2 Upvotes

Not sure if anyone feels the same way I do, I just updated to latest build and noticed faster PP and TG in general running Qwen3.8 27B UD_Q4_K_XL. Not trying to throw benchmarks or stuff but anyone feels the same?


r/LocalLLM • • 4h ago

Question Can you run any fun LLM on 1080 and 64gb of ram?

4 Upvotes

For a simple email sorting agent


r/LocalLLM • • 1d ago

Project 6x 7900XTX rig finally complete!

Thumbnail
gallery
278 Upvotes

5975WX (for 6x full 16 lanes of PCIe4.0)

3450W AliExpress special PSU and mining frame

128GB 3200MT/s Ram

Getting 120-130 tok/s generation on both qwen3.8 27b and flash-next


r/LocalLLM • • 7h ago

Project Half-bit rates to EXL3 and made two Qwen3.6-35B-A3B quants with them (~200 tok/s on a 3060 + 5060 Ti)

6 Upvotes

So

I've been messing with EXL3 quants of Qwen3.6-35B-A3B on my two mismatched cards, a 3060 12GB and a 5060 Ti 16GB. The thing that kept bugging me was that EXL3 only does whole-number bit rates per tensor. A tensor that wants about 4.5 bits gets either 4 or 5, and on a 35B MoE that rounding adds up fast.

So I forked it and added 1.5 / 2.5 / ... / 7.5 bpw, with the kernels to go with them. Then I measured where this model actually loses accuracy when quantized and placed the bits from that.

The weirdest finding: most of the damage on hard text (around 87% of the KLD) wasn't the experts being imprecise. It was tokens getting routed to different experts than the full model would pick. So the attention/DeltaNet layers that feed the routers get 8 bits, and the experts in the middle layers can drop to 4.5 without much harm.

Against BF16, a build with whole-number rates only came out at 24.6 GB with a KLD of 0.0166. The half-bit builds are smaller and closer to BF16:

T5 (24.5 GB, KLD 0.0156) is the one to grab if you want it as close to BF16 as possible. About 177 tok/s on my setup.

T5 FV (23.9 GB, KLD 0.0157) is the faster version: 0.6 GB smaller, about 204 tok/s.

Both keep the vision tower and the MTP head, so speculative decoding just works. I ran both through 13 benchmark suites (MMLU-Pro, AIME, GPQA Diamond, LiveCodeBench, HumanEval, IFEval and a few others, 1,910 questions total) with thinking on and the full thinking budget. They landed within a handful of questions of each other: 1,546 vs 1,557, which is basically noise for a single run. Per-suite numbers are on the model cards.

You do need the fork, since stock exllamav3 can't load the half-bit tensors. I tried to make that painless: clone it and run `MODEL=/path/to/model bash install.sh`. It sorts out PyTorch for your driver, compiles, tests every bit rate, then figures out how to split the model across whatever GPUs you have. No CUDA toolkit needed. Fair warning: the compile takes a while (15 to 40 minutes).

This also leads into my EXL3 quant project which I’ll be uploading some more models any recommendations please let me know as my hardware is meh bigger model may not be possible or will take weeks with my setup so will be aiming for under 40b models if yall have any recommendations

- Fork: https://github.com/hunterxma/exllamav3-half-bit
- T5: https://huggingface.co/HydraAiM/Qwen3.6-35B-A3B-EXL3-HalfRate-T5
- T5 FV: https://huggingface.co/HydraAiM/Qwen3.6-35B-A3B-EXL3-HalfRate-T5-FV

Big thanks to turboderp for EXL3 and to the Qwen team for the model. If anyone runs these on other cards, I'd love to see your speeds, especially single 24GB setups with some experts on the CPU

THIS HAS ONLY BEEN TESTED ON Ampree and Blackwell so any other cards have not been tested if EXL3 works on them me too broke for multi card brand setups lol sorry


r/LocalLLM • • 1h ago

Project Running HY4 FLASH 220B

Post image
2 Upvotes

Still working on optimization!

https://github.com/oscarsantanaarias/Sneo


r/LocalLLM • • 2h ago

Discussion Running 125B Qwen3.8-Flash on a 16 GB M4 Mac at ~18 tok/s and 16K context — works right alongside llama.cpp (Kraits)

Thumbnail
reddit.com
2 Upvotes

r/LocalLLM • • 5h ago

Question Budget LLM Setup for M365

3 Upvotes

Hi!

I’m looking into creating a local LLM setup for my father who runs a non-profit. Because his org is cash-conscious, they still only use the free versions of ChatGPT/Claude.

I want to see if I can build him something that can function as an executive assistant - e.g., triaging emails, creating simple word docs, reading an Obsidian graph, etc.

He basically only uses the M365 suite for his work.

Here are some questions I have:

  1. About how many parameters minimum would he need?
  2. Would 16gb RAM on a Windows PC work? This is also with other programs up and running too.
  3. If not, would I be able to convert a gaming PC with software configurations, or would I have to add additional hardware?