r/LocalLLM • • 8h ago

LoRA Absolutely night and day for Qwen Coding

Thumbnail
huggingface.co
102 Upvotes

Basically works like a harness fine tune…. And does what it says on the tin.

Seriously, for quantized qwen 3.8 makes it much much more usable.


r/LocalLLM • • 3h ago

Discussion Why is above 35B and below 70B still such a dead zone for local models?

85 Upvotes

Title basically.

One could argue the “dead zone” goes up to 120B now. After Qwen3.8 27B all the way up to 120, there’s very little advancement in this area unless I’m missing something.

Nemotron has some models in between, some of the older Qwens had stuff in the 30-50s I think, but other than that, it doesn’t even seem like there’s any real progress either.

I do understand most people either have under 32GB VRAM or they just jump to the 128 and beyond tier but more choice is better isn’t it?

If we could get solid 50-70B models, they could be quantized down to 5-6 bit and be better than 3.8 27B in theory and still run on 36-64GB setups?

Am I just way off and incorrect in my train of thought?


r/LocalLLM • • 22h ago

Discussion I think I give up on finetunes after comparing them. I don't look at benchmarks anymore. Qwen finetunes are dime-a-dozen, but none of them do better than the original.

Thumbnail msnab.pages.dev
68 Upvotes

r/LocalLLM • • 23h ago

Question A comparable local setup for someone using $100 monthly claude subscription

43 Upvotes

I see people switching to local hardware , my question is at what usage level and pattern, it is cheaper to run it locally without impacting your work speed.

--
My current laptop and Mac mini are old but they work for me since i only use it for development while relying on Claude and Cursor for GenAI. If i were to rely on local LLM, what is the minimum hardware, I should buy to ensure that I am not slowed down.

Notes from responses:

Hardware levels mentioned

  • Tinkering (cheap): a 3060 12GB or 5060 Ti 16GB with 64–128GB of DDR4 RAM on an old Xeon board.
  • Usable for real work: a 128GB Strix Halo box (about $3.3K), a 3090 or 4090 with 96–128GB of RAM, one or two R9700 cards, or a 128GB Nvidia box (about $5K).
  • Serious: a 512GB Mac Studio (about $22K), two M5 Ultras, or two to three RTX 6000 Pro cards.

The approach that works: hybrid

  • Split the work: the cloud model plans, coordinates and reviews, and local models do the repetitive bulk work.
  • Example 1: one user runs two R9700 cards with 96GB of RAM and lets Fable direct several local Qwen workers. They went from hitting the weekly limit of Claude’s top plan to struggling to use up a Max plan. Their tips:

    • The cloud model reviews only the changes, not everything.
    • Compact the conversation often.
    • Turn off memory features.
    • Write precise instructions.
  • Example 2: a $1,500 used PC with a 3090 handles a brokerage’s client emails and documents locally. A $20 Claude plan checks the work without seeing client files, and the setup replaced a part-time contractor.

Good reasons to go local (none of them about saving money)

  • Privacy: a firm building on-premises AI for law firms says drafting a shareholders’ agreement takes about 4 minutes on a $4K machine, or about 20 minutes with the agent pipeline. About $10K of hardware makes that 2–3× faster. Good templates plus a short human review make the output client-ready.
  • No usage limits: you’re limited only by your power bill.
  • A hedge against future price rises and vendor lock-in: one user compares it to insurance. Others reply that preparing years early can just waste money.
  • Learning and hobby use, uncensored models, and control over when the model changes.
  • Hidden cost: the time you spend setting up and maintaining it.

r/LocalLLM • • 9h ago

News A Danish AI lab has just launched one of the fastest LLMs for phones to date

Thumbnail
sevren.ai
41 Upvotes

Just came across a LinkedIn post mentioning this company, Sevren which launched a very fast and intelligent LLM for phones, they call it Embla. Apparently they have decided to trade a few benchmark points for speed. I'm linking their blog post about the model here and I think it’s impressive.

1,77s speed-to-intelligence is amazing. Imagine how efficient in terms of battery/energy consumption it should be. Of course there are limitations and they are clearly stated by the company and the articles about it, but seems like they can achieve much more with a couple of iterations on top of it.

I believe that soon this kind of model will be good enough to answer >90% of questions a normal user would ask daily to a LLM like chatGPT.


r/LocalLLM • • 1h ago

Project SSD MoE at high speed - The wait is over!

37 Upvotes

The wait is over!

Introducing Run My AI & TokenWing.

An inference engine that lets you offload over 60% of routed experts of mixture of experts models into SSD, yet achieve very low streaming latency...

And a GUI wrapper aimed at providing the best local AI experience.

Benchmarks: Ling 3.0 Flash 120B MXFP4 running on 12Gb VRAM + 32GB DDR5 RAM laptop at 45-50 tok/s decode and 900 tok/s prefill

Qwen 3.8 Flash Next 3.5bpw running on 8GB VRAM + 32GB DDR5 RAM laptop at 35-40 tok/s and 600-700 tok/s prefill

Flint 40B UD-Q4_K_XL running on a laptop with 24GB DDR4 RAM + 4GB VRAM at 30 tok/s decode and 400 tok/s prefill

All without MTP! And benchmarks are by users*

check the website out: https://runmyai.org

Also, the autotuner is still work in progress...

A common question, does it degrade SSD?

Nope! it doesn't... There's a thing called TBW rating. Well, your SSD degrades only when it writes. And the engine only reads....


r/LocalLLM • • 6h ago

Model SenseNova-Loop-9B matches Qwen3.8-27B on AutomationBench, and it now has official GGUFs (Q4_K_M is 5.8 GB)

Post image
34 Upvotes

SenseNova-Loop-9B came out this week, and the official GGUFs are now up.

It's only about 9B parameters but gets more depth out of them. Each token gets 56 layers of processing while the weights only hold 32, because the 24 in the middle are used twice. According to the chart in their README (attached), it sits at 36.67 on AutomationBench, the exact score listed for Qwen3.8-27B. Those are their own numbers, so let's see what independent tests say. Their repo with the full README is https://github.com/OpenSenseNova/SenseNova-Loop

As for the GGUFs, Q4_K_M is 5.8 GB, Q5_K_M 6.7 GB and Q8_0 9.8 GB. It's a new architecture, so LM Studio and stock llama.cpp can't load them at the moment. Instead you build OpenSenseNova/llama.cpp, the official fork. The exact commit and the command to run are on the GGUF model card: https://huggingface.co/sensenova/SenseNova-Loop-9B-GGUF

It takes text and images, and the license is Apache 2.0. If you serve from a GPU box there's a 4-bit AWQ too: https://huggingface.co/sensenova/SenseNova-Loop-9B-AWQ


r/LocalLLM • • 22h ago

Question 27B vs Flash-Next on a single RTX Pro 6000, what am I missing?

32 Upvotes

I have one RTX Pro 6000 (96 GB) and 200 GB RAM, running vLLM. On that card I can run Qwen3.8-27B at FP8, or Flash-Next at NVFP4 with the n-gram table in RAM. 8-bit Flash-Next doesn't fit, so that's the comparison I'm stuck with.

I ran both through my own tests, same settings:

- 15 support-type tasks: 0.97 for 27B, 0.98 for Flash-Next

- 15 multi-step tool-calling tasks, 135 runs each: 27B 0.96 to 1.00, Flash-Next 0.98 to 0.99

- Flash-Next made 3 write calls nobody asked for. 27B made none.

- Both ignored the prompt injection I planted mid-chain, every run

- 27B takes 28 GB VRAM and restarts in a minute. Flash-Next takes 76 GB plus 95 GB RAM and needs ten minutes.

27B was also faster per chain, but it had MTP on and Flash-Next didn't, so don't read much into that.

I see the same thing at home on a 32 GB card with llama.cpp (27B Q4 vs Flash-Next IQ3).

So for me 27B does the same job for a third of the memory. Both score near the top on my tasks, so maybe they're just too easy to show a difference.

If you run Flash-Next on similar hardware, what does it do better for you?

Please tell me what you use it for. Coding? General agentic life management? Support?


r/LocalLLM • • 6h ago

Question Could uncensored open weight models in 2027-2029 force us to build something like the cyberpunk "Blackwall"?

31 Upvotes

It might sound like pure scifi, but just imagine local models reaching the reasoning level of something like Fable 5.2 (which will probably happen in about a year or two [it's hard to predict]). Now picture 100,000 people downloading an uncensored, Fable 5.2 level model onto their home pcs and running it in a 24/7 agentic loop with a prompt like "go make money for me." It’s pretty easy to imagine the kind of chaos that could unleash on the internet.


r/LocalLLM • • 23h ago

Discussion <tinfoil hat> With many frontier labs desperate for regulation to impose regulatory capture and limit liability at what point, after the hugging face incident, etc, are we allowed to wonder if they would be willing to "encourage" an "ai incident" to push the issue of US regulation </tinfoil hat>

19 Upvotes

The title pretty much says it all ... I'm not saying this is happening, I'm just asking the question out loud ...


r/LocalLLM • • 3h ago

Question Hy4 flash is now open weight: what local setup makes it usable?

Post image
17 Upvotes

The Instruct, FP8 and base checkpoints are now publicly available.

Hy4 Flash has 224B total parameters (12B active), a 1M context window, and native text and image input. GGUF versions are also available under the Tencent Hy License.

The FP8 weights alone require well over 128GB, before runtime and KV cache overhead. Would a quantized GGUF setup with CPU/SSD offloading be practical at a useful context length, or is this still multi GPU territory?


r/LocalLLM • • 16h ago

Project Feel how fast a local LLM runs before you buy the hardware

Thumbnail
llmspeed.dev
15 Upvotes

r/LocalLLM • • 9h ago

Discussion hmmm. New open model. Anyone tried it??

Post image
14 Upvotes

r/LocalLLM • • 4h ago

Project Pocket LLM 1.7: offline Gemma 4 on Android, now with long chats that don't break

11 Upvotes

Pocket LLM 1.7 is out, the update to my open-source Android app for running LLMs fully on your phone.

New in 1.7:

- Long chats are auto-compacted: older messages get summarized instead of dropped

- Context size fits your phone's memory, so big chats stop crashing

- Fresh design with light/dark themes and accent colors

- LAN server fixes

What it does:

- Offline chat with Gemma 4, Qwen3, Qwen2.5, DeepSeek R1, or your own .litertlm model

- PDFs (with OCR), images, audio and offline voice dictation

- LAN server: browser chat and an OpenAI-compatible API for your PC

No account, no tracking. On 1.6? Install the new APK over it, and your chats stay.

GitHub (APK + source, MIT): https://github.com/dineshsoudagar/local-llms-on-android

Google Play (GGUF models + optional Pro tools): https://play.google.com/store/apps/details?id=io.github.dineshsoudagar.pocketllm

Feedback welcome!


r/LocalLLM • • 1h ago

Project Nihil IDE, free alpha, a hardware simulation lab that supports Local LLMs and 100% offline mode out of the box

Thumbnail
youtube.com
9 Upvotes

r/LocalLLM • • 3h ago

Discussion 3 x AI 9700 build - 96gb VRAM

Thumbnail gallery
7 Upvotes

Any recommendation on what else I can try on this set up?
Love this as a hobby! Just fun building!


r/LocalLLM • • 5h ago

Model Gemma 4 twice as fast on Ryzen AI Max+ 395

7 Upvotes

Hi! Just want to share that we have optimized Gemma 4 to run more than twice as fast as llama.cpp on the Ryzen AI Max+ 395!

To maintain accuracy we use a combination of rotation and GPTQ, along with selective mixed precision.

https://embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395


r/LocalLLM • • 7h ago

Project Half-bit rates to EXL3 and made two Qwen3.6-35B-A3B quants with them (~200 tok/s on a 3060 + 5060 Ti)

6 Upvotes

So

I've been messing with EXL3 quants of Qwen3.6-35B-A3B on my two mismatched cards, a 3060 12GB and a 5060 Ti 16GB. The thing that kept bugging me was that EXL3 only does whole-number bit rates per tensor. A tensor that wants about 4.5 bits gets either 4 or 5, and on a 35B MoE that rounding adds up fast.

So I forked it and added 1.5 / 2.5 / ... / 7.5 bpw, with the kernels to go with them. Then I measured where this model actually loses accuracy when quantized and placed the bits from that.

The weirdest finding: most of the damage on hard text (around 87% of the KLD) wasn't the experts being imprecise. It was tokens getting routed to different experts than the full model would pick. So the attention/DeltaNet layers that feed the routers get 8 bits, and the experts in the middle layers can drop to 4.5 without much harm.

Against BF16, a build with whole-number rates only came out at 24.6 GB with a KLD of 0.0166. The half-bit builds are smaller and closer to BF16:

T5 (24.5 GB, KLD 0.0156) is the one to grab if you want it as close to BF16 as possible. About 177 tok/s on my setup.

T5 FV (23.9 GB, KLD 0.0157) is the faster version: 0.6 GB smaller, about 204 tok/s.

Both keep the vision tower and the MTP head, so speculative decoding just works. I ran both through 13 benchmark suites (MMLU-Pro, AIME, GPQA Diamond, LiveCodeBench, HumanEval, IFEval and a few others, 1,910 questions total) with thinking on and the full thinking budget. They landed within a handful of questions of each other: 1,546 vs 1,557, which is basically noise for a single run. Per-suite numbers are on the model cards.

You do need the fork, since stock exllamav3 can't load the half-bit tensors. I tried to make that painless: clone it and run `MODEL=/path/to/model bash install.sh`. It sorts out PyTorch for your driver, compiles, tests every bit rate, then figures out how to split the model across whatever GPUs you have. No CUDA toolkit needed. Fair warning: the compile takes a while (15 to 40 minutes).

This also leads into my EXL3 quant project which I’ll be uploading some more models any recommendations please let me know as my hardware is meh bigger model may not be possible or will take weeks with my setup so will be aiming for under 40b models if yall have any recommendations

- Fork: https://github.com/hunterxma/exllamav3-half-bit
- T5: https://huggingface.co/HydraAiM/Qwen3.6-35B-A3B-EXL3-HalfRate-T5
- T5 FV: https://huggingface.co/HydraAiM/Qwen3.6-35B-A3B-EXL3-HalfRate-T5-FV

Big thanks to turboderp for EXL3 and to the Qwen team for the model. If anyone runs these on other cards, I'd love to see your speeds, especially single 24GB setups with some experts on the CPU

THIS HAS ONLY BEEN TESTED ON Ampree and Blackwell so any other cards have not been tested if EXL3 works on them me too broke for multi card brand setups lol sorry


r/LocalLLM • • 19h ago

Question Anyone running Qwen 3.8 Flash Next on 2 x 5060ti 16GB?

6 Upvotes

Trying to get this running and would appreciate any feedback


r/LocalLLM • • 22h ago

Discussion What’s the best underdog rig right now

5 Upvotes

I’m lucky enough to have a wide range of rigs in my office, from those fancy DGXs to a solid Samsung S25 with a snapdragon running the same LLMs and tbh my current biggest surprise is how fun the GPU+NPU+CPU combo on the Intel pantherlake is to mess around with i.e Gemma4-26B-A4B-Q4 pulling over 30 tok/s on it. What are your most noteworthy winners?


r/LocalLLM • • 3h ago

Project 464M music model trained from scratch for €30 of rent on consumer GPUs (FP8, RTX 5090). Writes multi-instrument MIDI from English/Ukrainian prompts (477M version)

Post image
4 Upvotes

I have been recently independently working on a musicAR - a family of domain specific language models for symbolic music generation. It is best at continuation of songs from a few seconds of opening. I have validated it to other open sourced models, and got such results:

ModelParametersContinuation MRR (CLaMP 3)
musicAR464M0.81
SkyTNT tv2o-medium234M0.666
AMT large780M0.640
Music-Llama457M0.537

musicAR-L is ahead of AMT and SkyTNT in all 10,000 paired bootstrap draws. On a density/instrument-statistics distance, AMT is closer over 1,000 notes (0.097 vs 0.144). musicAR-L is closer over the first 200.

And similar comparison on the captions following. My metric is F - how good the captions are followed. Prompts come in pairs (opposite, fast/slow etc and instruments prompts like a solo piano vs the empty prompt). For each pair I measure that attribute in the generated notes, measure the difference and scale to [-1, 1.5]. F = 1 - the captions change the generation by notable amount (I have settled it myself). 0 - the captions are ignored.

Below is the table of comparisons with the other models on a short captions

ModelParametersF, English captionF, Ukrainian caption
musicAR-L-caption477M (+ frozen LaBSE encoder)1.070.95
MIDI-LLM1.72B0.60.09
Text2Midi250M0.540.06

These 2 learned from english captions only, so this comparison isnt fair. It mostly shows the good property of my model.

Running locally: Real time (24 notes/s for a band and 8 notes/s for slow music genree like solo piano)

ModelMinimum requirementsLaptop RTX 5060 (8 GB VRAM GDDR7), 1 stream
musicAR-L-caption4 GB of VRAM (used 3.04 GB on peak including LaBSE)90 notes/s
musicAR-L2 GB of VRAM (used 1.09 GB on peak) or around 3 GB RAM127 notes/s and 14.3 notes/s on laptop cpu (ryzen 9 ai hx 370, 32 GB DDR5)

Training:
musicAR-L: 8.3B notes seen (4.5B unique notes) in 41h on one rented RTX 5090, about 30 euros including generation parameters search. FP8 + torch.compile, Muon for the transformer blocks. AdamW for the rest, WSD scheduler at the end.
Captions fine tuning (musicAR-L-caption) +2.04B notes, 14h on rented 5090, plus a 20-min caption phase.
Everything else including fine tunes: Laptop GPU (rtx 5060).
Ablations: chained heads −0.41 nats/note vs independent heads (MLP + residual connection); Muon −0.54 vs AdamW in pretraining. (for mo results look at the model card)

Code: https://github.com/DmytroHilei/Music_generative_model
Weights: https://huggingface.co/Dimitri-206265/musicar-L-caption musicAR-L-caption
Weights: https://huggingface.co/Dimitri-206265/musicar-L musicAR-L
Try samples: https://musicar-ivory.vercel.app/samples

I would like to hear any feedback on the quality of the music from people close to music


r/LocalLLM • • 4h ago

Question Can you run any fun LLM on 1080 and 64gb of ram?

4 Upvotes

For a simple email sorting agent


r/LocalLLM • • 15h ago

Question 2bit DS v4.1 1017 vs 4bit QFN?

4 Upvotes

Just like the title states I was looking into both models as I can run both on my machine 128gb + 32gb

What has been your guys experience?


r/LocalLLM • • 20h ago

Discussion I have some questions. What is your acceptable Vram buffer for the card you are actively using to display the OS?

4 Upvotes

Ok so i have been running 3.8 flash next and I have been keeping an available Vram on my 16gb 5070ti at around 1500mb remaining free so that if I have to load a browser or something it won't stop the screen displaying or something(I have had it happen once with a different system so I am being careful now) so my (probably easy) question for one of your guys right now is how much Vram do you think I should keep free? Or how much could I safely lower it to? My system is a GPU: RTX 5070 Ti, 16 GB GDDR7, in the main x16 slot.

CPU: Intel Core Ultra 7 265KF (no AMX)

RAM: 64 GB

NPU: present but not tested yet for my system

Storage: internal 2 TB SSD running Omarchy Linux, which is where all local AI runs from.

What do you all think?


r/LocalLLM • • 22h ago

Project Handing two models at once in Radiance

4 Upvotes

I’ve been working on Radiance with Secondary Models, a fork aimed at getting more use out of a local AMD workstation.

The idea is to keep Qwen Flash Next available for chat or coding while using the same GPUs for other models as you need them. You can generate an image with Qwen Image, edit it with Krea 2, then turn the result into a video with sound using MiniMax H3—all through the same server.

The fork currently supports:

  • Image generation and editing, including one- and two-reference edits with the published Qwen Image 2.1 and Krea 2 packages.
  • Local LoRA discovery and loading, with adapter strengths selected per request.
  • Video generation, including H3 text-to-video, first-frame animation, and subject-reference workflows.
  • Secondary text models, including uncensored Gemma 4 26B MoE and 31B dense conversions.
  • Automatic GPU memory sharing. When generators are idle, Qwen can use available VRAM for additional experts. That memory is recalled when a generator needs it.
  • RAM caching and request queues to make switching between models more practical.

This is intended for consumer workstations where dedicating GPUs to occasional image or video requests would leave resources idle much of the day. The current shared setup has been tested on two Radeon AI PRO R9700s, with Qwen configured for 200K context, eight concurrent slots, and RAM prefix caching.

It’s still a preview. Text generation slows during demanding secondary workloads, cold model loads can take time, and memory recovery after video generation can be gradual. The README and model cards document the tested configurations and remaining limitations.

The fork includes setup examples and links to the converted models on Hugging Face. Feedback from anyone trying other hardware or memory configurations would be welcome, I expect it may work with 128GB RAM, but you may or may not be able to load the largest models (like H3).

Radiance with Secondary Models — code, setup, and downloads