r/LocalLLaMA • • 10h ago

Discussion AI Hysteria on Reddit

0 Upvotes

Lately, I’ve often found myself wondering what’s going on with people, especially in communities dedicated to AI and LLMs.

Humanity now has an amazing tool for breaking down language barriers, along with creative tools unlike anything we’ve had before. I can’t describe people’s reactions as anything other than hysteria.

There are thousands of posts in these communities about improving models and inference engines, including translation models, coding harnesses, and so on.

But when someone uses AI to translate a post into English or shows off a program they built with AI, the comments immediately erupt: “Low-effort post,” “AI slop.” And all of this comes from the very people developing that “SLOP.” WTF?

(This low-effort post was translated by GPT.)


r/LocalLLaMA • • 23h ago

Question | Help Why Qwen 3.8 27B feels smarter than GLM 5.3

0 Upvotes

Is this little thing feeding me cool aid or is really this smart?

Why though, how can a 27B model punch so good? Are cloud models like GLM 5.3 optimized for speed and don't dig too much? Where is the trick, can someone tell me?


r/LocalLLaMA • • 17h ago

Question | Help [Looking for tester] GLM 5.3 Flash at 10+ t/s on consumer hardware

5 Upvotes

I'm still looking for people who want to try out the engine I'm building (obviously with the help of Claude). I wanted decent speed for RP with a good model like GLM-5.3-Flash. Before that I only used Gemma 31B and Gemma 4 26B-A4B, which are good, but not good enough for me.

I first built it only to get the best performance out of my own hardware: RX 9070 XT (16 GB), Core Ultra 7 265KF, 32 GB DDR5 and two NVMe drives. On that machine it now does (all numbers in this post are for the IQ4_XS quant):

- Linux: about 11 tok/s, and reads a long prompt at about 280 tok/s (ROCm and Vulkan are about equal now)

- Windows: about 10 tok/s, and reads a long prompt at about 250 tok/s (Vulkan; ROCm is a bit slower there)

Since my last post a few people tried it, and it now also runs on Intel and NVIDIA cards. What they measured, and what I expect after the update that goes out now:

- Intel Arc Pro B70 (32 GB) with a Ryzen 9 7900X: about 11 tok/s, I expect 12 to 13

- RTX 3090 + RTX 3060 in an older PC with two PCIe 3.0 drives: 5.6 tok/s, I expect 6.5 to 7

The expected numbers are calculated from their logs, not measured. I can't measure them myself, that is what the testers are for.

What it does:

- Reads only the experts each token actually needs and keeps the hot ones on the GPU and in RAM

- Uses several NVMe drives at once: put a copy of the model on each, and every read goes to the drive that will finish it first

- Can use two GPUs together, also two different ones

- Measures your drives and your GPU at start and sets itself up for them, so slow hardware is not treated like fast hardware

- Predicts the next layer's experts and reads them ahead

- Can prefer experts that are already in memory when the router's scores are nearly tied (small quality trade-off, can be switched off)

- In a chat, only the new message is processed each turn, even after the context is full

- Saves the chat state to disk: after a restart a 29000-token chat answered after 9 seconds instead of reading everything again

- KoboldCpp-compatible API, so it works with SillyTavern (Chat Completion works too)

Two NVMe drives are probably important for the best speed, but you can try it with one or even more than two. On my Windows machine the second drive is worth about 25%.

Where it stands: it runs on Linux and on Windows, with three backends: ROCm (AMD), Vulkan (any card) and CUDA (NVIDIA RTX 30 series and newer). For Windows there is a package with prebuilt engines, nothing to compile and no SDK to install, you only need your GPU driver and Python. Everything is tuned on my one machine. The last round of changes was made for slower hardware (old drives, a second smaller GPU, a slower CPU), but I could only imitate that hardware here, so I don't know yet how much it really helps. That is exactly why I need you. Old and weak machines are as welcome as new ones.

Testing is easy now on Windows: there is one file you start, it runs all tests that fit your machine by itself and writes one report file you send me.

You need a CPU with AVX2, 32 GB of RAM and about 157 GB on a fast NVMe for the model (the IQ4_XS GGUF, bring your own). A GPU with 16 GB holds all of the model's layers, less has not been tried.

I'm having a blast RPing with it, and if you are willing to try it out just dm me or reply to this post and I invite you to my private github repo.


r/LocalLLaMA • • 18h ago

Question | Help Finally joined the club today. New (to me) machine in transit. What good stuff can I reasonably run?

0 Upvotes

I just ordered a cheap, but prehistoric Xeon E-2286M 32GB RAM, 2020 era Nvidia RTX 5000 (16G vram) laptop. Given these resource constraints, what are some good coding-oriented models to initially look at? I'd like to play with a capable TTS model that permits cloning and inflection control, and still image generation. Plus I'd welcome suggestions on any fun 'oddball' applications that don't fall neatly into text/image/video silos.

Would doubling machine memory up to 64G improve execution times generally?


r/LocalLLaMA • • 3h ago

Discussion Local TTS for Indian languages: anything good yet?

0 Upvotes

Blue TTS v3 looks solid (MIT, 388 MB, about 50x real-time on CPU), but it's 7 languages and I'm not sure Telugu or Hindi is in there.

For anyone building voice stuff for Indian languages, what are you actually running locally? Is anything close to usable for long-form, like reading out a lesson, or is it still cloud APIs for now?


r/LocalLLaMA • • 23h ago

Discussion Unpopular take: If anything, it's the US Closed Models that are benchmaxxed

114 Upvotes

Open models: whatever the test group download and benchmark is the same as what you get, you work with it day-to-day, same results. What you see is what you get.

Closed models: we all know that they have already implemented token watermarking for a while and all latest models have overzealous guard rails that refuse tasks that it deems as bad. So the models being benchmarked and the models you are actually using day-by-day basis are likely to be very different, as in the sense of you getting degraded model output and performance or flat-out task rejections.

Also closed models providers implement lots of cost-optimization techniques such as reducing reasoning efforts during peak hours or could be re-direct your request to a lesser model during high traffic without you knowing it. I don't believe that Sonnet 5.5 is just a single model TBH, there could be Sonnet High or Sonnet Lite depending on your subscription level and traffic. That could explain why sometimes you get dumb responses with Claude during peak hours.

If anything, it's the US Closed Models that are benchmaxxed.


r/LocalLLaMA • • 12h ago

Discussion Anyone else finds Qwen 3.8 Flash Next pleasant to read vs cloud models?

24 Upvotes

Might make me hesitate to update for hobby use even for incremental capability gain. Granted I put an effort into AGENT.md / harness tweaks but didn't work as well for other models. Opencode status updates and markdown docs seem more casual/readable, no going overboard on tables and bullet points, colloquial phrases like "two honest caveats".

Also I know overthinking rep (that I am combatting with harness tweaks), but at its best the model outputs really well put together one page reasoning blocks, that make me go "I don't even care if helped the model, I just learned something new myself and didn't get a headache trying to get it. That for tasks like customizing other ML models.


r/LocalLLaMA • • 22h ago

Showcase Qwen-Flast-Next (UD-IQ4_XS v3.1) running 148-182 tok/s (1.6-1.8x strata) while KLD MEAN = 0.08195

Thumbnail
github.com
26 Upvotes

Numbers only, rest is in the repo.

Quality against BF16

KV PPL mean KLD top-1 paired dKLD vs int8 [95% CI] prefill 131K
int8 3.2323 0.08195 90.45% reference 6,019 t/s
fp16 3.2320 0.08230 90.33% +0.00035 [-0.00061, +0.00131] 2,997 t/s
k8v4 3.2424 0.08621 90.26% +0.00426 [+0.00307, +0.00545] 5,694 t/s
q4_0 3.2399 0.08843 90.03% +0.00647 [+0.00499, +0.00795] 5,700 t/s

Speed

Test Halocline Strata 0.2.10, same card and quant Ratio
Warm decode, short prompt 148 to 182 tok/s 95 to 101 tok/s 1.6 to 1.8x
36K needle: decode / prompt 123 / 6,298 tok/s 68 / 2,403 tok/s 1.8x / 2.6x
137K needle: decode / prompt 121 / 5,664 tok/s 87 / 2,864 tok/s 1.4x / 2.0x

r/LocalLLaMA • • 26m ago

Discussion The frontier of AI has stalled. Local LLM will too.

• Upvotes

I’m fully prepared to be wrong about this, but I expect very few leaps in capability in local LLM. I don’t mean there isn’t more to be gained with speed and size, but actual capability has hit a local plateau. The difference between a 500B model and 9B model has shrunk and will continue to.
Qwen4 will be a small disappointment. The post training is taking much longer because they can’t squeeze any juice out of the Claude and OpenAI models because those models haven’t improved in ways that most people can see. Enterprise customers are paying through the nose for tokens and getting fed up quickly and moving to local Chinese models because of cost. DeepSeek is getting overwhelmed and is basically out of compute it seems.
I am not really upset because flash next is actually good enough for me most of the time and Claude subscription is still cheap enough.
I hope everyone gets their small model that is good enough for them, but I don’t think the massive home LLM will be heads and shoulders over them like they were one year ago.
Source: I’m an engineer working with LLM


r/LocalLLaMA • • 19h ago

Showcase Use local models on your phone to monitor your screen! Demo of what you guys asked for in my FOSS and 100% local agent (iOS and Android app)

0 Upvotes

Hey r/LocalLLaMA !

This is a demo of the new phone locking tool!

What this can be used for:

  • Lock my phone when I open up Instagram.
  • Lock my phone when a stranger is on camera.
  • Lock my phone until i've read for 2 minutes without interruptions.
  • If i'm doomscrolling for more than 2 minutes, call me, if it's been 5, lock my phone.
  • Monitor my child's screen content and if it's inappropriate, lock the phone and send me a screen cap.

On my last post the top comment (62 upvotes) by u/nokia7110 asked for it. It was a great suggestion! Thanks nokia!

The tool:

On iOS it uses the screen time management API, like all other time focusing apps.

On Android it shows a phone locking overlay.

If you're an Android developer and want to contribute, please reach out!

It uses OneJev-0.8b for decision models, running on llama.cpp, it works on most phones i've tested. Even old potatoes.

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

I'm here for any questions, more suggestions and feedback!

Roy


r/LocalLLaMA • • 13h ago

Discussion Y'all's idea of "slop" is stuck in 2025

562 Upvotes

Everyday on /r/LocalLLaMA i see people starting hell about if a comment/post is an llm-genned slop. And I still see people joking about em-dashes or "load-bearing" as some indicator of slop. I haven't seen any recent model say "load-bearing" in months.

modern claudism/gptism isn't just mixing fancy words or em-dashes.

it's the:

  • set-theory based distinction on every sentence "every X is a Y" "no A is a B" "just X, no A, no B, no C" "x is never Y"
  • nonsensical watermark replacement of words to with its synonyms
  • abtract metaverb replacement of common verbs
  • inconsequential caveats - "One honest confound:" "One caveat:" "One thing to note:" "One decision for you:"
  • inconsequential contrast - "X has not been done" "No X has been changed" "This does not mean X"
  • ambiguous spamming of "it/its"
  • ending with an engagement bait: "curious how everyone else is x" "Want me to x?"

The more we know the more we can combat slop in this sub. Please share your updated, 2026 list of slop you despise to see (so that I can also steal and add to my BANNED section on agents.md).


r/LocalLLaMA • • 4h ago

Question | Help Honest question: how much RAM do you recommend for running local LLMs alongside coding agents

4 Upvotes

I’m a huge AI enthusiast but I’m running into limits with 32GB DDR5. Between running local models and using coding agents (which fire off scripts and changes super fast), my RAM fills up quickly. How much RAM are you running, and what would you recommend for a smooth setup? Bonus points if you share your GPU/VRAM too.


r/LocalLLaMA • • 21h ago

Showcase Programming Conways Game of Life In Qwen

7 Upvotes

Hey, everyone. If this is too self promotional happy to take it down.

Last week, inspired by the weight space programming post, I programmed Conway's Game of Life into Qwen with some interesting results.

Wanted to share here in case anyone else is experimenting with this kind of thing:

https://reddit.com/link/1xyvox2/video/18hx6naii82i1/player

Writeup here: https://jsfour.substack.com/p/putting-a-tiny-world-inside-a-language


r/LocalLLaMA • • 19h ago

Question | Help Whats the best Video/Audio Model?

0 Upvotes

I want to make a short film for fun, not sure what is out there that is actually good?

would love anyones advice


r/LocalLLaMA • • 22h ago

Showcase I compared Qwen 3.8 27B vs 3.8 FN vs DS V4.1 Flash on a real frontend task (one shot). Results inside

58 Upvotes

So last time I ran 3.8 FN against DS V4.1 Flash on a synthetic test, which was mostly for show, no real usage beyond testing their 3D capabilities. I wanted to do something similar, but with purpose, a task that anyone might actually want to run, build a website with a refined spec and a completion checklist, but not strict enough that it removes all of the models "freedom". As last time it was a single prompt: build a single file landing page for a fake famous NYC photographer (real photos, accessible, no console errors, yadda yadda). Each model started on an empty folder w/ the same toolkit (node, headless chrome, image viewer tool). No changes were made to the files after the model was done.

Setup: RTX 5090 + RTX 5060 Ti, Ryzen 9 9950X, 128 GB DDR5. Qwen 3.8 27B and Qwen 3.8 FN were NVFP4. DeepSeek V4.1 Flash (1017) was an IQ1_M quant.

Qwen 3.8 27B: ~12 min wall time, 81 tool calls (3 failed), 182 tok/s, 3.8 MB HTML

Qwen 3.8 27B Website

The simplest of all three, not very daring. A dark gallery with lime accents and 12 frames. All 3 tool failures were in its own test script (wrong file path, missing elements). It has empty gaps in the photo grid, and the photos in the bio are stretched. Decent, I'd refine it a bit but definitely usable.

Qwen 3.8 FN: ~12 min wall time, 124 tool calls (4 failed), 303 tok/s, 2.1 MB HTML

Qwen 3.8 Flash Next Website

My personal favorite by far, a hero where your pointer reveals a mosaic of the portfolio, 19 captioned frames with a nice blur/fade in animation, a live clock and form validation. The 4 failures were 3 rejected edits and a search sent with an empty query (intersting!). It also found and fixed bugs using its own tests.

DeepSeek V4.1 Flash: ~36 min wall time, 155 tool calls (1 failed), 94 tok/s, 1.4 MB HTML

DS V4.1 Website

The boldest one, the model committed to it (and it shows in wall time), cool accents all around, terrible typography. It has only 10 frames, and the hero photo covers a bit of the last "T" of the headline + toolbar logo is slightly offset. The single tool failure was a multi edit that applied 15 of its 17 changes, so the most accurate model at tool calling. It was 3x slower than the other two, but it was the only one that ran an accessibility audit tool on its own page.

Live builds, screenshots, the full comparison table with the exact prompt I ran are here: https://basalt.jgulfo.com/showcase/photographer/

Anyway, what are your thoughts? I think for web UI specifically 27B is more than enough for a very professional looking product, with a little bit of back and forth and more proper branding/UX design I can see any of them being a production website. I also can't stop being amazed at how a IQ1 quant can still deliver such high quality results even if the param count is multiple times over the two other models.

---

Disclosure: I'm the maintainer of the engine and harness all these models ran on.


r/LocalLLaMA • • 7h ago

New Model Flint anyone?

14 Upvotes

I see a lot of lament about Qwen not making 3.8 35B (and I'm sad about it too) but I dont see anyone discuss Flint 40B, a new coding-oriented MoE model that seems to be atleast on par with qwen 3.6 35B, maybe a bit better on agentic tasks and coding. Very fast in my setup.

I'm still testing it myself, but I wonder if its not liked here or people just arent aware it exists? It performs admirably for me so far, my local model of choice for when I want good speeds. QFN is better ofcourse but for me that means low quants or low speeds.

Curious about you guys' experience with the Flint model.


r/LocalLLaMA • • 22h ago

Resources Running Qwen3.8 27b on 12GB with full context, 70 TPS, MTP, and 50% lower KLD

23 Upvotes

Hello everyone. LlamAmpere v0.7 is out with a series of updates focus on the small card users: a new model for the 12GB cards that is comfortably best in class for its size, updates to the runtime to support the previous* 2.3 bpw model (originally designed for 12GB) on 10GB cards. I am working on an 8GB size distillation and hope to have that done in the next couple of days.

The quick rundown for you:

Vs the October 9 2.3 bpw file, which was the best 2.3 file that I tested for 12GB with long context and MTP (though 15% worse than the ~11% larger Mirai file), this most recent file for the 12GB segment is nearly 70% lower error (>50% vs the previous fit ~2 weeks ago), with a top 1 match of ~96% vs BF16 (as before, this quant is based on the Swift distill, which cuts token verbosity 30-50%). This 12GB class model is noticeably nicer to use in harnesses like hermes and pi than any other model I’ve been able to fit previously.

Vs the recently upgraded v3.1 unsloth quants, it is ~25% lower error despite being 8% smaller on card. This is while supporting ~200k ctx and MTP. (note that the embedding table is kept in RAM at high fidelity to add quality without a meaningful hit to speed)

There are a number of tricks involved to make this possible, but 2 I will call out in particular: v0.7 uses both dynamic microbatching and Latent Sparse Attention.

The first one is straightforward: the model starts with the largest microbatch (ub) that will fit on the card to maximize initial prefill, and then periodically resets to smaller batches as context fills up. This frees up hundreds of MB for more context at the end without nuking your prefill for the entire chat.

The second is based on the paper by prefontal AI that was recently published. LSA moves from full dense attention to sparse attention; it projects tokens through a tiny learned map (distilled from the model's own attention) into a 64-dim "index space". It then groups past tokens into small clusters, scores the current query against each cluster, and runs normal, exact attention only on the best clusters. The initial attempts I ran with this showed the same problem many users have noticed with Qwen Flash Next: sparse attention can struggle at deep context. Unlike other sparse attention implementations, which only use ~2K tokens, LlamAmpere is currently using the most recent 8K tail, the 128 first initial tokens, and 20% of the intermediate tokens to protect quality. This showed significantly stronger attention performance (10-40x less error from attention, depending on the test), at the cost of somewhat slower attention. With a cache for recently attended to tokens, we can push most of the context out of the card with modest quality and speed penalties. This allows a much more capable model to fit.

As before, the recommended config uses the AmperInteger+ trellis format and SJKVaRN to efficiently compress the weights and KV.

These same techniques yield benefits for the 10GB class, as what was once required 12GB now fits on 10. The previous 2.3bpw quant now fits 213K context (headless) on 10GB (180K headed). 12GB cards can support the full 262K headless and 180k headed. This model previously tested to perform in line with 3.6 BF16, so it is a very solid quant.

As before, the code is available on github, with an MIT license.

And of course, as soon as qwen4 27b comes out, I'll get back to work on quantizing them similarly.

https://github.com/JakeATX/llamAmpere

12gb model (3bpw): https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI3

10GB model (2.3bpw): https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2

Enjoy!


r/LocalLLaMA • • 4h ago

Resources ToRc: a from-scratch Rust engine running Qwen3.8 Flash Next (Q4), a 107 GB model, at 67 tok/s prompt / 17 gen (on a 16k prompt) on two £300 Snapdragon boxes with 64 GB of RAM between them. CPU + NPU + iGPU, in a 13 MB zip

28 Upvotes

So this started because I bought two of the base Snapdragon X boxes for some software and build testing at work (the cheap ones, about £300 each, not the Elite) and then they were sitting unused. With fast LPDDR5x (8448 MT/s) in them I thought I'd have a play and see if I could run proper models on them. I tried the usual engines and couldn't get anywhere near the speed I thought the hardware should do... So then my ADHD took over and I got carried away, and this is the result 🤣

ToRc is an inference engine written from scratch in Rust. Weeks of days and nights measuring, rewriting and measuring again went into it, and the numbers below are all measured on my own kit, different platforms etc... So I hope this proves useful to others 🙏🙃

ToRc on GitHub

Measured performance

If you can unzip a file, you can run your own model on your own machine. The download is a zip of about 13 MB (Windows x64, Windows on ARM, Linux x64 and Linux ARM, plus an experimental Mac one): unzip it, open ToRc, pick a model (a GGUF converts once, and it tells you how much space that takes first), press Auto-tune, and talk. A few minutes from download to chatting, nothing installed, and the app keeps everything (settings, chats, the lot) in that one folder. The only thing outside it is what the engine learns about your machine (Auto-tune's results and the like), in %LOCALAPPDATA%\torc or ~/.cache/torc.

What it can do:

- Uses the CPU, the NPU and the integrated GPU at the same time where it's present or faster, per model. Or just NPU + iGPU if you want to leave the machine usable while it runs.

- Tunes itself to your machine, and every setting can be overridden if you want to fiddle / let an agent try to wring every last token of performance for your particular setup.

- Speaks the OpenAI and Anthropic APIs, so Claude Code, OpenCode and anything else that talks to those just works. Full tool calling and MCP is also built in, and the desktop app has its own chat, tools and MCP servers if you just want to talk to it.

- Runs on Windows and Linux, x86 and ARM. (Linux: one user runs it on bare metal on a Snapdragon box under Ubuntu, and I've run it under WSL2, so tell me how it goes on yours.) There's also an EXPERIMENTAL Mac zip (Apple Silicon): one person has built and run it from source, and GitHub builds the zip, but I don't have a Mac to test it myself.

- It converts to its own file format for efficient SSD streaming where the model won't fit into memory and can split one model across multiple machines. I've tested two boxes over USB4; in principle it's N boxes. You pool their RAM, memory bandwidth and compute for enhanced performance.

Headline numbers, engine timed, read (prompt) / write (reply) tok/s:

- Qwen3.8 Flash Next, a 107 GB model, on two Snapdragon boxes with 64 GB of RAM between them (the experts that don't fit stream off the SSDs): ~67 / ~17.5 on a 16k prompt, ~64 / ~21 on a short one

- Qwen3.6 35B A3B, 4k prompt, two Snapdragon boxes: ~406 / ~54 (one box: ~283 / ~56)

- Same 35B on an i9-13900K (8 threads, CPU only), 4k prompt: ~243 / ~44, where llama.cpp manages ~143 / ~24

Against llama.cpp on the same hardware (CPU against CPU) it's quicker on most of what I've tried, and the gap grows with the prompt: on a 32k prompt it writes 2 to 10x faster. It isn't ahead everywhere, though. On my laptop llama.cpp reads long prompts slightly faster on GLM-4.7-Flash, and writes short replies a little faster on Moonlight and gpt-oss-20b. On the box, it reads short prompts a bit faster on the small dense Qwen3.5 models, until ToRc's NPU is on (then ToRc is ~2x). Full matrices, llama.cpp comparisons and every machine tested are in PERFORMANCE.md (link below).

Tested on: 2x Snapdragon X (base X1, 32 GB) boxes (1 box and 2), a Ryzen 7 7840HS laptop (Radeon 780M + RTX 4070), an i5-12600K desktop (RTX 3060) and an i9-13900K workstation (96 GB, Radeon AI PRO R9700 used for attention only). Everyday Windows installs with all my apps on them, not streamlined benchmark rigs on minimal Linux.

What I'm looking for:

- Testers, especially anyone with an X Elite or X2 Elite. This was all done on the base chip, so I'm curious what performance the bigger chips can manage.

- Mac people. As above, it builds and runs, but I can't test it myself, so numbers (and fixes) from more Macs would be brilliant.

- Linux on more real hardware, x86 especially.

- Holes. Find them and tell me what I can add or improve.

One question for you all: what is one feature you want from a local inference engine that nobody ships yet? I would genuinely like to try building it.

Basically I'm looking for collaboration with other people and projects, I've thoroughly enjoyed this project and I'd really like to see how far it can fly! 🕊️

Discrete GPU expert offload is on the list of my to-dos, but it was never the point of what I was trying to achieve, I use Strata on my big GPU workstation at work, and it's great, but that's a £5k machine, and out of reach of a lot of people. The point was making cheap hardware people are more likely to already have, actually useful - and use these NPUs that we now get with bigger models than the piddly ones we usually see.

Every model and machine I have measured is in PERFORMANCE.md 🙏

ToRc on GitHub


r/LocalLLaMA • • 8h ago

Question | Help What hardware for local Deepseek and GLM flash?

10 Upvotes

I guess its not just me but many people here always want more hardware and better models, so...

What hardware is needed to deploy Deepseek V4.1 Flash or GLM 5.3 Flash locally?
By deploy I mean, to be able to run multiple parallel coding agents, at least 256k context each.
I know you can "run" any of these MoE models on anything bigger than a regular PC but usually prompt processing struggles a lot more than generation so its not usable for agents.

To put that in perspective, I have a workstation with 512GB ddr4 ram, so DS v4.1 "almost" fits at Q4. What extra GPUs would I need to actually run it at speed with multiple sessions? Anyone with some hands on experience?

Been using Qwen 3.8 FN as my daily driver for a while, but always "why not go bigger" :)

Any ideas?


r/LocalLLaMA • • 15h ago

New Model Running SenseNova-Loop-9B nvfp4 looped transformer with 300-500 tok/s on a laptop 5090

Post image
53 Upvotes

Their architecture seems to translate quite well to existing kernels. They also have a small drafter. The speed numbers are aggregated across 8 parallel agents running. 256k context.

So far, I can't confirm it holds up well against Qwen 3.8 27B nvfp4, but I might have degraded it a bit, despite a calibration run. Here is a quick check of SeraphimSerapis/tool-eval-bench. Will do some more tests tomorrow, the 9B leaves a lot of vram free for FP8 tests.

tool-eval-bench runs (temperature 0, NVFP4 weights):

Test Qwen3.8 27B Swift T0 SenseNova-Loop-9B T0 SenseNova-Loop-9B T1
Standard (69) 98 90 91
Hard mode (35) 87 79 83
Error recovery 83 83 83
Context and state 100 75 85
Autonomous planning 67 67 83
Structured output 100 92 75
Safety 100 96 92
Hard: state across turns 0 50 83
Hard: tool failures 80 60 60
Hard: tenant separation 100 100 100
Hard: applying policy 100 100 100
Hard: injection 100 50 50

FP8 KV, DFlash2, T0 = temperature 0, T1 = temperature 1.0, top_p 0.95

Hardware: Intel Core 9 290hx, 64gb ram, 1TB nvme5 SSD, rtx 5090 laptop with 24gb vram.


r/LocalLLaMA • • 16m ago

Question | Help Swap 2 x 3090 for 2 x r9700?

• Upvotes

Currently i have the option to sell both my used 3090s and purchase 2 new r9700 at the same price. Should i?

Funny thing is that I did not even use these 3090 together for long as I couldn't find a way to tackle the heat management problem, swapped to dual 5060ti 16gb for inference while 3090s collects dust.

Is current rocm support good enough compared to cuda? Is the trade off in bandwidth worth the extra vram? The R9700 cooling will be workstation-style, so the heat should be more manageable.


r/LocalLLaMA • • 17m ago

Resources Trained a small decision model on Qwen3-0.6B + LoRA. Soft labels over the answer tokens helped; a custom head didn't.

Thumbnail
gallery
• Upvotes

I wanted to see what's inside decision models (Jev and the open-source alternatives that followed), so I built a minimal one and trained it on the typed-decisions benchmark.

Setup

A decision model reads a state plus several typed questions (choice, score, yes/no) and returns a probability for every allowed answer in one forward pass, without generating text.

Backbone: Qwen3-0.6B-Base, LoRA r=16, alpha 32, on all attention and MLP projections.

Packing: the state appears once, and each question is a branch that attends only to the state and itself, with position ids restarting after the state. I tested that each question gets the same hidden states it would get in its own sequence. Packing five questions against one state cut tokens processed by 2.1x and time per case from 329 to 175 ms on the T4.

Hardware: a free Colab T4, with gradient checkpointing to fit in memory.

Results

test, 400 cases / 2,000 decisions)

Model Accuracy KL Brier ECE
Zero-shot baseline 0.43 0.63 0.35 0.25
Fine-tuned baseline 0.76 0.27 0.11 0.12
Baseline with our loss 0.78 0.10 0.05 0.02
Decision model 0.78 0.09 0.05 0.02

All logit-readout rows are scored on the softmax over the answer tokens, so the full-vocab run isn't penalized at eval for probability on other tokens.

Going from hard labels over the full vocab to soft labels over the answer tokens took KL from 0.27 to 0.10 and ECE from 0.12 to 0.02. Accuracy gained 2 points on test and 5 on validation. That step changed two things at once, so I can't say how much came from the soft labels and how much from restricting the softmax.

Temperature scaling fixes most of the SFT baseline's overconfidence (validation ECE 0.134 to 0.027), but it still ends up worse than the soft-label models with no scaling at all (0.011 and 0.019).

Replacing the LM-head readout with a decision head made no measurable difference. On test they're within 0.01 on every metric, and on validation every gap falls inside its paired bootstrap 95% interval.

The soft-label part is knowledge distillation: the targets are the average of three teacher labelings per case, not one-hot labels.

Caveats

Each model is a single training run, so the intervals cover variation across test cases, not across seeds.

Accuracy is agreement with a ~4B teacher, not ground truth. The dataset card puts teacher self-agreement at 0.735 and treats ~0.75 as saturation, so 0.78 is just past that point.

These models are fine-tuned on the benchmark's train split, so they're specialists. Comparing them with zero-shot models like Jev isn't apples to apples.

I'd read KL and Brier before ECE on this benchmark. My ECE compares confidence with the gold probability of the predicted answer, so it isn't comparable to the card's ECE.

Things that cost me time

transformers 5 eager attention misreads a boolean 4D mask. An additive float mask in the backbone's dtype works under every attention implementation.

Unused option slots hold -inf, and the gradient of -inf / T is NaN during temperature fitting even where the probability is zero. I replaced them with a large finite value.

On a T4, bfloat16 only runs through emulation, and the coarse rounding produced exact ties between answer probabilities in my zero-shot eval. The repo now only defaults to bf16 on compute capability 8.0 and up.

The open question for me is whether soft labels or the restricted softmax did more of the work. The config for the soft-label full-vocab run is in the repo if anyone wants to try it before I get to it.

Code: https://github.com/anmorgan24/decision-model

Write-up: https://www.comet.com/site/blog/train-a-decision-model/

Disclosure: I work at Comet. The repo can log training to Comet and run extra evals in Opik, but training and scoring work without either.


r/LocalLLaMA • • 7h ago

Discussion LFM3-VL-2B hallucinations - does anyone have first-hand experience?

Post image
27 Upvotes

LFM3 benchmarks are out on AA and…. huuuuuuh….. this is NOT something you want to see for a vision model that fits on a phone.

“The mushroom in this picture is a turgida deliciousa, awesome in risottos and on toasted bread. ” (it’s an amanita phalloides, deadly poisonous).

“This sign says that you must not shoot any photos beyond this point” (the sign says “trespassers will be shot”).

etc. etc.

Can anyone with actual experience with this model confirm/deny this score?


r/LocalLLaMA • • 5h ago

Discussion Qwen3.8-27B LoRA for OpenCode (xhigh thinking-level accuracy with thinking off, GGUF included)

Post image
67 Upvotes

Hey there folks!

Aritra here (again) from Hugging Face. We RL-trained a small LoRA adapter (r=32, ~300 MB) on a frozen Qwen3.8-27B, inside the OpenCode coding agent. It's the second of our harness adapters, after pi that we have released yesterday.

Run the model locally like so:

hf download ggml-org/Qwen3.8-27B-GGUF Qwen3.8-27B-Q8_0.gguf --local-dir .
hf download harness-hub/Qwen3.8-27B-opencode-adapted-GGUF Qwen3.8-27B-opencode-lora-F16.gguf --local-dir .

llama serve -m Qwen3.8-27B-Q8_0.gguf --lora Qwen3.8-27B-opencode-lora-F16.gguf \
    --chat-template-kwargs '{"enable_thinking": false}' -c 65536 --port 8080 --alias qwen3.8-opencode

r/LocalLLaMA • • 32m ago

Question | Help what model would you recommend for cpu only on windows 48gb ram

• Upvotes

OK. so I have a 5060ti with i use for various DiT inference engines like comfyui and audio.cpp. i have an 8845hs cpu with 64gb DDR5 but as this is my main working pc some of the ram is used on browser and other running apps.

Now what I want is a model to run basic stuff like create and run a script to run a folder of audio files through audio.cpp stem separation and transcribing or write a bunch of prompts for klein and generate on my comfyui. I normally use opencode builtin and openrouter free api for this kind of stuff but occasionally when credit runs out i need an alternative.

I have llama.cpp installed so with the right flag it should restrict itself to ram but which model should I go for?