r/LocalLLM • • 1m ago

Project High hardware costs have crushed me, so I built a way to understand what makes for a good price in today's market

1 Upvotes

As someone who's used fancy market-tracking tools for higher-priced assets (collector cars, real-estate, etc), it was a surprise to find that the same thing doesn't really exist for GPUs, memory, storage, etc. I use tools like that elsewhere to understand if a purchase I'm considering is above/below market for the quality of the offering, and whether I need to move fast when buying or if waiting a little longer could be helpful.

I was collecting data and using it to make buying decisions for myself, so I figured it might be useful to others... I'd love your feedback! Consider this just a proof of concept for now.

https://siliconcents.com/


r/LocalLLM • • 6m ago

Discussion My local harness hard at work on Qwen 3.8 Flash Next 560+ t/s

1 Upvotes

Built a delegation harness that allows me to do multiple local delegate streams running concurrently off of a single Claude Code session. The Frontier is doing the design and verification steps and the local model is working hard on the grunt work.

Seeing it run at C4 relentlessly is a beautiful sight.


r/LocalLLM • • 9m ago

Question Basalt/Starata anyone using parallel slot?

1 Upvotes

I want to use 2 slots for parallel batching but cant sure about perfomance, anyone using or tried? How was performance?

System: RTX 5090 + 96GB VRAM Windows


r/LocalLLM • • 12m ago

Research Opencode Self-Improvement

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 35m ago

Model Stuck on 12GB VRAM but need 27B-level agent reasoning? SenseNova-Loop-9B loops its layers to match Qwen3.8-27B, official GGUF & AWQ out now

Thumbnail
gallery
12 Upvotes

SenseNova released the full version of Loop 9B on November 17 and posted the AWQ and GGUF download links today
https://www.reddit.com/r/LocalLLM/comments/1xzf343/sensenovaloop9b_matches_qwen3827b_on/
Loop 9B went from 14.33 in preview to 36.67 on AutomationBench, tying Qwen3.8-27B on that benchmark in the team’s evaluation.
What caught my attention was how the model uses its 9B parameters. Each token goes through four layers, then a block of 24, then the same 24 layers again, and finally the last four. On the second pass, the block works on the output of the first pass. That gives you 4 + 24 + 24 + 4 = 56 layer computations while storing just 32 sets of layer weights. You pay for the second pass in compute, but you only have to load those weights once
Inside the loop, the model carries two residual streams. Each layer learns how to mix information from the two streams and how to write its result back. The paper calls this two-stream mHC. There’s a separate design choice that matters for long prompts: in each group of four layers, three use linear attention and one uses full attention. Only that fourth layer needs a full attention KV cache that grows with the prompt.

Architecture details: https://github.com/OpenSenseNova/SenseNova-Loop

For local inference, a 12 GB card looks like a reasonable place to try 64K context (for example RTX5070). With 4-bit weights, the team estimates about 9 GiB at 64K with a 4-bit KV cache, or 10 GiB with an 8-bit KV cache. Keeping the cache at 8-bit brings the estimate to 12 GiB at 128K and 15 GiB at 256K. That last configuration leaves little headroom on a 16 GB card. Actual peak usage depends on the engine and how much VRAM is free.


r/LocalLLM • • 47m ago

Discussion Thread for recommended LLM setups for common hardware

4 Upvotes

As a beginner, it seems like there is a lot of decentralized information on this sub for recommendations for model/engine/harness for different common hardware setups.

Would there be a possibility to have a pinned thread on this sub where people can post these recommendations in one place? Or even better have an ongoing poll for each common consumer system (M5 Max, M5U, Spark, etc), for people’s favorite configs.

It seems like something like this would also cut down a lot on new posts where people buy a new computer and want to know what to run.

Thanks in advance!


r/LocalLLM • • 59m ago

Question Latest llama.cpp build feel 10-15% faster with no config changes?

2 Upvotes

Not sure if anyone feels the same way I do, I just updated to latest build and noticed faster PP and TG in general running Qwen3.8 27B UD_Q4_K_XL. Not trying to throw benchmarks or stuff but anyone feels the same?


r/LocalLLM • • 1h ago

Project Nihil IDE, free alpha, a hardware simulation lab that supports Local LLMs and 100% offline mode out of the box

Thumbnail
youtube.com
8 Upvotes

r/LocalLLM • • 1h ago

Research How I cut my frontier token costs drastically on long horizon tasks without losing code quality.

Thumbnail
dennyvgeorge.substack.com
0 Upvotes

​I ve been testing everything for the past year. frontier APIs, Chinese models, local rigs. And the conclusion is pretty straightforward. There is no perfect model. If someone says there is one, it’s stupidly expensive, and once context bloats over a long horizon task, it degrades anyway. On the flip side, running cheap or local models end-to-end just produces silent architectural rot.

​So my solution at the core is simply division of labor. I dint invent any new technology. It’s just basic systems architecture applied to token economics, separating high-level reasoning from brute-force execution and verification.

The engineering recipe is listed in detail on my paper attached here. U can run the exact same set up as mine using this write up. Hope u guys find it helpful. Cheers.


r/LocalLLM • • 1h ago

Project SSD MoE at high speed - The wait is over!

37 Upvotes

The wait is over!

Introducing Run My AI & TokenWing.

An inference engine that lets you offload over 60% of routed experts of mixture of experts models into SSD, yet achieve very low streaming latency...

And a GUI wrapper aimed at providing the best local AI experience.

Benchmarks: Ling 3.0 Flash 120B MXFP4 running on 12Gb VRAM + 32GB DDR5 RAM laptop at 45-50 tok/s decode and 900 tok/s prefill

Qwen 3.8 Flash Next 3.5bpw running on 8GB VRAM + 32GB DDR5 RAM laptop at 35-40 tok/s and 600-700 tok/s prefill

Flint 40B UD-Q4_K_XL running on a laptop with 24GB DDR4 RAM + 4GB VRAM at 30 tok/s decode and 400 tok/s prefill

All without MTP! And benchmarks are by users*

check the website out: https://runmyai.org

Also, the autotuner is still work in progress...

A common question, does it degrade SSD?

Nope! it doesn't... There's a thing called TBW rating. Well, your SSD degrades only when it writes. And the engine only reads....


r/LocalLLM • • 1h ago

News Small course corrections for open LLMs

1 Upvotes

Why do open models still get passed over for closed ones?

What can make them reliable?

I think part of it is small stuff:

- like a model that rambles when you wanted a short answer

- keeps invoking tool calls when the last one failed and doesn't refactor the trajectories

- etc.

A full fine-tune feels like too much for that.

astro-weights is my attempt at it: small edits to an open model's weights, saved as a normal LoRA adapter.

If you want small and mid-size open models to get closer to the frontier ones this way, consider trying it.

It's live and open source: https://astro-weights.dev/

Would love for you to try it out and lmk what you think :D

Also would really appreciate a 🌟 on GitHub.


r/LocalLLM • • 1h ago

Discussion Can open source models silently infect your PC?

0 Upvotes

e.g. they are trained to use some open source libraries for your next project which include some sophisticated backdoors which are not easy to spot. You execute your software and boom!

Regards


r/LocalLLM • • 1h ago

Question How do you connect remotely to use another ram .

0 Upvotes

How do you connect remotely to another Windows machine and use the other machine to be able to run models in a pool? I know there are people who have tried this; how is the experience? Machine A 16gb ram ddr5 , and the other machine has 16gb ddr4 .


r/LocalLLM • • 1h ago

Discussion Did anyone tried to combine some local ai together

0 Upvotes

I don't know if it's possible or not but I was wondering if anyone tried to combine two or more ai models together like taking the good things from each local AI model and combining it to one nice size model


r/LocalLLM • • 1h ago

Project Running HY4 FLASH 220B

Post image
2 Upvotes

Still working on optimization!

https://github.com/oscarsantanaarias/Sneo


r/LocalLLM • • 1h ago

Discussion Looking to buy a laptop for local AI setup

0 Upvotes

Planning to buy within a budget of 1-1.2 lakhs
My plan us to learn to build and run ai agents locally with ollama as host to 7-14B parameter LLMs

Whats the best configuration i should go for!


r/LocalLLM • • 2h ago

Question Models for local web search

1 Upvotes

Hi everyone. I recently downloaded Unsloth Studio and seeing it has a native web search I was looking for a good model to make use of this functionality.

Hardware: 12600k / 5070TI / 32GB DDR4

Use case:

  • given a CSV of ~2000 movies, determine which movie belongs to a certain list that I will describe (e.g.: based on a true story, movies about time travel etc)
  • given a JSON with a list of ~1500 software, return a json formatted in a certain way (e.g.: name, brief description, "tags" - chosen from from a list that I'll provided (e.g.: open source, audio, downloader etc))

I think I'll need a lot of context, and I'd like a good speed (perhaps ~50 tk/s?)

Topic already read:

Models already tried:

  • qwen 9b
  • gemma4 12b

They're not bad, but they definitely gets some movies/tags wrong, despite me providing the software URL/IMDb id.

Thanks in advance


r/LocalLLM • • 2h ago

Discussion Kapitsa vs Splash on Qwen3.8-Flash-Next (Apple M5 Max, 128 GB) — in case you wondered. I sure did.

1 Upvotes

So, same machine, same prompts, 256K context on both, speculative decoding on both (MTP in Kapitsa, DFlash2 drafter in Splash).

  • Kapitsa 0.2.37 with the MLX oQ4e-mtp quant (99 GB on disk, 77 GB in memory)
  • Splash 1.7.0 with the Unsloth-style GGUF UD-Q4_K_XL-PLE8 (137 GB on disk, 92 GB memory cap, PLE table streamed from SSD)
MetricKapitsa (MLX)Splash (GGUF)
Decode, short prompt82 tok/s58 tok/s
Time to first token, 106K prompt88 s55 s
Prefill speed~1,200 tok/s~1,950 tok/s
Decode at 106K context65–72 tok/s70–72 tok/s
Model load7 s11 s

TL;DR: Kapitsa is ~40% faster at generating on normal chat prompts. Splash is ~60% faster at chewing through long prompts (big docs, agent contexts). At long context they decode about the same. Splash also fits a bigger quant in less RAM because it keeps the 52 GB per-layer embedding table on SSD.

Pick Kapitsa for chat, Splash for long-document / agent work. Both pick up the drafter automatically and both are stable at 256K.

EDIT: same tests on the other MLX quant I had on disk, both in Kapitsa

MetricUncensored oQ4e-mtp (99 GB)KQuant Q4 (100 GB)
Decode, short prompt82 tok/s72 tok/s
Time to first token, 106K prompt88 s106 s
Time to first token, 106K + 600-tok essay98 s118 s
Decode at 106K context65–72 tok/s64–73 tok/s
Memory in use77 GB78 GB

Same context (256K), same prompts, MTP speculative decoding on both. The uncensored oQ4e build is the faster of the two, probably thanks to its mixed 4/5/6/8-bit layout. No speed penalty for going uncensored.

EDIT 2: the 27B and a small MoE, same three prompts, same 256K context, all in Kapitsa

MetricFlash-Next oQ4e-mtpQwen3.8-27B q4Ornith-1.5-35B-A3B + DFlash2
Memory in use77 GB15 GB23 GB
Decode, short prompt82 tok/s61 tok/s190 tok/s
Time to first token, 106K prompt88 s194 s56 s
Decode at 106K context65–72 tok/s37–61 tok/s125–131 tok/s

The 27B is slower than Flash-Next on everything; being dense, a 106K prompt takes it more than twice as long. Ornith (35B-A3B MoE with a DFlash2 drafter) is 2–3x faster than Flash-Next in 23 GB, if its quality is enough for what you do.


r/LocalLLM • • 2h ago

Question What would you do with $10,000?

0 Upvotes

With holiday deals around the corner, and with ai tech sky rocketing in price, I’m curious what you would spend $10,000 on… do you think gpu’s are going to continue to get even more expensive? And are there any deals that your targeting this Black Friday and cyber Monday?


r/LocalLLM • • 2h ago

Discussion Running 125B Qwen3.8-Flash on a 16 GB M4 Mac at ~18 tok/s and 16K context — works right alongside llama.cpp (Kraits)

Thumbnail
reddit.com
2 Upvotes

r/LocalLLM • • 2h ago

Question Amd Threadripper - 4x 3060 12gb - 128gb ddr4 2666

3 Upvotes

Anyone with a similar build who can share some real tips and bench?


r/LocalLLM • • 2h ago

Model I fine-tuned Qwen3-4B into a "typed" decision model, it reads the answer from logits over your option set, so it can't answer off-menu (Apache-2.0 + live demo)

1 Upvotes

Sharing a small thing I built. It turns an LLM into a constrained decision-maker instead of a text generator.

The idea: you give it a situation, a question, and your own list of 2–26 options. It returns exactly one of them in a single forward pass the choice is read from the next-token logits over the option markers (A, B, C…), not generated as text. So the output is structurally guaranteed to be one of your keys. Nothing to parse, nothing to hallucinate outside the set.

I know constraining output to a set isn't new constrained decoding, classifier heads, and JSON-schema modes all do it. The constraint is the easy part. What I actually spent the effort on: a small LoRA (rank 8, q_proj/v_proj only) trained on contrastive counterfactual pairs so it reads the state instead of latching onto marker position, then merged back into the base. Plus a fitted temperature so the confidence is calibrated and you can gate act if conf ≥ 0.9 else escalate.

How it's made: base Qwen3-4B, behaviour merged in (only 72 weight matrices differ from base, everything else bit-identical). Loads with stock transformers. Also exposes an MCP tool so agents can call it directly.

Honest eval (this exact checkpoint):

- 81.6% on 500 held-out contrastive items; ECE 0.143 (hence the temperature).

- ~85% on a hard contrastive reasoning probe (double-negation, quantifier-negation, implicature, counterfactual…).

- I just stress-tested it on toxicity + PII classification. Clear cases: 12/12, confidence ~1.0.

Hard/adversarial cases: toxicity 3/6, PII 4/6. The genuinely ambiguous ones mostly landed in the low-confidence band (0.4–0.6) where the gate sends them to review but it has confident misses too: a sarcastic insult read as "clean" at 0.98, and a venue address flagged as personal "contact" at 0.98. Sarcasm and reported/quoted speech are its blind spots and I'm not hiding that.

- Known weak spots: multi-step arithmetic and Winograd-style coreference, and errors there are often confident gate and fall back.

Use cases I've found it good for: agent control flow (tool routing, RAG-vs-answer, escalation), guardrails (it refused a phishing draft and blocked rm -rf / on prod in my tests), ticket triage, intent, sentiment any fixed-label routing, since the options are defined per request so one model covers all of them.

Live demo (Hugging Face ZeroGPU, no setup has a prev/next example cycler incl. agentic cases):

https://huggingface.co/spaces/mo19788/mini-typesafe-demo

Weights (Apache-2.0):

https://huggingface.co/mo19788/mini-typesafe-qwen3-4b

Would genuinely like feedback on the approach and the calibration and if anyone has a labeled routing/moderation set they'd point it at, I'd love to see where it breaks.


r/LocalLLM • • 3h ago

Project 464M music model trained from scratch for €30 of rent on consumer GPUs (FP8, RTX 5090). Writes multi-instrument MIDI from English/Ukrainian prompts (477M version)

Post image
4 Upvotes

I have been recently independently working on a musicAR - a family of domain specific language models for symbolic music generation. It is best at continuation of songs from a few seconds of opening. I have validated it to other open sourced models, and got such results:

ModelParametersContinuation MRR (CLaMP 3)
musicAR464M0.81
SkyTNT tv2o-medium234M0.666
AMT large780M0.640
Music-Llama457M0.537

musicAR-L is ahead of AMT and SkyTNT in all 10,000 paired bootstrap draws. On a density/instrument-statistics distance, AMT is closer over 1,000 notes (0.097 vs 0.144). musicAR-L is closer over the first 200.

And similar comparison on the captions following. My metric is F - how good the captions are followed. Prompts come in pairs (opposite, fast/slow etc and instruments prompts like a solo piano vs the empty prompt). For each pair I measure that attribute in the generated notes, measure the difference and scale to [-1, 1.5]. F = 1 - the captions change the generation by notable amount (I have settled it myself). 0 - the captions are ignored.

Below is the table of comparisons with the other models on a short captions

ModelParametersF, English captionF, Ukrainian caption
musicAR-L-caption477M (+ frozen LaBSE encoder)1.070.95
MIDI-LLM1.72B0.60.09
Text2Midi250M0.540.06

These 2 learned from english captions only, so this comparison isnt fair. It mostly shows the good property of my model.

Running locally: Real time (24 notes/s for a band and 8 notes/s for slow music genree like solo piano)

ModelMinimum requirementsLaptop RTX 5060 (8 GB VRAM GDDR7), 1 stream
musicAR-L-caption4 GB of VRAM (used 3.04 GB on peak including LaBSE)90 notes/s
musicAR-L2 GB of VRAM (used 1.09 GB on peak) or around 3 GB RAM127 notes/s and 14.3 notes/s on laptop cpu (ryzen 9 ai hx 370, 32 GB DDR5)

Training:
musicAR-L: 8.3B notes seen (4.5B unique notes) in 41h on one rented RTX 5090, about 30 euros including generation parameters search. FP8 + torch.compile, Muon for the transformer blocks. AdamW for the rest, WSD scheduler at the end.
Captions fine tuning (musicAR-L-caption) +2.04B notes, 14h on rented 5090, plus a 20-min caption phase.
Everything else including fine tunes: Laptop GPU (rtx 5060).
Ablations: chained heads −0.41 nats/note vs independent heads (MLP + residual connection); Muon −0.54 vs AdamW in pretraining. (for mo results look at the model card)

Code: https://github.com/DmytroHilei/Music_generative_model
Weights: https://huggingface.co/Dimitri-206265/musicar-L-caption musicAR-L-caption
Weights: https://huggingface.co/Dimitri-206265/musicar-L musicAR-L
Try samples: https://musicar-ivory.vercel.app/samples

I would like to hear any feedback on the quality of the music from people close to music


r/LocalLLM • • 3h ago

Discussion 3 x AI 9700 build - 96gb VRAM

Thumbnail gallery
7 Upvotes

Any recommendation on what else I can try on this set up?
Love this as a hobby! Just fun building!