r/LocalLLaMA • • 5h ago

Resources AMA Announcement: Tencent Hy, the Team Behind Hy4 & Hy4 flash (Friday, Nov 20, 8–10 AM PST)

Post image
76 Upvotes

Hi r/LocalLLaMA 👋

We’re welcoming the Tencent Hy team for an AMA this Friday, November 20, from 8–10 AM PT!

Fresh off launching Hy4 and Hy4 flash, their research team—including Shunyu Yao—will join us to discuss how the models were built, inference speed and cost, and real-world applications, with extra perks for great questions!

⚠️ Note: The AMA itself will be hosted in a separate thread. Please don’t post questions here.


r/LocalLLaMA • • 1m ago

Question | Help Jetson Thor vs RTX 5070 Ti: Unexpected LLM Inference Performance

1 Upvotes

Hi everyone,

I have a question that might sound silly, but I haven't been able to find an answer.

As far as I understand, memory bandwidth is the primary bottleneck for a model's inference performance.

With my 5070 Ti (896 GB/s), if I run a Qwen3.8 27B Q4 model, I can roughly estimate the performance as 896 / 13.5 = ~66 tok/s, which is pretty much what I actually get (without MTP or DFlash).

Recently, however, I had the chance to try a Jetson Thor (very similar to the DGX Spark). Based on its specs, I would expect 273 / 13.5 = ~20 tok/s, but I'm only getting around 12 tok/s.

Even more surprisingly, with DFlash, which delivers exceptional performance gains on discrete GPUs, I'm barely getting a 2x speedup here, only slightly better than MTP.

Does anyone know what might be causing this performance gap on this device?

Thanks!


r/LocalLLaMA • • 16m ago

Question | Help Swap 2 x 3090 for 2 x r9700?

1 Upvotes

Currently i have the option to sell both my used 3090s and purchase 2 new r9700 at the same price. Should i?

Funny thing is that I did not even use these 3090 together for long as I couldn't find a way to tackle the heat management problem, swapped to dual 5060ti 16gb for inference while 3090s collects dust.

Is current rocm support good enough compared to cuda? Is the trade off in bandwidth worth the extra vram? The R9700 cooling will be workstation-style, so the heat should be more manageable.


r/LocalLLaMA • • 17m ago

Resources Trained a small decision model on Qwen3-0.6B + LoRA. Soft labels over the answer tokens helped; a custom head didn't.

Thumbnail
gallery
1 Upvotes

I wanted to see what's inside decision models (Jev and the open-source alternatives that followed), so I built a minimal one and trained it on the typed-decisions benchmark.

Setup

A decision model reads a state plus several typed questions (choice, score, yes/no) and returns a probability for every allowed answer in one forward pass, without generating text.

Backbone: Qwen3-0.6B-Base, LoRA r=16, alpha 32, on all attention and MLP projections.

Packing: the state appears once, and each question is a branch that attends only to the state and itself, with position ids restarting after the state. I tested that each question gets the same hidden states it would get in its own sequence. Packing five questions against one state cut tokens processed by 2.1x and time per case from 329 to 175 ms on the T4.

Hardware: a free Colab T4, with gradient checkpointing to fit in memory.

Results

test, 400 cases / 2,000 decisions)

ModelAccuracyKLBrierECE
Zero-shot baseline0.430.630.350.25
Fine-tuned baseline0.760.270.110.12
Baseline with our loss0.780.100.050.02
Decision model0.780.090.050.02

All logit-readout rows are scored on the softmax over the answer tokens, so the full-vocab run isn't penalized at eval for probability on other tokens.

Going from hard labels over the full vocab to soft labels over the answer tokens took KL from 0.27 to 0.10 and ECE from 0.12 to 0.02. Accuracy gained 2 points on test and 5 on validation. That step changed two things at once, so I can't say how much came from the soft labels and how much from restricting the softmax.

Temperature scaling fixes most of the SFT baseline's overconfidence (validation ECE 0.134 to 0.027), but it still ends up worse than the soft-label models with no scaling at all (0.011 and 0.019).

Replacing the LM-head readout with a decision head made no measurable difference. On test they're within 0.01 on every metric, and on validation every gap falls inside its paired bootstrap 95% interval.

The soft-label part is knowledge distillation: the targets are the average of three teacher labelings per case, not one-hot labels.

Caveats

Each model is a single training run, so the intervals cover variation across test cases, not across seeds.

Accuracy is agreement with a ~4B teacher, not ground truth. The dataset card puts teacher self-agreement at 0.735 and treats ~0.75 as saturation, so 0.78 is just past that point.

These models are fine-tuned on the benchmark's train split, so they're specialists. Comparing them with zero-shot models like Jev isn't apples to apples.

I'd read KL and Brier before ECE on this benchmark. My ECE compares confidence with the gold probability of the predicted answer, so it isn't comparable to the card's ECE.

Things that cost me time

transformers 5 eager attention misreads a boolean 4D mask. An additive float mask in the backbone's dtype works under every attention implementation.

Unused option slots hold -inf, and the gradient of -inf / T is NaN during temperature fitting even where the probability is zero. I replaced them with a large finite value.

On a T4, bfloat16 only runs through emulation, and the coarse rounding produced exact ties between answer probabilities in my zero-shot eval. The repo now only defaults to bf16 on compute capability 8.0 and up.

The open question for me is whether soft labels or the restricted softmax did more of the work. The config for the soft-label full-vocab run is in the repo if anyone wants to try it before I get to it.

Code: https://github.com/anmorgan24/decision-model

Write-up: https://www.comet.com/site/blog/train-a-decision-model/

Disclosure: I work at Comet. The repo can log training to Comet and run extra evals in Opik, but training and scoring work without either.


r/LocalLLaMA • • 26m ago

Discussion The frontier of AI has stalled. Local LLM will too.

0 Upvotes

I’m fully prepared to be wrong about this, but I expect very few leaps in capability in local LLM. I don’t mean there isn’t more to be gained with speed and size, but actual capability has hit a local plateau. The difference between a 500B model and 9B model has shrunk and will continue to.
Qwen4 will be a small disappointment. The post training is taking much longer because they can’t squeeze any juice out of the Claude and OpenAI models because those models haven’t improved in ways that most people can see. Enterprise customers are paying through the nose for tokens and getting fed up quickly and moving to local Chinese models because of cost. DeepSeek is getting overwhelmed and is basically out of compute it seems.
I am not really upset because flash next is actually good enough for me most of the time and Claude subscription is still cheap enough.
I hope everyone gets their small model that is good enough for them, but I don’t think the massive home LLM will be heads and shoulders over them like they were one year ago.
Source: I’m an engineer working with LLM


r/LocalLLaMA • • 32m ago

Question | Help what model would you recommend for cpu only on windows 48gb ram

3 Upvotes

OK. so I have a 5060ti with i use for various DiT inference engines like comfyui and audio.cpp. i have an 8845hs cpu with 64gb DDR5 but as this is my main working pc some of the ram is used on browser and other running apps.

Now what I want is a model to run basic stuff like create and run a script to run a folder of audio files through audio.cpp stem separation and transcribing or write a bunch of prompts for klein and generate on my comfyui. I normally use opencode builtin and openrouter free api for this kind of stuff but occasionally when credit runs out i need an alternative.

I have llama.cpp installed so with the right flag it should restrict itself to ram but which model should I go for?


r/LocalLLaMA • • 55m ago

Resources Windows OS SLM

Thumbnail
huggingface.co
8 Upvotes

r/LocalLLaMA • • 1h ago

Resources Nvidia driver 621.24 leads to constant writing of drives

Thumbnail
troublegrid.com
110 Upvotes

r/LocalLLaMA • • 1h ago

Discussion We evaluated DeepSeek V4.1 Flash 1017 and Kimi K3.1 Pro on LongTail-SWE

Post image
36 Upvotes

Last week we shared LongTail-SWE, 202 real GitHub tasks in 13 less common languages. Many of you asked for DeepSeek V4.1 Flash 1017, so we ran it along with Kimi K3.1 Pro.

DeepSeek V4.1 Flash 1017: 76%, up from 57% for V4 Flash.
Kimi K3.1 Pro: 77%, the top open-weight model, with 5.6M tokens per attempt.

We also re-ran Kimi K3 with a 1M context instead of 262K (now 72%, +8 pts) and fixed tests in a few tasks.

Leaderboard: longtail-swe.com


r/LocalLLaMA • • 2h ago

News SSDs, laptops, and phones may get cheaper as consumer NAND spot pricing drops 20% — Bank of America expects contract price drops, report cites oversupply in consumer market amid declining demand and growing Chinese competition

Thumbnail
tomshardware.com
185 Upvotes

r/LocalLLaMA • • 3h ago

News [Reuters] Trump administration weighs forcing US firms to disclose Chinese AI use

Thumbnail reuters.com
83 Upvotes

r/LocalLLaMA • • 3h ago

New Model Goes further with Koharu, the local-first manga translator with AI agents built in!

104 Upvotes

Hi LocalLLaMA,

I have spent more than two months rebuilding Koharu to be the most powerful manga translator!

TLDR: https://github.com/koharu-rs/koharu (MIT or Apache 2.0, at your choice)

It's been a while since I posted the Koharu update here. This time, I have added more features and built an established editor to make it feel more like Photoshop or Clip Studio Paint.

You can type directly onto the page, restyle selected characters, and reshape text frames. There are also layers, history, marquee and lasso selection, Generative Fill, and a Remove brush. The canvas is GPU-rendered now, so it stays smooth even on big chapters.

Everything still runs locally. LLMs run in-process through llama.cpp (Index-Translate 9B by default, plus Gemma 4, Qwen 3.x and LFM3), and inpainting uses LaMa, or FLUX.3 Klein / Qwen-Image through stable-diffusion.cpp. If you already run Ollama or LM Studio, you can point Koharu at that instead. On my machine, a page goes from raw to lettered in about 1 second once the models are loaded.

I also added an agent you can ask to proofread or fix pages. It's exposed over MCP too, so you can drive Koharu from Claude Code or Codex if you'd rather.

It works on CUDA, ROCm, Metal, and Vulkan.

I'd like to hear your feedback or any form of suggestion!


r/LocalLLaMA • • 3h ago

Discussion Local TTS for Indian languages: anything good yet?

0 Upvotes

Blue TTS v3 looks solid (MIT, 388 MB, about 50x real-time on CPU), but it's 7 languages and I'm not sure Telugu or Hindi is in there.

For anyone building voice stuff for Indian languages, what are you actually running locally? Is anything close to usable for long-form, like reading out a lesson, or is it still cloud APIs for now?


r/LocalLLaMA • • 4h ago

Resources ToRc: a from-scratch Rust engine running Qwen3.8 Flash Next (Q4), a 107 GB model, at 67 tok/s prompt / 17 gen (on a 16k prompt) on two £300 Snapdragon boxes with 64 GB of RAM between them. CPU + NPU + iGPU, in a 13 MB zip

Post image
28 Upvotes

So this started because I bought two of the base Snapdragon X boxes for some software and build testing at work (the cheap ones, about £300 each, not the Elite) and then they were sitting unused. With fast LPDDR5x (8448 MT/s) in them I thought I'd have a play and see if I could run proper models on them. I tried the usual engines and couldn't get anywhere near the speed I thought the hardware should do... So then my ADHD took over and I got carried away, and this is the result 🤣

ToRc is an inference engine written from scratch in Rust. Weeks of days and nights measuring, rewriting and measuring again went into it, and the numbers below are all measured on my own kit, different platforms etc... So I hope this proves useful to others 🙏🙃

ToRc on GitHub

Measured performance

If you can unzip a file, you can run your own model on your own machine. The download is a zip of about 13 MB (Windows x64, Windows on ARM, Linux x64 and Linux ARM, plus an experimental Mac one): unzip it, open ToRc, pick a model (a GGUF converts once, and it tells you how much space that takes first), press Auto-tune, and talk. A few minutes from download to chatting, nothing installed, and the app keeps everything (settings, chats, the lot) in that one folder. The only thing outside it is what the engine learns about your machine (Auto-tune's results and the like), in %LOCALAPPDATA%\torc or ~/.cache/torc.

What it can do:

- Uses the CPU, the NPU and the integrated GPU at the same time where it's present or faster, per model. Or just NPU + iGPU if you want to leave the machine usable while it runs.

- Tunes itself to your machine, and every setting can be overridden if you want to fiddle / let an agent try to wring every last token of performance for your particular setup.

- Speaks the OpenAI and Anthropic APIs, so Claude Code, OpenCode and anything else that talks to those just works. Full tool calling and MCP is also built in, and the desktop app has its own chat, tools and MCP servers if you just want to talk to it.

- Runs on Windows and Linux, x86 and ARM. (Linux: one user runs it on bare metal on a Snapdragon box under Ubuntu, and I've run it under WSL2, so tell me how it goes on yours.) There's also an EXPERIMENTAL Mac zip (Apple Silicon): one person has built and run it from source, and GitHub builds the zip, but I don't have a Mac to test it myself.

- It converts to its own file format for efficient SSD streaming where the model won't fit into memory and can split one model across multiple machines. I've tested two boxes over USB4; in principle it's N boxes. You pool their RAM, memory bandwidth and compute for enhanced performance.

Headline numbers, engine timed, read (prompt) / write (reply) tok/s:

- Qwen3.8 Flash Next, a 107 GB model, on two Snapdragon boxes with 64 GB of RAM between them (the experts that don't fit stream off the SSDs): ~67 / ~17.5 on a 16k prompt, ~64 / ~21 on a short one

- Qwen3.6 35B A3B, 4k prompt, two Snapdragon boxes: ~406 / ~54 (one box: ~283 / ~56)

- Same 35B on an i9-13900K (8 threads, CPU only), 4k prompt: ~243 / ~44, where llama.cpp manages ~143 / ~24

Against llama.cpp on the same hardware (CPU against CPU) it's quicker on most of what I've tried, and the gap grows with the prompt: on a 32k prompt it writes 2 to 10x faster. It isn't ahead everywhere, though. On my laptop llama.cpp reads long prompts slightly faster on GLM-4.7-Flash, and writes short replies a little faster on Moonlight and gpt-oss-20b. On the box, it reads short prompts a bit faster on the small dense Qwen3.5 models, until ToRc's NPU is on (then ToRc is ~2x). Full matrices, llama.cpp comparisons and every machine tested are in PERFORMANCE.md (link below).

Tested on: 2x Snapdragon X (base X1, 32 GB) boxes (1 box and 2), a Ryzen 7 7840HS laptop (Radeon 780M + RTX 4070), an i5-12600K desktop (RTX 3060) and an i9-13900K workstation (96 GB, Radeon AI PRO R9700 used for attention only). Everyday Windows installs with all my apps on them, not streamlined benchmark rigs on minimal Linux.

What I'm looking for:

- Testers, especially anyone with an X Elite or X2 Elite. This was all done on the base chip, so I'm curious what performance the bigger chips can manage.

- Mac people. As above, it builds and runs, but I can't test it myself, so numbers (and fixes) from more Macs would be brilliant.

- Linux on more real hardware, x86 especially.

- Holes. Find them and tell me what I can add or improve.

One question for you all: what is one feature you want from a local inference engine that nobody ships yet? I would genuinely like to try building it.

Basically I'm looking for collaboration with other people and projects, I've thoroughly enjoyed this project and I'd really like to see how far it can fly! 🕊️

Discrete GPU expert offload is on the list of my to-dos, but it was never the point of what I was trying to achieve, I use Strata on my big GPU workstation at work, and it's great, but that's a £5k machine, and out of reach of a lot of people. The point was making cheap hardware people are more likely to already have, actually useful - and use these NPUs that we now get with bigger models than the piddly ones we usually see.

Every model and machine I have measured is in PERFORMANCE.md 🙏

ToRc on GitHub


r/LocalLLaMA • • 4h ago

Discussion You'd never guess who opened up their checkbook to buy all the ram

146 Upvotes

Nvidia's commitment to buy ram is near $400 billion. https://x.com/jukan05/status/2122805536118702121 . https://www.reddit.com/r/investing/comments/1xz9eob/nvidia_just_added_111_billion_of_supply/

Change over 3 quarters: $119B, $279B, $390B. One would think that $390B would buy a lot of memory fabrication plants.


r/LocalLLaMA • • 4h ago

New Model SenseNova-Loop-9B official GGUFs and a 4-bit AWQ are up (5.8 GB at Q4_K_M), plus their llama.cpp fork to run them

Post image
19 Upvotes

The official quants for SenseNova-Loop-9B (the 9B Looped Transformer from earlier this week) are here: https://huggingface.co/sensenova/SenseNova-Loop-9B-GGUF

The repo has Q4_K_M, Q5_K_M, Q8_0 and BF16 (sizes in the screenshot), plus a 0.9 GB mmproj for image input.

The sensenova_loop architecture isn't in mainline llama.cpp yet (issue #31571 is open), so for now these run on the fork they published, OpenSenseNova/llama.cpp. The GGUF card pins the commit, and the card's command once it's built is:

llama serve -hf sensenova/SenseNova-Loop-9B-GGUF:Q4_K_M -ngl 99 -c 4096 --port 8000

There's also a 4-bit AWQ, with a matching MTP draft model in the same repo: https://huggingface.co/sensenova/SenseNova-Loop-9B-AWQ

Their README at https://github.com/OpenSenseNova/SenseNova-Loop puts total GPU memory (4-bit weights plus an 8-bit KV cache) at about 10 GiB for 64K context and 15 GiB for 256K.


r/LocalLLaMA • • 4h ago

Question | Help Honest question: how much RAM do you recommend for running local LLMs alongside coding agents

4 Upvotes

I’m a huge AI enthusiast but I’m running into limits with 32GB DDR5. Between running local models and using coding agents (which fire off scripts and changes super fast), my RAM fills up quickly. How much RAM are you running, and what would you recommend for a smooth setup? Bonus points if you share your GPU/VRAM too.


r/LocalLLaMA • • 5h ago

Discussion Qwen3.8-27B LoRA for OpenCode (xhigh thinking-level accuracy with thinking off, GGUF included)

Post image
65 Upvotes

Hey there folks!

Aritra here (again) from Hugging Face. We RL-trained a small LoRA adapter (r=32, ~300 MB) on a frozen Qwen3.8-27B, inside the OpenCode coding agent. It's the second of our harness adapters, after pi that we have released yesterday.

Run the model locally like so:

hf download ggml-org/Qwen3.8-27B-GGUF Qwen3.8-27B-Q8_0.gguf --local-dir .
hf download harness-hub/Qwen3.8-27B-opencode-adapted-GGUF Qwen3.8-27B-opencode-lora-F16.gguf --local-dir .

llama serve -m Qwen3.8-27B-Q8_0.gguf --lora Qwen3.8-27B-opencode-lora-F16.gguf \
    --chat-template-kwargs '{"enable_thinking": false}' -c 65536 --port 8080 --alias qwen3.8-opencode

r/LocalLLaMA • • 5h ago

News AI-powered Sashiko introduces experimental patch series design assessment

Thumbnail
phoronix.com
5 Upvotes

r/LocalLLaMA • • 6h ago

News Splash 1.7.0 Released - Even faster decode on Apple Silicon

Thumbnail
github.com
23 Upvotes

New version of Splash just released. The biggest improvement from previous version is a noticeable speed-up in decoding. It's even faster now.

BTW, this isn't just another slop engine. It's coming from the researchers behind DFlash and DFlash2, who also operate one of the fastest API providers for open models. These guys clearly know what they're doing.


r/LocalLLaMA • • 7h ago

New Model sensenova/SenseNova-Loop-9B-GGUF · Hugging Face

Thumbnail
huggingface.co
45 Upvotes

probably this fork is required: https://github.com/OpenSenseNova/llama.cpp

Model Highlights

  • Shared looped block: 24 decoder layers are executed twice in sequence, producing 56 layer computations from 32 distinct layers per forward pass.
  • Two-stream mHC: learned mappings route and combine information between two residual streams within the looped block.
  • Hybrid attention: the decoder interleaves linear and full attention in a repeating four-layer pattern.
  • Long-context multimodal training: the training recipe includes text, images, code, and tool-use trajectories, with the context length extended to 512K tokens.
  • Low-memory deployment: With 4-bit weights and an 8-bit KV cache, multimodal inference at a 256K context length requires approximately 15 GiB of GPU memory.
  • Official quantized release: An official AWQ 4-bit version is available on launch day for easier deployment.

r/LocalLLaMA • • 7h ago

Discussion LFM3-VL-2B hallucinations - does anyone have first-hand experience?

Post image
27 Upvotes

LFM3 benchmarks are out on AA and…. huuuuuuh….. this is NOT something you want to see for a vision model that fits on a phone.

“The mushroom in this picture is a turgida deliciousa, awesome in risottos and on toasted bread. ” (it’s an amanita phalloides, deadly poisonous).

“This sign says that you must not shoot any photos beyond this point” (the sign says “trespassers will be shot”).

etc. etc.

Can anyone with actual experience with this model confirm/deny this score?


r/LocalLLaMA • • 7h ago

New Model Flint anyone?

14 Upvotes

I see a lot of lament about Qwen not making 3.8 35B (and I'm sad about it too) but I dont see anyone discuss Flint 40B, a new coding-oriented MoE model that seems to be atleast on par with qwen 3.6 35B, maybe a bit better on agentic tasks and coding. Very fast in my setup.

I'm still testing it myself, but I wonder if its not liked here or people just arent aware it exists? It performs admirably for me so far, my local model of choice for when I want good speeds. QFN is better ofcourse but for me that means low quants or low speeds.

Curious about you guys' experience with the Flint model.


r/LocalLLaMA • • 8h ago

Discussion Qwen 4 35B is not ruled out

Post image
699 Upvotes