r/Qwen_AI • u/Fun-Muffin6049 • 16h ago
Discussion We got GTA 6 before Qwen 4 27b
We got GTA 6 before Qwen 4 27b
r/Qwen_AI • u/Fun-Muffin6049 • 16h ago
We got GTA 6 before Qwen 4 27b
r/Qwen_AI • u/DivideHorror3217 • 18h ago
You're acting like you're releasing Qwen5 or something
After carefully analyzing all the benchmarks, scaling laws, architectural improvements, and the 47326 posts predicting exactly 10%, 15%, or 23.7% performance gains, I’ve come to a groundbreaking conclusion:
Qwen 4 27b will most likely be better than Qwen 3.8 27b
And here’s where things get controversial:
Qwen 4 Flash Next might also be better than Qwen 3.8 Flash Next.
I know. Bold predictions.
According to my calculations, we could see anywhere between 0% and 100% improvement in certain benchmarks.
Please remember this post when the models release. You heard it here first.
r/Qwen_AI • u/zemondza • 5h ago
I'm just shocked at how fast this model can run on my local computer, sometimes the results reach up to 103 tokens per second through the cache, and the model as a whole holds around 40-50 and it has a window of 400 thousand tokens, I also gave them a task to reverse and it ate up 298 thousand tokens and the speed was around 34-45 tokens per second, I'm shocked
My laptop
RTX 5070 8 gb 115w
AMD Ryzen 7 350 ai
64 gb ddr5 5600
SSD M2 Samsung 990pro 2Tb
SSD SAMSUNG 1Tb
r/Qwen_AI • u/GapNew4766 • 16h ago
ran the same game prompt 3 ways in local ai agent to see how little qwen4-max you actually need now that it's closed.
the task: one html file, a canvas game called star courier. 3 levels, arrow keys, and a ?demo=1 mode where it plays itself to the end screen. a checker opens it in headless chrome and runs 10 checks: no console errors, keys move the rocket, demo scores points, reaches level 2, level 3, the end screen, etc.
the 3 setups:
same prompt, temp 0.2, 20 min limit, 2 runs each.
results (run 1 / run 2, api cost):
what happened:
so the 27b on its own is a coin flip between broken and pretty good. with max only planning and reviewing, it matched max alone: 10/10 both runs, for 1 to 9 cents instead of 13 to 21, with every line of code written locally. only 2 runs per setup, so a signal, not proof.
anyone else pairing a closed planner with a local qwen as the coder? which local model holds up best as the worker for you? (disclosure: i work on atomic agent so AMA)
r/Qwen_AI • u/Cod3Conjurer • 17h ago
Instead of waiting 400ms for an LLM to generate "Option A" token-by-token, I fine-tuned QJev-1.5B on a single laptop GPU across all 655k samples of SargeDev/jev-distill-corpus-v3.
It routes the final hidden state of Qwen2.5-1.5B into an MLP decision head, outputting calibrated probabilities over up to 10 choices in a single ~31ms forward pass.
Progressive Training:
tol=0.05): 62.53% (#1 in the sub-2B bracket, beating 4B models like jiwo-4b and decision-4b).Hugging Face (16-bit, LoRA, 4-bit): https://huggingface.co/AnkitNayak-dev/QJev-1.5B
r/Qwen_AI • u/veizik • 12h ago
Processing img j9s5oa7dem2i1...
What this is (so you can weight the numbers): a native inference engine + compiler we built from scratch — no Python and no CUDA toolkit at inference time — that runs your existing checkpoint. It's an early / experimental release, the engine itself is sealed (not open source), and the benchmark below is our own, first-party. I'm one of the people who built it (founder/team), so apply the usual discount for a vendor posting their own numbers. I've put the operating point on every row and the places where we're worse in bold.
The result: on Qwen3.6-35B-A3B (35B-param
MoE, ~3B active), our decode throughput beats llama.cpp Q4_K_M on the same GPU, same pod, at a comparable ~4-bit quant (ours = VQ-balanced lut4, theirs = Q4_K_M).
Operating point for every row below: autoregressive decode, 512 ctx, single-stream (batch=1), our measurement on the same machine.
| GPU | veizik (tok/s) | llama.cpp Q4_K_M (tok/s) | speedup |
|---|---|---|---|
| RTX 4090 | 272.5 | 184.5 | 1.48× |
| A100 80GB | 192.1 | 140.9 | 1.36× |
| RTX 5090 | 345.0 | 258.9 | 1.33× |
| H100 80GB | 268.8 | 215.2 | 1.25× |
Not speculative decoding (the thing this s
ub asks first): both sides are plain batch-1 autoregressive decode — no draft model, no MTP on either side. The gap is the engine, not a speculation trick.
The tradeoff (quant quality): at the same ~4-bit, our fidelity check puts us a touch behind Q4_K_M — kld_vs_q4km = 1.28 vs a BF16 reference, i.e. slightly less faithful. So: a throughput win at comparable bitrate with a small fidelity cost — want the most faithful 4-bit, Q4_K_M is still it. (llama.cpp also wins on hardware breadth and being open source.)
Other boundaries:
Fits on a 3090 too (supporting point, not the headline): the 35B-A3B is 21.44 GiB resident (balanced) on a single 24GB 3090 and decodes 194–2
21 tok/s (our measurement). Fitting a 35B on a 3090 is table-stakes — llama.cpp does it too — so I'm calling this a supporting point, not a brag. The llama.cpp 3090 figure I have is cited, not same-pod, so I'm deliberately not putting a multiplier on the 3090.
If you want to sanity-check the engine at sizes you can actually run locally, small dense Qwen3.5 on the 3090, same-pod vs llama.cpp (AR decode, single-stream):
| model | speedup vs llama.cpp (3090, same-pod) |
|---|---|
| 0.8B | 1.93× |
| 2B | 1.47× |
| 4B | 1.23× |
| 9B | ~par |
Other measured points (own numbers, different workloads — deliberately NOT paired with the llama.cpp table):
Why we built it: one engine, no Python glue and no CUDA toolkit to install at inference, that takes a checkpoint you already have and runs it — and the same engine also covers image, video and robotics, down to a Raspberry Pi CPU. That breadth under one runtime is the thing we couldn't get out of any single-model engine (llama.cpp / vLLM / NInfer / ik_llama), so we wrote our own.
Repro: our full own-number board — all these GPUs, every mode — is live at veizik.com/benchmarks, and the kit at github.com/veizikhq/veizik-cuda (cuda
_deploy/bench/, method in REPRODUCIBILITY.md) runs veizik run on your own GPU so you can clock the engine yourself. The llama.cpp rows are our own same-pod Q4_K_M measurement — run llama.cpp Q4_K_M on the same box and check the gap. Got a 4090 / 5090 / A100 / H100? Post your numbers — especially if the gap doesn't hold on your setup.
License: the engine is sealed / proprietary (not open source); it runs your open checkpoints.
Happy to get into kernel-level, quant-fairness or determinism questions in the comments.
r/Qwen_AI • u/LocalhostRonin • 20h ago
r/Qwen_AI • u/Fit-Blueberry-5390 • 7h ago
Basalt (https://github.com/jesdga95/basalt) is one of the fastest local engines for Qwen3.8-Flash-Next, but it only targets Blackwell (sm_120) and upstream does not take other architectures. I have two V100-SXM2 32GB, so I ported it to Volta (sm_70) as a fork, called **Lapilli**.
**Repo:** https://github.com/chris83-dev/lapilli-v100 (a Windows x64 build is in the release)
**What it is**
- Every newer PTX instruction Basalt uses (cp.async, ldmatrix, the Turing/Ampere mma shapes, bf16 conversions, clusters) gets a correct sm_70 equivalent, then the hot kernels get V100 paths: llama.cpp's dp4a MMQ for prompt products, Strata's mma.m8n8k4 prompt attention, and a rewritten router, decode experts and GDN recurrence.
- Needs CUDA 12.6 or newer (CUDA 13 dropped Volta).
- Comes with a chat web UI on top of Basalt's OpenAI/Anthropic-compatible server: conversations, reasoning levels, images, tool calls shown as cards, and a live monitor of the GPUs.
**Numbers** (2x V100-SXM2 32GB, i9-14900K, 96GB RAM, Windows 10, UD-Q4_K_XL pack, medians of 3 runs)
| | Lapilli | Strata v0.2.13, same machine |
|---|---|---|
| Decode, short answer | 91 tok/s | 58-72 tok/s |
| Decode, long answer with reasoning | 140 tok/s | 68-94 tok/s |
| Prompt processing, 4k tokens | 1,170 tok/s | 250-490 tok/s |
| Prompt processing, 8k / 16k tokens | 1,504 / 1,448 tok/s | |
Quality with Basalt's own 100-chunk gate: mean KLD 0.0493 and same top-1 92.75% (Basalt publishes 0.0496 and 92.63% for this pack on an RTX 5090).
**Caveats**
- Only tested on my machine (2x V100 32GB, Windows). Single card, 16GB or PCIe cards, Linux and NVLink setups are untested: reports from other V100 owners are welcome.
- Peer-to-peer is off by default on Volta: on my machine, direct copies between the two cards returned wrong data.
- NVFP4 packs need FP4 hardware, so use a K-quant pack (tested: UD-Q4_K_XL).
- It is a snapshot of Basalt as of November 15, shared as is.
All the credit for the engine goes to jesdga95 (Basalt) and Niko1221 (Strata), both MIT.
r/Qwen_AI • u/BrodyFriend • 5h ago
Hey guys, this is a repost from Local LLM thread, I figured maybe this is a good place to ask as well.
WhELP!!!
r/Qwen_AI • u/SteaknBakin • 9h ago
I have a 64 GB RTX Spark N1X laptop and wanted Qwen3.8-27B to run faster on it than LM Studio does. Of the three engines I tried (LM Studio, a vLLM port and Basalt), Basalt was the fastest, but it officially supports only x64 and the desktop RTX 50 cards. So I've published Emerald, a port for Windows on Arm and the N1X's GPU, with a prebuilt download so you don't need a compiler:
https://github.com/BenGoldfeder17/Emerald
Same laptop, same prompts, 32K context, thinking off, mean of two runs:
| Emerald | LM Studio |
|---|---|
| Essay, temperature 0.6 | 39.1 tok/s |
| Code, temperature 0 | 70.3 tok/s |
| First read of an 8.8k-token prompt | 2.9 s |
| Time to first token on a tool call | 0.4 s |
Caveats first:
What it adds for the N1X beyond the port:
The full 262K context works too: about 32 GiB of GPU memory, and a 212k-token prompt was read cold in 171 s.
Setup is: unzip, copy two cuBLAS DLLs from NVIDIA's archive, make a Python venv, download the 21 GB model pack, run basalt.exe serve. You get an OpenAI-compatible API on localhost.
All credit for the engine goes to Jesús Gulfo and the Basalt contributors. This is an unofficial fork, so please don't send him bug reports about it. If you have the same laptop I'd like to hear what numbers you get.
r/Qwen_AI • u/Voodmany • 10h ago
Built Mference 2.0 (building on NeelM0906/Mference and drumih/turbo-fieldfare) to run Qwen 3.6 35B-A3B (4-bit, 256 experts) comfortably on a 16 GB unified-memory Mac — without quantization collapse or swap storms.
v2.0.0 macOS .app + CLI binaries: https://github.com/fjw4cfv2fb-afk/Mference-2.0/releases/tag/v2.0.0voodmany): https://huggingface.co/voodmany/Qwen3.6-35B-A3B-4bit-MTP-Surgery-GTurbo25.0% (2/8) raw INT4 baseline → 100.0% (8/8) (+75.0 pts)16.47 tok/s avg, up to 18.36 tok/s (Swift coding) and 17.88 tok/s (AIME math)83.1% overall (up to 94.3% on AIME, 2.55 tokens emitted/round)<think> Tokens: 413.4 → 126.1 avg tokens (-69.5%)MTPAttachTool): Raw INT4 Qwen 3.6 kept getting trapped in endless <think> loops (6/8 timeouts before emitting a visible answer). Because MLX group-64 affine weights satisfy w = s * q + b, scaling weights by α transforms linearly into (α * s) * q + (α * b) in the BF16 scale/bias tables and BF16 norms — zero nibble unpacking, zero re-quantization, zero fine-tuning. Scaling late full-attention Q/K norms (1.03x), router temperature (0.96), late o_proj (0.98x), shared experts (1.025x), and co-adapting mtp.norm.weight fixed all 6 failures while keeping 83%+ MTP acceptance.guru87/Qwen3.6-35B-A3B-MTP draft block (~453 MB routed experts + 17 resident tensors), pairs it with a 4-slot Gated-DeltaNet rollback arena (so partial draft rejections rewind both KV and linear-attention recurrent state bit-for-bit cleanly), lossless topK=6 draft pruning, exact within-block top-2 EVICT logit-margin truncation, async-pipelined accept synchronization, and single-pass 6/5/4/3-gram PLD for verbatim code/tool-call spans.PreadExpertStreamer with prefill-tail warm start so decode step 1 hits warm cache slots, POSIX_MADV_RANDOM slab-only MTP mapping, plus a memory-bounded 2-slot LRU prompt cache and a native SwiftUI app + SwiftNIO OpenAI server (:8080) + Open WebUI launcher.Hardware reports welcome in GitHub Discussions! Repro benchmark harness (Scripts/eval_aa_scorecard.py) and full architecture writeup (MTP_WEIGHT_SURGERY_AND_ARCHITECTURE.md) are in the repo.
r/Qwen_AI • u/prochdavis • 14h ago
I tested r3lax's 4-bit FP16 conversion of DavidAU's Qwen 3.8 27B on my 32 GB M1 Max. Its scale and bias arrays used 1.681 GB of active MLX memory. Storing repeated pairs once per matrix reduced that to 0.644 GB. The four-bit weight codes stayed in the original checkpoint, and every scale and bias was restored exactly.
I wrote a scanner so it's easy to check another conversion before packing anything. I'd be interested in results from the same model quantized with a different group size. Please include the model ID or revision so we can make a meaningful comparison.
https://github.com/Davcca/mlx-weightpack https://github.com/Davcca/mlx-weightpack/blob/main/benchmarks/README.md
r/Qwen_AI • u/SpeculationAllTheWay • 1h ago
Hey everyone, we just released BaseQuant V1 for Qwen3.6-35B-A3B at Base Compute.
There are two builds: a 22.13 GB GGUF for llama.cpp, and a 23.08 GB .base bundle for baseRT that includes the vision tower and an MTP head for speculative decoding. In our held-out tests, the .base bundle had ~20% lower KL divergence from BF16 on long context and ~10% lower on agentic coding than a similarly sized Unsloth quant. The GGUF is on par overall.
Would love for you to try it and tell us how it holds up in your everyday workloads!
r/Qwen_AI • u/yeah280 • 22h ago
I make talking-head videos about science. I want someone with no background in the topic to understand the idea on the first watch. I have access to Qwen 4 Max, Magnific, and Higgsfield, and I’d like to try WAN 3.0 for short animated inserts. My budget for getting the workflow running is €2,000 (roughly US$2,300).
The goal is to teach the system my style once. After that, I want to hand it a raw recording and get back an edited video I can review, without writing a new brief every time. I’d also like help turning research notes into a script, but the hard part is deciding what to show and when.
If you’ve built something similar, what would you test on one raw clip before spending the rest of the budget?