r/Qwen_AI • • 8h ago

Funny My extremely realistic prediction for Qwen 4 27b and Qwen 4 Flash Next

139 Upvotes

After carefully analyzing all the benchmarks, scaling laws, architectural improvements, and the 47326 posts predicting exactly 10%, 15%, or 23.7% performance gains, I’ve come to a groundbreaking conclusion:

Qwen 4 27b will most likely be better than Qwen 3.8 27b

And here’s where things get controversial:

Qwen 4 Flash Next might also be better than Qwen 3.8 Flash Next.

I know. Bold predictions.

According to my calculations, we could see anywhere between 0% and 100% improvement in certain benchmarks.

Please remember this post when the models release. You heard it here first.


r/Qwen_AI • • 22h ago

Help πŸ™‹β€β™‚οΈ What should I give Qwen 4 Max so it can creatively edit my talking-head science videos?

0 Upvotes

I make talking-head videos about science. I want someone with no background in the topic to understand the idea on the first watch. I have access to Qwen 4 Max, Magnific, and Higgsfield, and I’d like to try WAN 3.0 for short animated inserts. My budget for getting the workflow running is €2,000 (roughly US$2,300).

The goal is to teach the system my style once. After that, I want to hand it a raw recording and get back an edited video I can review, without writing a new brief every time. I’d also like help turning research notes into a script, but the hard part is deciding what to show and when.

  1. What would you give Qwen up front: annotated reference edits, a style guide, a reusable graphics library, or something else?
  2. Which plugins, skills, or tools would you connect? Would you have Qwen write a timestamped edit plan and let a separate tool assemble and render it?
  3. What would you still review manually, especially to catch visuals that look convincing but explain the science incorrectly?

If you’ve built something similar, what would you test on one raw clip before spending the rest of the budget?


r/Qwen_AI • • 5h ago

Agent Qwen 3.8 flash next 125B 51tok/sec my laptop wow

77 Upvotes

I'm just shocked at how fast this model can run on my local computer, sometimes the results reach up to 103 tokens per second through the cache, and the model as a whole holds around 40-50 and it has a window of 400 thousand tokens, I also gave them a task to reverse and it ate up 298 thousand tokens and the speed was around 34-45 tokens per second, I'm shocked

My laptop

RTX 5070 8 gb 115w

AMD Ryzen 7 350 ai

64 gb ddr5 5600

SSD M2 Samsung 990pro 2Tb

SSD SAMSUNG 1Tb


r/Qwen_AI • • 23h ago

Funny Today is must be to day

127 Upvotes

r/Qwen_AI • • 17h ago

LLM I fine-tuned QJev-1.5B: A 31ms decision model on qwen2.5-1.5b instruct using 655k samples of jev-distill-corpus-v3 on an RTX 5060 laptop

Post image
7 Upvotes

Instead of waiting 400ms for an LLM to generate "Option A" token-by-token, I fine-tuned QJev-1.5B on a single laptop GPU across all 655k samples of SargeDev/jev-distill-corpus-v3.

It routes the final hidden state of Qwen2.5-1.5B into an MLP decision head, outputting calibrated probabilities over up to 10 choices in a single ~31ms forward pass.

Progressive Training:

  1. Decision Head First: Froze Qwen and trained only the MLP projection head to map hidden states into probability distributions.
  2. Joint Weight Training: Unfroze attention projections via QLoRA and jointly trained the backbone + head across all 655k samples (pegged at 5.9 GB VRAM).
  3. Consistency Tuning: Soft-target KL loss across permutation pairs to eliminate format bias.

Why Use It? (Key Advantages):

  • 10–20Γ— Faster Decisions: Single-pass ~31ms inference instead of slow autoregressive decoding.
  • Zero Schema Failures: Direct float probabilitiesβ€”no regex, JSON parsing, or retry loops needed.
  • Calibrated & Dynamic: True expected values across 2 to 10 choices via dynamic masking (not just binary yes/no).
  • Ultra-Lightweight: Runs in 1.07 GB VRAM in 4-bit, or as a 20 MB LoRA adapter if you already cache Qwen.

Benchmark (JevBench):

  • Approaching 27B Scale: Achieves 71.50% consistency (matching 9B server models at ~72% and close to 27B at 76.5%) while being 18Γ— smaller and 6Γ— faster.
  • Official Strict Benchmark (tol=0.05): 62.53% (#1 in the sub-2B bracket, beating 4B models like jiwo-4b and decision-4b).
  • Serving: Native support for Ollama, vLLM, llama.cpp / GGUF, and FastAPI.

Hugging Face (16-bit, LoRA, 4-bit): https://huggingface.co/AnkitNayak-dev/QJev-1.5B

GitHub: https://github.com/AnkitNayak-dev/QJev-1.5B


r/Qwen_AI • • 14h ago

Discussion Edit still removed...

0 Upvotes

Its been like month after Edit got removed... what was Qwen doing anyway? can i just edit the Response again like old time? where is the edit button now?... Seriously i like qwen but i wish we still had many option like keeping edit... Im just scared its not going to work anymore...


r/Qwen_AI • • 18h ago

Funny Release my Qwen4_27B_Q4_K_M.gguf

Post image
281 Upvotes

You're acting like you're releasing Qwen5 or something


r/Qwen_AI • • 1h ago

Model We just released BaseQuant V1 for Qwen3.6-35B-A3B. Would love to hear what you think

Thumbnail
huggingface.co
β€’ Upvotes

Hey everyone, we just released BaseQuant V1 for Qwen3.6-35B-A3B at Base Compute.

There are two builds: a 22.13 GB GGUF for llama.cpp, and a 23.08 GB .base bundle for baseRT that includes the vision tower and an MTP head for speculative decoding. In our held-out tests, the .base bundle had ~20% lower KL divergence from BF16 on long context and ~10% lower on agentic coding than a similarly sized Unsloth quant. The GGUF is on par overall.

Would love for you to try it and tell us how it holds up in your everyday workloads!


r/Qwen_AI • • 16h ago

Experiment qwen4-max plans, local qwen 3.8 27b writes all the code: 10/10 both runs, same as max alone. the 27b alone got 2 and 8

Post image
29 Upvotes

ran the same game prompt 3 ways in local ai agent to see how little qwen4-max you actually need now that it's closed.

the task: one html file, a canvas game called star courier. 3 levels, arrow keys, and a ?demo=1 mode where it plays itself to the end screen. a checker opens it in headless chrome and runs 10 checks: no console errors, keys move the rocket, demo scores points, reaches level 2, level 3, the end screen, etc.

the 3 setups:

  • local: qwen 3.8 27b (unsloth q4_k_xl) on llama.cpp, m5 max macbook pro 64gb, thinking off
  • max: qwen4-max over the api (aimlapi), default thinking
  • fusion: max writes the plan and reviews the result, the local 27b writes all the code

same prompt, temp 0.2, 20 min limit, 2 runs each.

results (run 1 / run 2, api cost):

  • qwen4-max: 10/10 and 10/10, $0.13 / $0.21
  • fusion: 10/10 and 10/10, $0.09 / $0.01
  • 27b alone: 2/10 and 8/10, $0

what happened:

  • max thought for about 3 min, then wrote the whole game (523 lines) in one go. atomic agent's verify step opened it in a browser and read the game state: level 1 β†’ 3, score 0 β†’ 330. done in 6 min.
  • 27b alone, run 1: also wrote it in one go. atomic agent flagged a syntax error the second the file was saved, but the 27b couldn't find it. it re-read the same line 6 times and stopped at 10 min. at least it said the game was broken instead of claiming it worked.
  • 27b alone, run 2: working game, but the demo dies in level 2. it kept rerunning the same failing browser check until the agent's loop guard blocked it, then ran out of time.
  • fusion, run 1: max spent under a minute on a plan, handed it to the 27b, which wrote a working 426 line game, then max reviewed it and sent back a few edits. the demo collects stars and plays all 3 levels to the end screen. 10/10.
  • fusion, run 2: the 27b's first version had a syntax error too, got flagged on save, and its rewrite fixed it. working game, demo plays through to the end screen. 10/10, and only one max call, so the whole run cost $0.01.

so the 27b on its own is a coin flip between broken and pretty good. with max only planning and reviewing, it matched max alone: 10/10 both runs, for 1 to 9 cents instead of 13 to 21, with every line of code written locally. only 2 runs per setup, so a signal, not proof.

anyone else pairing a closed planner with a local qwen as the coder? which local model holds up best as the worker for you? (disclosure: i work on atomic agent so AMA)


r/Qwen_AI • • 9h ago

LLM Qwen3.8-27B at 39-70 tok/s on an RTX Spark N1X laptop: prebuilt Basalt port for Windows on Arm (about 1.6-1.8x LM Studio on the same machine)

2 Upvotes

I have a 64 GB RTX Spark N1X laptop and wanted Qwen3.8-27B to run faster on it than LM Studio does. Of the three engines I tried (LM Studio, a vLLM port and Basalt), Basalt was the fastest, but it officially supports only x64 and the desktop RTX 50 cards. So I've published Emerald, a port for Windows on Arm and the N1X's GPU, with a prebuilt download so you don't need a compiler:

https://github.com/BenGoldfeder17/Emerald

Same laptop, same prompts, 32K context, thinking off, mean of two runs:

Emerald LM Studio
Essay, temperature 0.6 39.1 tok/s
Code, temperature 0 70.3 tok/s
First read of an 8.8k-token prompt 2.9 s
Time to first token on a tool call 0.4 s

Caveats first:

  • Different quantizations. Emerald runs the NVFP4 pack (QUASAR) with Basalt's speculative drafting; LM Studio ran the Q4_K_S GGUF. This compares speed, not quality, and I have not run a perplexity/KLD check on this build.
  • One laptop. Basalt's own test suite has not been run on ARM64.
  • Only the dense 27B. The MoE models (Flash-Next, DeepSeek) do not work in this port.
  • It is not the only N1X port. SolettaSolaris and yanmindi have ports with NEON CPU kernels that also run Flash-Next. Use those if you want that model or Linux. Emerald is for the 27B on Windows with the least setup.

What it adds for the N1X beyond the port:

  • CUDA's default wait kept one CPU core at 100% while generating. A new wait mode brings that to about 30% at the same speed.
  • The 2.4 GiB token embedding can sit in GPU memory instead of pinned RAM, which gave Windows back about 2.5 GB (the N1X's GPU memory is its own pool).
  • Periodic prompt checkpoints for the dense model: a new question about a document it had just read went from 2.7 s to 0.3 s.
  • Optional relaxed drafting for sampled replies: about 15% faster at temperature 0.6. It changes the sampling, so it is off unless you set it.

The full 262K context works too: about 32 GiB of GPU memory, and a 212k-token prompt was read cold in 171 s.

Setup is: unzip, copy two cuBLAS DLLs from NVIDIA's archive, make a Python venv, download the 21 GB model pack, run basalt.exe serve. You get an OpenAI-compatible API on localhost.

All credit for the engine goes to JesΓΊs Gulfo and the Basalt contributors. This is an unofficial fork, so please don't send him bug reports about it. If you have the same laptop I'd like to hear what numbers you get.


r/Qwen_AI • • 10h ago

Resources/learning Running Qwen 3.6 35B-A3B at 100% Pass@1 & up to 18.36 tok/s on a 16 GB Mac β€” Pure Swift + Metal SSD Expert Streaming, 19-Tensor MoE MTP, and Lossless INT4 Weight Surgery

Thumbnail
gallery
2 Upvotes

Built Mference 2.0 (building on NeelM0906/Mference and drumih/turbo-fieldfare) to run Qwen 3.6 35B-A3B (4-bit, 256 experts) comfortably on a 16 GB unified-memory Mac β€” without quantization collapse or swap storms.

Results on a 16 GB Mac (8-task Coding / AIME Math / GPQA / IFBench / Tool-Call suite, temp=0, 32 expert cache slots)

  • Pass@1 Accuracy: 25.0% (2/8) raw INT4 baseline β†’ 100.0% (8/8) (+75.0 pts)
  • Decode Speed: 16.47 tok/s avg, up to 18.36 tok/s (Swift coding) and 17.88 tok/s (AIME math)
  • MTP Acceptance: 83.1% overall (up to 94.3% on AIME, 2.55 tokens emitted/round)
  • Wasted <think> Tokens: 413.4 β†’ 126.1 avg tokens (-69.5%)

3 Things That Made the Difference

  1. Lossless INT4 Resident Weight Surgery (MTPAttachTool): Raw INT4 Qwen 3.6 kept getting trapped in endless <think> loops (6/8 timeouts before emitting a visible answer). Because MLX group-64 affine weights satisfy w = s * q + b, scaling weights by Ξ± transforms linearly into (Ξ± * s) * q + (Ξ± * b) in the BF16 scale/bias tables and BF16 norms β€” zero nibble unpacking, zero re-quantization, zero fine-tuning. Scaling late full-attention Q/K norms (1.03x), router temperature (0.96), late o_proj (0.98x), shared experts (1.025x), and co-adapting mtp.norm.weight fixed all 6 failures while keeping 83%+ MTP acceptance.
  2. Native 19-Tensor MoE MTP + Hybrid 6-Gram PLD: Attaches the guru87/Qwen3.6-35B-A3B-MTP draft block (~453 MB routed experts + 17 resident tensors), pairs it with a 4-slot Gated-DeltaNet rollback arena (so partial draft rejections rewind both KV and linear-attention recurrent state bit-for-bit cleanly), lossless topK=6 draft pruning, exact within-block top-2 EVICT logit-margin truncation, async-pipelined accept synchronization, and single-pass 6/5/4/3-gram PLD for verbatim code/tool-call spans.
  3. 16 GB Unified-Memory Survival: Streams routed experts from NVMe via PreadExpertStreamer with prefill-tail warm start so decode step 1 hits warm cache slots, POSIX_MADV_RANDOM slab-only MTP mapping, plus a memory-bounded 2-slot LRU prompt cache and a native SwiftUI app + SwiftNIO OpenAI server (:8080) + Open WebUI launcher.

Hardware reports welcome in GitHub Discussions! Repro benchmark harness (Scripts/eval_aa_scorecard.py) and full architecture writeup (MTP_WEIGHT_SURGERY_AND_ARCHITECTURE.md) are in the repo.


r/Qwen_AI • • 7h ago

News Basalt ported to NVIDIA V100: Qwen3.8-Flash-Next at 91 tok/s (140 tok/s on long answers) on 2x V100 32GB

5 Upvotes

Basalt (https://github.com/jesdga95/basalt) is one of the fastest local engines for Qwen3.8-Flash-Next, but it only targets Blackwell (sm_120) and upstream does not take other architectures. I have two V100-SXM2 32GB, so I ported it to Volta (sm_70) as a fork, called **Lapilli**.

**Repo:** https://github.com/chris83-dev/lapilli-v100 (a Windows x64 build is in the release)

**What it is**

- Every newer PTX instruction Basalt uses (cp.async, ldmatrix, the Turing/Ampere mma shapes, bf16 conversions, clusters) gets a correct sm_70 equivalent, then the hot kernels get V100 paths: llama.cpp's dp4a MMQ for prompt products, Strata's mma.m8n8k4 prompt attention, and a rewritten router, decode experts and GDN recurrence.

- Needs CUDA 12.6 or newer (CUDA 13 dropped Volta).

- Comes with a chat web UI on top of Basalt's OpenAI/Anthropic-compatible server: conversations, reasoning levels, images, tool calls shown as cards, and a live monitor of the GPUs.

**Numbers** (2x V100-SXM2 32GB, i9-14900K, 96GB RAM, Windows 10, UD-Q4_K_XL pack, medians of 3 runs)

| | Lapilli | Strata v0.2.13, same machine |

|---|---|---|

| Decode, short answer | 91 tok/s | 58-72 tok/s |

| Decode, long answer with reasoning | 140 tok/s | 68-94 tok/s |

| Prompt processing, 4k tokens | 1,170 tok/s | 250-490 tok/s |

| Prompt processing, 8k / 16k tokens | 1,504 / 1,448 tok/s | |

Quality with Basalt's own 100-chunk gate: mean KLD 0.0493 and same top-1 92.75% (Basalt publishes 0.0496 and 92.63% for this pack on an RTX 5090).

**Caveats**

- Only tested on my machine (2x V100 32GB, Windows). Single card, 16GB or PCIe cards, Linux and NVLink setups are untested: reports from other V100 owners are welcome.

- Peer-to-peer is off by default on Volta: on my machine, direct copies between the two cards returned wrong data.

- NVFP4 packs need FP4 hardware, so use a K-quant pack (tested: UD-Q4_K_XL).

- It is a snapshot of Basalt as of November 15, shared as is.

All the credit for the engine goes to jesdga95 (Basalt) and Niko1221 (Strata), both MIT.


r/Qwen_AI • • 16h ago

Discussion We got GTA 6 before Qwen 4 27b

493 Upvotes

We got GTA 6 before Qwen 4 27b


r/Qwen_AI • • 17h ago

Help πŸ™‹β€β™‚οΈ I have my SWE paper in the morning and Qwen had to do this. WTF are guardrails on a damn chinese model.

Thumbnail
0 Upvotes

r/Qwen_AI • • 23m ago

Experiment LexiEngine All rust - Faster than strata, built for AMD cards. No faster engine for AMD cards exists.

Thumbnail
github.com
β€’ Upvotes

r/Qwen_AI • • 30m ago

News Basalt running QFN Q3 on RTX4090 beats Strata, nearly doubling the tps!

Post image
β€’ Upvotes

I came across Basalt some time ago, but it was reserved for Blackwell owners; recently, an Adamant fork showed up dedicated to Ada Lovelace and Ampere, so I decided to give it a go on my 4090.
And the results are impressive; the same quant gives me nearly double the tps as with Strata.


r/Qwen_AI • • 5h ago

Help πŸ™‹β€β™‚οΈ QWEN 27B vs FN AMD GPU Speed question

2 Upvotes

Hey guys, this is a repost from Local LLM thread, I figured maybe this is a good place to ask as well.

WhELP!!!


r/Qwen_AI • • 12h ago

Discussion Qwen3.6-35B-A3B decode: our native engine runs 1.25–1.48Γ— faster than llama.cpp Q4_K_M on the same GPU (single-stream, AR @512 ctx) β€” with an honest fidelity tradeoff

3 Upvotes

Processing img j9s5oa7dem2i1...

What this is (so you can weight the numbers): a native inference engine + compiler we built from scratch β€” no Python and no CUDA toolkit at inference time β€” that runs your existing checkpoint. It's an early / experimental release, the engine itself is sealed (not open source), and the benchmark below is our own, first-party. I'm one of the people who built it (founder/team), so apply the usual discount for a vendor posting their own numbers. I've put the operating point on every row and the places where we're worse in bold.

The result: on Qwen3.6-35B-A3B (35B-param

MoE, ~3B active), our decode throughput beats llama.cpp Q4_K_M on the same GPU, same pod, at a comparable ~4-bit quant (ours = VQ-balanced lut4, theirs = Q4_K_M).

Operating point for every row below: autoregressive decode, 512 ctx, single-stream (batch=1), our measurement on the same machine.

GPU veizik (tok/s) llama.cpp Q4_K_M (tok/s) speedup
RTX 4090 272.5 184.5 1.48Γ—
A100 80GB 192.1 140.9 1.36Γ—
RTX 5090 345.0 258.9 1.33Γ—
H100 80GB 268.8 215.2 1.25Γ—

Not speculative decoding (the thing this s

ub asks first): both sides are plain batch-1 autoregressive decode β€” no draft model, no MTP on either side. The gap is the engine, not a speculation trick.

The tradeoff (quant quality): at the same ~4-bit, our fidelity check puts us a touch behind Q4_K_M β€” kld_vs_q4km = 1.28 vs a BF16 reference, i.e. slightly less faithful. So: a throughput win at comparable bitrate with a small fidelity cost β€” want the most faithful 4-bit, Q4_K_M is still it. (llama.cpp also wins on hardware breadth and being open source.)

Other boundaries:

  • Single-stream decode only. For batched / concurrent serving, vLLM and TensorRT-LLM win β€” not claiming that regime.
  • Our own benchmark, same-pod β€” run it yourself (below).
  • NInfer: on the 5090 we're on par (~1%), not ahead.
  • SGLang exl3 u/3**.0bpw** is cited faster β€” but a much lower bitrate / quality point, not apples-to-apples with ~4-bit.

Fits on a 3090 too (supporting point, not the headline): the 35B-A3B is 21.44 GiB resident (balanced) on a single 24GB 3090 and decodes 194–2

21 tok/s (our measurement). Fitting a 35B on a 3090 is table-stakes β€” llama.cpp does it too β€” so I'm calling this a supporting point, not a brag. The llama.cpp 3090 figure I have is cited, not same-pod, so I'm deliberately not putting a multiplier on the 3090.

If you want to sanity-check the engine at sizes you can actually run locally, small dense Qwen3.5 on the 3090, same-pod vs llama.cpp (AR decode, single-stream):

model speedup vs llama.cpp (3090, same-pod)
0.8B 1.93Γ—
2B 1.47Γ—
4B 1.23Γ—
9B ~par

Other measured points (own numbers, different workloads β€” deliberately NOT paired with the llama.cpp table):

  • 5090 fast-mode decode: 544.
  • 7 tok/s (a different config from the AR@512 row above β€” don't compare it to the 258.9).
  • H100 long-context: scales to 131,072 tokens at 172 tok/s.

Why we built it: one engine, no Python glue and no CUDA toolkit to install at inference, that takes a checkpoint you already have and runs it β€” and the same engine also covers image, video and robotics, down to a Raspberry Pi CPU. That breadth under one runtime is the thing we couldn't get out of any single-model engine (llama.cpp / vLLM / NInfer / ik_llama), so we wrote our own.

Repro: our full own-number board β€” all these GPUs, every mode β€” is live at veizik.com/benchmarks, and the kit at github.com/veizikhq/veizik-cuda (cuda

_deploy/bench/, method in REPRODUCIBILITY.md) runs veizik run on your own GPU so you can clock the engine yourself. The llama.cpp rows are our own same-pod Q4_K_M measurement β€” run llama.cpp Q4_K_M on the same box and check the gap. Got a 4090 / 5090 / A100 / H100? Post your numbers β€” especially if the gap doesn't hold on your setup.

License: the engine is sealed / proprietary (not open source); it runs your open checkpoints.

Happy to get into kernel-level, quant-fairness or determinism questions in the comments.


r/Qwen_AI • • 14h ago

Benchmark A lossless packing result for the DavidAU Qwen 3.8 27B MLX conversion

2 Upvotes

I tested r3lax's 4-bit FP16 conversion of DavidAU's Qwen 3.8 27B on my 32 GB M1 Max. Its scale and bias arrays used 1.681 GB of active MLX memory. Storing repeated pairs once per matrix reduced that to 0.644 GB. The four-bit weight codes stayed in the original checkpoint, and every scale and bias was restored exactly.

I wrote a scanner so it's easy to check another conversion before packing anything. I'd be interested in results from the same model quantized with a different group size. Please include the model ID or revision so we can make a meaningful comparison.

https://github.com/Davcca/mlx-weightpack https://github.com/Davcca/mlx-weightpack/blob/main/benchmarks/README.md


r/Qwen_AI • • 20h ago

Other Qwen3.8-Flash-Next proactively caught errors in my tenancy renewal

Thumbnail
4 Upvotes