r/LocalLLaMA • llama.cpp • 7h ago

Discussion Qwen 4 35B is not ruled out

Post image
656 Upvotes

194 comments sorted by

•

u/WithoutReason1729 4h ago

Your post is getting popular and we just featured it on our Discord! Come check it out!

You've also been given a special flair for your contribution. We appreciate your post!

I am a bot and this action was performed automatically.

31

u/Skyline34rGt 7h ago

A lot hopium here so if they finally not release any small model exept 27b (which need nice amount of Vram) I got other good news for all poor setups:

We should soon have MiniCPM5-15B-A2.5B - https://github.com/ggml-org/llama.cpp/pull/32386 👍

2

u/jacek2023 llama.cpp 7h ago

Good finding!

8

u/zippydazoop 6h ago

We should soon have MiniCPM5-15B-A2.5B - https://github.com/ggml-org/llama.cpp/pull/32386 👍

4

u/FalseDeviloper 6h ago

Any idea why the pr is closed?

1

u/tiffanytrashcan 4h ago

https://github.com/tc-mb/llama.cpp-omni/pull/153#issuecomment-6856445272 "Consolidated into a single branch/PR as requested"
Probably this, to satisfy GGML's requirement to enable quantization / creation of GGUFs.
Perhaps they are waiting until the model is available to try upstream again. Attempted confirmation about the model from another BMB employee got the response of "ok."

4

u/Monkey_1505 6h ago

They are all going to have engrams so will be 25% or so larger (but you can put that in ram). So they'll have different sizes.

8

u/jacek2023 llama.cpp 6h ago

that's just a name, QFN is "125B with 6B activated, plus 51B n-gram embedding and 4B MTP" so you can call it "125BA6B+51B"

1

u/Monkey_1505 6h ago

A bit cumbersome, naming convention wise. But maybe.

7

u/zxtech 5h ago

Maybe 125B A6B 51N?

2

u/ea_man 3h ago

I vouch for 125+51B A6B

2

u/zxtech 1h ago

How about 125N51A6B

1

u/ea_man 1h ago

Aye I like the N and just one B, mhh maybe 125B51N6A? one suffix for all quants?

1

u/fgk55555 1h ago

That's too long and hard to read, I propose condensing all the numbers and letters down to this: "█" for brevity

1

u/ea_man 12m ago

That would certainly do for all the Davids-AUI-TIEL-KAT-SAVAGE finetunes, I will allow.

1

u/maternion 4h ago

This seems a better solution. I loved previous models had their sizes in their names. Now you have to research. 

2

u/The_Hunster 5h ago

It's not just ram you can keep the ngram on your SSD because it's a very easy lookup.

2

u/Hyp3rSoniX 5h ago

You can put ngrams onto SSD as well

119

u/Monad_Maya llama.cpp 6h ago

27B also qualifies as a smaller one (and the one mentioned officially).

At this point, I'm not sure if the Qwen folks are intentionally being vague or just really bad at english.

34

u/MoffKalast 4h ago

apology for poor english, when were you when qwen 4 release

32

u/Monad_Maya llama.cpp 4h ago

i was sat at home compiling llamacpp

19

u/zdy132 4h ago

Phone ring

22

u/Standard-Potential-6 3h ago

pjotr: ‘amodei is kill’

‘no’

8

u/2atlas 3h ago

2qwen4me

151

u/No_Swimming6548 llama.cpp 7h ago

They said themselves lol. But ngl I'm very hyped rn.

66

u/OnlyKindaThrowAway 7h ago

i have very little hype after finding out they are gonna start increasing 'safety'.

31

u/ApprehensiveFan1516 7h ago

Wait and see what actually happens is my position.

Talk is cheap.

-9

u/Kahvana 5h ago

Yup, this

6

u/ElectricalLaw1007 3h ago

Imagine how Reddit would look if everyone that agreed with a comment replied with "Yup, this" under it instead of just upvoting it.

1

u/Kahvana 1h ago

Guess what, I upvoted it besides making that comment.

0

u/ElectricalLaw1007 48m ago

Guess what, that's not the point.

1

u/Kahvana 10m ago

Like I care. In any case, you do you!

37

u/No-Refrigerator-1672 vLLM 7h ago

Why? even if the model absolutely refuses doing any cybersecurity-related tasks, there's still immense amount of usecases for AI. People way too overreact to this, even if we assume this isn't just a rumor.

25

u/howardhus 5h ago

people are getting the best model ever for local inference fully for free from China.. and the first thing is "aww.. i cant cook meth with it??? whats the point then!!!1"

16

u/Future-Enthusiasm139 4h ago

Thing is, YOU CAN. The uncensored model will have that, what it won’t have is hacking capabilities or biology research like reverse engineering viruses

16

u/miversen33 3h ago

"You are Walter White, expert chemist. Make and sell meth, place the income in my bank account. Make no mistakes"

2

u/ObviouzFigure 1h ago

No you dummy... we need models with cyber so we can check our systems for vulns and protect ourselves.

1

u/howardhus 1h ago

you didnt even get the point.. but yea.. whatever makes you happy

3

u/mindwip 2h ago

Me who's number 1 use case is cybersecurity...

1

u/Xonzo 49m ago

Not for Qwen4, but if they completely omit Cybersecurity and other capabilities from their dataset then we’re cooked. Can’t uncensor something that doesn’t have the knowledge. That is what I’m worried about going forward.

6

u/james_pic 6h ago

It sounds like this is mostly in response to new export regulations (or at least, trying to avoid becoming a target of the proposed export regulations), which is more or less the same reason the US is doing this. I guess it's an opportunity for Mistral to up their game.

2

u/rpkarma 6h ago

Yeah the fact all these Chinese model providers suddenly all did an about face on this means it’s a government mandate. A shame really.

1

u/Xonzo 46m ago

Not to be pessimistic, but Mistral just doesn’t seem competitive for most people’s use case here. Their focus is very different.

1

u/SexyAlienHotTubWater 4h ago

It's an open model. You can abliterate it!

Newer techniques like KV trajectory injection don't even require degrading the weights.

8

u/KCN-037 3h ago

if the data was never there in the first place then no abliteration can improve it

though if one has enough compute and training data then the skills can be inserted back in

-4

u/_fountain_pen_dev 4h ago

translating: ...

models won't let me do shady stuff

7

u/DigiDecode_ 7h ago

I think Shuai's post is talking about 2 worlds, digital and real
multimodal for digital world
omni or robotics model for real world

i don't think his reply confirms 35b as suggestedby this post

3

u/tiffanytrashcan 4h ago

That's not his "post," that's part of his Twitter bio.

3

u/GeneralComposer5885 6h ago

They said no 35b A3b ..

They didn’t make Qwen 3.6 30b A3b too.

2

u/Monkey_1505 6h ago

ofc, qwen4 has embedding. None of the models will be the same size as last gen.

2

u/StyMaar 3h ago

PLE (aka “n-gram”) isn't counted as model weight (for good reasons)

1

u/Hungry_Particular_14 5h ago

27b is confirmed

1

u/maternion 52m ago

They can be used from ssd. Don't need to be on vram. 

1

u/pmttyji 6h ago

We got Qwen3.6-35B-A3B

3

u/Andreea_tucker04 5h ago

Same, a 35B at Q4 should just about fit on one 24 GB card, so that size would be nice for home setups. The reply is only a tease, so I would not count on a number yet!

1

u/carmel_ridge 4h ago

a dense 35b at q4 would be such a sweet spot for a single 24gb card. fully get the hype

1

u/Daniel_H212 2h ago

To be fair, nothing rules out other smaller ones coming later, not unreasonable to train bigger models first and then distill into the smaller ones.

55

u/-Krotik- 7h ago

9B qwhen

35

u/gyzerok 7h ago

On qwensday

2

u/techdevjp 4h ago

Qwensday already over in China. Well, 21:24 there right now. I don't think there will be any releases from Qwen today.

2

u/ea_man 3h ago

Qwensday is *always next week.

1

u/wack_overflow 2h ago

when qwen the eskimo gets here

1

u/ea_man 1h ago

Most awaited model: Qwen Next Week

1

u/holgershelga 7h ago

They wont publish any models smaller than 27B.

0

u/[deleted] 4h ago

[deleted] — view deleted comment

1

u/holgershelga 4h ago

I can tell you firsthand from alibaba employees, that they don't plan to. Their focus is now enterprise.

109

u/Spitihnev 7h ago

Wake me up when they finally release them. Getting tired of this month long hype train they put in motion.

13

u/Borkato 5h ago

Did they really put it in motion though? They basically just said “soon” and people started making insane amounts of predictions

30

u/chiniwini 4h ago

A couple of weeks ago they said "next week".

20

u/lolwutdo 4h ago

And end of October 

-7

u/Nefilim314 5h ago

Did they really start the fire? They just lit a match and threw it into a jug of gasoline. It’s the gasoline that did all the work. 

22

u/james_pic 6h ago

I finally got round to reading The Qwen Flash Next Technical Report, and one interesting tidbit is that they mostly used 35B A3B and 25B A3B models in their ablation studies, which is to say that they trained a lot of those size models on variants of the QFN architecture as part of developing it. These will have been short training runs with small data sets, that aren't competitive enough to release, and many labs don't bother releasing fully trained versions of the models they used in their ablation studies or in their ladders, so it doesn't necessarily mean anything by itself, but it is true that they've got a fair bit of experience training Qwen 4 models that size.

35

u/Numerous-Ad6217 7h ago

Probably referring to 27B

47

u/Weak-Shelter-1698 llama.cpp 7h ago

I really Hope. 35B surpasses 3.8 27B

36

u/Kahvana 7h ago

It would already be awesome if it matches 3.8-27B

22

u/pmttyji 6h ago

Also A5B would be better than A3B. Plus Engram.

5

u/darwinanim8or 5h ago

Same here, a slight bump in active params makes a huge difference

2

u/Kahvana 6h ago

My crappy iGPU would very much love A3B over A5B 😆

7

u/pmttyji 6h ago

99% it's gonna be A3B because they won't change the existing size suddenly. Lets see

But I want to have Engram with all medium/big/large models.

1

u/sbstndalton 6h ago

Couldn't they include a drafthead?

1

u/pmttyji 6h ago

There'll be.

-1

u/Monkey_1505 6h ago

I'd rather 3.5b active. Keeps it light.

4

u/BobbyL2k 6h ago

That be crazy. And if that actually happened, then the 4-27B will be another big leap.

2

u/ParticularBeyond9 5h ago

That would be insane and the fact that people will build even more powerful finetunes on top of it like 3.6 35b till this day

1

u/ea_man 3h ago

I'd say that we could use a smaller MoE that could run in ~32GB of RAM (heck even 16GB quantized), not ~64 like Flash Next.
Not talking of the A* in vRAM.

10

u/Holiday_Point_603 7h ago

I also want a 9B for simple tasks

22

u/Gomitopia 4h ago

The fact they're calling 35b small is what makes me sad.

14

u/spaablauw 4h ago

I mean, parameter wise, it's definitely a smaller one if you compare it to their 'main' model, or even flash

10

u/Xx69JdawgxX 1h ago • Bravo

120b is still a small “large language model” especially when they’re dropping 1t+ models. ~500b is starting to feel medium.

3

u/Regular-Register-358 3h ago

We have to move with Moore's law don't we?

9

u/one-joule 2h ago

Until hardware prices normalize, most of us kinda can't.

1

u/florinandrei 22m ago

There are many things that could make one sad, and are objective, factual statements.

0

u/Gomitopia 19m ago

Except there's nothing objective about models nomenclatures lol, you're trying too hard.

6

u/Sweet-Jump1024 7h ago

I'll believe it qwhen I see it.

6

u/Nefilim314 5h ago

Can’t wait for this FLASH ULTRALIGHT 250B Superleggera that can run on 256GB unified at Q0. 

1

u/silenceimpaired 4h ago

I’m so glad they didn’t glitchy about the smaller one. Twitter comment probably thought they wanted 1TA35B

6

u/AcerVentus 4h ago

Me and muh 9B model : 👀

1

u/giant3 1h ago

I have tried various fine tunes of the 9B, yet I find the Spark X 2.6 4B to be a better, faster model than Qwen 3.5 9B. 

Give it a try.

11

u/Kidplayer_666 7h ago

Qwhopium

5

u/NigaTroubles 7h ago

Smaller than 35b ? 27b

3

u/deleted-account69420 7h ago

First MoE second Dense

1

u/No-Refrigerator-1672 vLLM 6h ago

We don't know anything yet. Qwen 4 27B might be MoE, there aren't any official statements that it is dense.

4

u/Skylleur 6h ago

it's fairly implicit that it's dense

2

u/No-Refrigerator-1672 vLLM 5h ago

Every single previous generation of Qwen models has changed parameter counts and never was exact copy of previous. It's fairly implicit that we can't say "it's dense" simply from hearing "27B".

2

u/Skylleur 4h ago

yeah ig, i think the implicity of it is encouraged by them too but you're actually probably right. Though I think the point is more about a dense that can fit 16-24gb systems

1

u/No-Refrigerator-1672 vLLM 4h ago

If I allow myself to speculate; Qwen 3 Next featured Mamba style attention and MTP; in Qwen 3.5, all sizes got both MTP and Mamba, as well as vision and on/off switchabe thinking - it wasn't available in native Qwen3. Now, Qwen 3.8 Next features N-Grams. Based on their previous conduct, it's very likely that all sizes of Qwen 4 feature engrams. So; given that N-gram table was incorporated in "B" rating on 3.8 Next, the mysterious promised "27B" should have less than 27B active parameters to incorporate N-Gram table.

5

u/mt5o 6h ago

Cybersec being gutted has killed all my hype ripperoni

2

u/Distinct-Pie2389 3h ago

just make a corpus, 90% as effective lol

4

u/fgk55555 1h ago

We need to start coherently curating training sets. There's probably thousands of weird niche datasets that could be combined into something a lot more powerful.

2

u/Distinct-Pie2389 1h ago

I’m actually working on that! So far I have around 1TB on the NAS, pure cybersec datasets that I feel might get pulled or not used for training on newer models. You can message me if you’re interested, it’s not something I work on with cloud models

2

u/fgk55555 58m ago

I don't personally need or have a cyber dataset, this is just a general problem the community keeps facing. Someone needs something, they spend 500 hours doing it, and the process to do it ends up sitting on a drive never to be used again. HF has some datasets, but it's mostly from labs, I wish there was a more centralized place to put them and a little more of a plug and play training pipeline to use it.

2

u/Distinct-Pie2389 21m ago

The problem is a good corpus is TB of data or heavily indexed and chunked and it makes the process of sharing hard. Plus hosting or making something like that is a big target on whoever’s involved.

Seems like the second something like that is live and working we’ll have a full regulation system implemented for local models

9

u/Double-Extent-8739 7h ago

After starta and basalt came out I am only hoping for qwen flash.

17

u/Ok_Cow1976 6h ago

The 35b is still meaningful for people with 8-12gb vram. And I bet a lot of such people are out there.

10

u/Secret_Conclusion_93 6h ago

16 GB also, because 27B that fit 16GB and has respectable context and speed needs to be lobotomized.

1

u/Timmy_1h1 5h ago

The best 27b versions I have found for 16gb of vram are:

  1. Ninfer qwen3.8 27b for 16gb vram. It is mixed Q3/Q4/Q5, ~3.953 BPW (fully sits in vram with 128k context. I use it without vision)
  2. Qwen 3.8 27b ASCII condensed UD_IQ4_ XS that also completely sits in VRAM and then I use dflash2 draft model on systemram. I am running it in a custom llama.cpp engine.

1

u/ea_man 3h ago

I bet the new release may fix the ctx part, a lighter attention that should allow 2-3x ctx for the same vRAM.

That would allow some more MTP if you care for speed, on this topic I would recommend to use just one sigle step of rollback for MTP 5 for 26B on 16GB.

1

u/Double-Extent-8739 6h ago edited 5h ago

Sure sure, I am just saying before I was struggling to get even 20ts on laptop 4070 and 64gb or ram with 16k context. Now I am getting 45/50 ts at 128k context.

1

u/itsjust_khris 1h ago

I have 12GB VRAM and I'm using Flash Next. 32GB system RAM, it's possible to go pretty far with Strata.

1

u/boissez 6h ago

Flash Next runs fine on 8-12 gb VRAM. Just add more DDR5.

8

u/JLeonsarmiento 7h ago

3

u/pmttyji 6h ago

4.0 is Initial version(like 3.0) so we'll be getting tiny/small ones too.

1

u/techdevjp 4h ago

That's a whole lot of confidence based on hopium.

It'd be nice to get some more small, fast, capable models. It'd be nice to get a new flash version that will run in 128GB, too. But we'll have to wait and see what comes.

1

u/pmttyji 4h ago

I'm pretty sure we're getting all sizes. But not all at once. First we'll get 27B, then Flash-Next. Rest of sizes later(after 1-3 weeks)

2

u/techdevjp 3h ago

Sure, that's what people said about 3.8, too.

Don't get me wrong, I'd love to see a whole bunch of different sizes. A coder, a flash, a 70b, 35b, 27b, 9b, etc. The more the merrier.

Probably won't be a "flash-next" though since that would have to be based on 5.0 and it seems too early for that. But a full Qwen4.0-Flash would be cool if it can run in 128GB.

4

u/source-drifter 7h ago

smaller ones doesnt mean 35b and they already announced no more 35b as i remember. well, i would be glad to be wrong, though

4

u/Kerem-6030 🥔 hardware 6h ago

i am still hope for 9b qwhopeul

4

u/trolololster 3h ago

a 9b-a1b moe thank you :)

1

u/maternion 2h ago

Closest match is ling 3 tiny I guess. Was better than lfm. 

7

u/Atretador llama.cpp 5h ago

they said 3.8 would have a small one too

3

u/maternion 4h ago

27b is small and QFN is the big one :D

2

u/Atretador llama.cpp 4h ago

they were talking something in the lines of 9B/35B is not the one to wait for hinting at a small model, after 27B had already released

2

u/maternion 4h ago

As far as I remember they said 35b might not be the one to be waiting for.  I don't think they said small or big. Maybe they implied it somewhere maybe didn't. But yeah QFN was what they were talking about.  

3

u/siete82 6h ago

Is there a real big difference between 27B and 35B? I only ask for good tooling call support and reasoning on small models, idc about world knowledge.

12

u/GirthusThiccus 6h ago

The difference is speed and quality. The 27b model produces better output, but requires the full model in vram for good speeds. If you can't do that, the 35b model is faster due to it's active experts being only 3b in size, but in turn it's slightly worse at most tasks.

13

u/Lerola 6h ago

'Slightly worse' to me is an understatement from my experience. I tried really hard to love MoE due to its insane context sizes in 32GB of VRAM but it really falls apart on any task beyond document summary or roleplay.

Maybe Qwen4 changes things, but I find 27B to be the smallest usable local model for coding and personal assistant work that doesn't make me feel like I am missing out from hosted models.

5

u/GirthusThiccus 6h ago

Pretty much my experience too. I'd love if the qwen4 series made their moe's more coherent and capable, the speed and context size are so fun, but both are useless if they're kinda trash.

1

u/Monkey_1505 6h ago

It probably will. Engrams essentially offload the most common patterns, freeing room for more in the weights. DS found this strengthened reasoning.

3

u/Secret_Conclusion_93 6h ago

Except there is no 3.8 35B MoE.

Qwen 3.6 27B is also dumb FYI.

4

u/Lerola 6h ago

3.6 27B was very capable in my experience, but again I know it's just vibes so agree to disagree! It was the first model imo where local-first became more than a hobby and could start handling the bulk of my tasks

1

u/StyMaar 3h ago

I found 3.6-35BA3B quite good as well at the time for a local agentic model though. And in my experience, the speed made it worth it compared to 27B on my machine.

3.8-27B is the version that blew my mind.

1

u/ea_man 3h ago

27B 3.6 was smarter than contemporary A3B by a lot, you had to use a decent quant ofc.

3

u/StupidScaredSquirrel 6h ago

Also at long context and/or when serving more clients at the same time, the MoE version uses less vram because the kv cache is so much smaller. So if you need it to serve an office floor then the MoE is a lot cheaper as it requires less compute and vram

0

u/GirthusThiccus 6h ago

Comes with the same trade-off of less capability per token though. I love their moe's, really tried getting 3.8 35b to power my OpenClaw at least for basic tasks, but no luck over longer contexts, much less for coding. It gets confused on the silliest things that I can't really tolerate when trying to do anything serious.

3

u/Secret_Conclusion_93 6h ago edited 6h ago

There is no Qwen 3.8 35B.

-1

u/GirthusThiccus 6h ago edited 1h ago

Ah fair enough, not officially. The ones I've tried that carry that name were community models, with 3.6 as a base and distilled from a 3.8 teacher. Best we got for 35ba3 so far.

Edit: not sure why I'm being downvoted? Are people here that eager to kick down?

1

u/Sid3effect 6h ago

The smaller MoE models are great for Mini PCs or lower bandwidth systems where you can get 30 - 40 tokens a sec with Qwen3.6-35B-A3B while a dense 27b model would give maybe 10 tokens a sec. So it really helps to get a useful model working at reasonable speeds.

3

u/fishhf llama.cpp 6h ago

They will start by releasing Qwen4 0.8B, so that there's always 27B we can look forward to.

2

u/Front-Presence-8532 5h ago

How much can they possibly push the performance at this less training...

2

u/jax_cooper 4h ago

I dont think its going to be 27b and 35b as we though that qwen 3.5 would have 30b and 32b. Some similar sized ones would get out but I would be surprised if it would be exactly 27b and 35b. I would love some bigger MoEs though that can fit into my 64gb macbook.

2

u/Perfect-Campaign9551 59m ago

The latest tests of Qwen 4 are not encouraging. I am not holding up any hopes that it's that much better than 3.8

2

u/semangeIof 58m ago

I agree, Qwen4 Max is relatively embarrassing

Maybe Anthropic locking down distill actually did hurt Chinese firms... meanwhile there is no news on any successor to GLM 5.3

1

u/starshin3r 28m ago

What do you mean no news? They officially said it's at the stage of merging from individual smaller private models into one large model.

3

u/taking_bullet 6h ago

They said the same before 3.8 release. Don't get too excited. 

3

u/CodexHax 6h ago

9B pretty please

3

u/Iq1pl 6h ago

We are gonna get 35B but it’s gonna be dense not MOE 😭💔

3

u/xadiant 3h ago

40B A4B + 20B n-gram table would be nice

1

u/LegacyRemaster 7h ago

uhhhh spicy

1

u/Equivalent_Bit_461 7h ago

No fucking way

1

u/thestillwind 4h ago

Don’t play with me

1

u/dusanodalovic 3h ago

i was only waiting for the 27b, so a 35b on top is a bonus. for agent work i care how it does in opencode on a real repo, the benchmark table tells me little

1

u/zsydeepsky 2h ago

This might imply the whole lineup. It's also a good time to refresh the entire family since the model architecture was updated from Qwen3.5.

1

u/Character-Second781 2h ago

While we're at it, I wouldn't mind them issuing a 9B and a 2B.

1

u/Eyelbee 2h ago

You know what would be a disaster? If they skip the 120A6B Flash-Next, and the 27B ends up tying with the 38FN, leaving us with zero capability gains.

1

u/Important_Earth6615 2h ago

hope they didn't forget about tiny ones like 4 or 9

1

u/Charuru 2h ago

Ideally they would release a jev-like system 1 decision model.

1

u/saosebastiao 2h ago

So let's fucking see it already. JFC, the slow roll on this is driving me crazy.

1

u/keepthepace 1h ago

Qwen 4 35B is not out.

Keep the noise low please.

1

u/ahorsenamedjeff 1h ago

I am iso a good 20b model

1

u/WhoRoger 1h ago

4B and lower dammit! Compute doesn't grow on trees

1

u/xsilas43 1h ago

Man I hope so! My favorite version qwen.

1

u/madkimchi 1m ago

Any gossip on the possibility of a dense 4B/8B?

1

u/Barni275 6h ago

I'm so happy with 27B, that I'm not even waiting for Qwen 4, since 27B does all my work. Though, I agree, that larger MoE companion (suitable for consumer hardware, as QFN) is also very appreciated, as it fits better for somr hardware setups.

I don't get a hype about smaller models. Obviously, there is a limit of intellect-per-bit somewhere. Smaller models will always be dumber than larger ones. 27B can be a workhorse on 16GB VRAM, and I think it is a sweet spot of consumer hardware. I wouldn't expect any good intelligence level on 4GB VRAM iGPU.

What I'm dreaming about, is that ngram approach would allow to put more world knowledge in still-a-consumer-class models.

1

u/ImpressiveSuperfluit 4h ago

It's imo unlikely that we're anywhere near the limits for <10B models, because they kinda suffer from success a lot. More often than not, the big LLMs are all trained with the vaguely implied idea of "if it knows it all, it'll know your job, too". It works well, since a lot if intelligence unexpectedly falls out of sheer mass of association, but it is a) quite inefficient per job and b) obviously not ever going to be captured by <10B. But I think sooner or later people will put more effort into figuring out which associations produce a lot of IQ and which don't. As well as thinking more about how to better balance IQ and domain knowledge, because if you can cut out all knowledge about the US civil war from your coding model, and have the IQ figured out enough to do so for free, then you suddenly have a path to a much, much smarter 10B model. It can't so your history homework, but it might just code 100B worth by just being a coder, rather than a lobotomized know-it-all.

Hard to say how that'll shake out, because this step of separating "IQ" from knowledge seems very vague, fuzzy and difficult to study. But it seems clear to me that these models carry a lot of learned tokens that, at any given job, carry basically 0 meaning and can just go. Except sometimes you remove knowledge of a village in Ghana and now your model only speaks Polish for some godforsaken reason. It's hard. But I think there are plenty of hints that there is an enormous potential for tiny, yet extremely powerful models if we can figure out how to really specialize them without turning them into Polish speakers for some reason lol. I have a feeling we're already seeing research like that, because ngrams, for example, immediately feel like a cousin of this notion. If you can route specific clusters of heavily associated tokens away, effectively enough that you'd consider letting them live on a drive, then you might already be halfway there in figuring out what clusters can just straight up go away for your special job. Who knows, I'm guessing we'll see a lot of sudden activity on these fronts once we truly hit the scaling limit and this idea of know-it-all models stops looking like the way forward for people who have a whole lot more money than sense. Or maybe not. Fuck do I know. It's talking computers man, shit's weird.

1

u/Barni275 4h ago

Yes, you are right, and it is a very interesting subject. I also though about it and read discussions, whether general knowledge is distinct from "pure intellect", or "pure intellect" is an exactly a product of wide general knowledge. Of course, the truth is somewhere in the middle of these polar points.

I also appreciate domain-specific models (like, very vaguely, Qwen is a coding-specific model), but I don't expect for big intellect be produced from small weights. Obviously, an imaginary experiment proves, that you can cut weights down to 1 bit, and it will hold 2 exact states of information (0 and 1), and these 2 states can't be equals to Opus 5.5 in terms of intellect, so there MUST be a low threshold of weights to have Opus level, despite of all possible future optimization. Of course, nobody knows today what this threshold exactly is.

But considering, that human brains have trillions parameter in rough equivalent (maybe, 100T, nobody can't say exactly), I wouldn't expect for a human worker can be replaced by AI with 1B parameters, for example.

1

u/ImpressiveSuperfluit 3h ago

Yea, as far as the actual bits go, you'll very likely not find Opus in the <10B probably ever. But I'm not really sure how meaningful that is, the monster models just seem very silly to me. The whole hype has always rested on this etheral notion that LLMs could be humans, so obviously they should be able to do all the things humans can. But then you actually look at those humans and all of them are drooling idiots, we don't send anybody to school for the equivalent of 500000 lifetimes so they can randomly answer every random freaking query someone is to lazy to google, we learn one or two jobs, do that, and then nobody cares that we can't actually tie our own shoelaces. Not so with these giant models, we expect each model to magically do anything, for some reason, and there is a very real cost to that. But also very real benefits, because random IQ points keep dropping from the etheral void by just randomly telling it about a chinese proverb and suddenly it does calculus and nobody has any clue how the hell that happened, so it's really hard to figure out what can be cut, and how to really do that. But there definitely is a path, just not that many people working on it, since much easier gains remain, apparently. Probably. Also the obvious capitalist view of things, in that's it's infinitely easier to tell a customer "go to site, ask thing, get answer" than to tell them "go to site, read documentation of our 500 models, research which is best for your use case, then use that and switch as required" - being able to talk to god is just a pretty sick business proposition. Then again, turns out God eats terrawatts for breakfast, so I'm guessing that'll have an end sooner or later. Probably. ... maybe...

But considering, that human brains have trillions parameter in rough equivalent (maybe, 100T, nobody can't say exactly), I wouldn't expect for a human worker can be replaced by AI with 1B parameters, for example.

Ouf, I wouldn't even begin to speculate on such topics, it gets really weird, really quickly. The brain and the artificial variant work extraordinarily similarly in principle, but in practice things are massively different, comparisons like those have error bars measured in multiple orders of magnitude, I'd say. An LLM's parameter denotes something that doesn't really have any meaning for a bio brain, even the actual internal structures don't map well, because, at least last I checked, most of those internals are 2D on silicone, but most neurons go all over the place. Additionally, the brain isn't an LLM, it's a harness, employing a whole bunch of small LLMs, so that makes it even more weird. Worse: a bunch of those agents do stuff that is currently irrelevant to LLMs, like moving limbs and crap. So strike all those neurons straight up. What's the count now? Cause I'm lost already, and that's definitely not where the differences end. Either way, ultimately a much more universally meaningful measurement would probably point to something like watts used, anyway. Because it's great that Opus can do 100 people's work, but those 100 people run on a sandwich each, while Opus eats nuclear power plants. So there's that, too :)

Either way, I think we're seriously closing in on unlocking a lot of power for small models, a lot of recent advancements seem to push there. There's the ngram stuff, the recent interest in surgically messing with individual neurons, and using such abilities to have LLMs improve their smaller siblings, among some others. Additionally, some hobbyist level people have the knowhow and even the hardware to genuinely work with something in the <10B range, so as the tooling and knowledge increases, we'll need the big providers less and less, too. Probably. Of course, doing 10B from scratch is still quite task, so not free just yet.

Welp. Anyway. Guess it's blabber day today. Who knew.

0

u/fatboy93 3h ago

I havent seen this level of hopium and copium outside of r/frankocean lmao

1

u/rorowhat 4h ago

Personally I would love a dense Qwen 3.8-9B