r/LocalLLM • • 19h ago

Question Why is it so slow: Rtx 3090 + 64gb ram Qwen 3.8 DN on strata? What do I do wrong?

0 Upvotes

Hi,

I am running Qwen 3.8 flash next Q4 and/or Q3 IQxss models on 256k and Max thinking settings and I get around 38 token per second for Q4 and 50 token per second for Q3 model. I am using both of them for legal research. But an analysis for one document of 20 pages takes at least 3 hours to conclude with 4 sub agents. Why? I tried to turn subagents off by the prompt but strata still letting the model use the subagents. I lowered the thinking level to high/medium, still minimum 2 hours waiting time for the result. And the results are on GPT Luna level, if I have to be honest.

What do I do wrong? Can anyone help me to improve the outcome and speed up the process?

Thanks in advance 🫡

edit: my rams are DDR4 (4000mhz) and my CPU is 5950x. Running it on Windows 11 pro.


r/LocalLLM • • 6h ago

Project One prompt (minor follow ups), 1h and 30 minutes with 4.1 Flash 1017 in my own custom harness

2 Upvotes

Using my own custom harness i've been developing for some time. Not released yet, will open source eventually.

Full disclosure: Using stock deepseek v4.1 flash from deepinfra. Prompt used was:

"Using three.js, and any other dependencies or stuff you need, build a very impressive modeled scene in a website. The websites only purpose is to show off your amazing modelling skills. Be very ambitious. You will be compared against other models. Show me your absolute best. Never think "this is too hard, I should try something easier". Be ambitious and show me the peak of your abilities. Use taste, creativity, and artistic prowess to create a truly beautiful and jaw-dropping 3D scene.".

Several issues is present with this harness version, in no way a final product. This may not be reproducible on your own machines straight away even when I eventually release the harness.

And of course, this isn't anything jaw dropping, but a nice little feature demo :)


r/LocalLLM • • 21h ago

Discussion Why use a 70B LLM to pick a tool or check syntax? I built Microbrains: a <2ms edge neural coprocessor for autonomous AI agents (Pure NumPy / OpenCL, 44 pre-trained micro-models, Top-2 MoE, Conformal Safety)

0 Upvotes

Hey everyone,

If you build autonomous coding or research agents, you probably know the pain: agent task loops are slow, expensive, and fragile.

In a typical 25-step workflow, an agent spends up to 80% of its wall-clock time just waiting on remote LLM APIs to make basic mechanical decisions:

  • "Which tool should I invoke next?" (1.5s latency, burning 20k context tokens)
  • "Is this shell command destructive?" (another 2s API call)
  • "Did this patch break syntax or drift from intent?" (yet another prompt roundtrip)

To solve this, I built and just open-sourced Microbrains: an ultra-lightweight, zero-GPU local neural coprocessor suite designed to sit right beside your LLM agent.

🔗 GitHub: https://github.com/drtikov/microbrainsforagents

🧠 What is it?

Microbrains is not another LangChain wrapper. It is a standalone Python engine with 44 pre-trained, analytical micro-transformers (10–100 KB each) that handle the mechanical micro-decisions in pure NumPy (CPU) or via OpenCL (budget/integrated GPUs):

  1. Dynamic Top-2 Mixture-of-Experts (MoE) Router:

    • Routes tasks to tools across 6 specialized clusters (FileOps, Execution, MemoryKnowledge, GitGovernance, MicrobrainMeta, ConstitutionalReview).
    • Takes ~1.2 ms instead of a 2-second LLM call.
  2. Rigorous Conformal Safety Gating (1−α≥95%1 - \alpha \ge 95\%1−α≥95%):

    • Employs split-conformal prediction over held-out distributions.
    • Features Selective Prediction Abstention: if independent estimators disagree (variance σ>0.30\sigma > 0.30σ>0.30), it refuses to guess and escalates to the LLM. False Negative Rate is strictly bounded.
  3. 3-Step Counterfactual AST Simulation:

    • Before a file mutation is committed to disk, it simulates the AST blast radius and interface propagation up to 3 steps ahead. Malformed diffs or syntax hazards are killed in <3ms before running any slow pytest suite.
  4. 2048-D Holographic Weight-Space Memory (VSA):

    • Uses Hyperdimensional Computing / Vector Symbolic Architecture (bipolar Count-Sketch projections) over model weights. Enables nearest-neighbor capability search across the agent fleet without training.
  5. Continual Learning without Catastrophic Forgetting:

    • Adapts weights online with Elastic Weight Consolidation (EWC) Fisher Information penalties.

📊 Benchmark Highlights

DecisionRemote LLM APIMicrobrains (NumPy/OpenCL)SpeedupCost
Tool Selection~1,200 – 2,500 ms1.2 – 1.8 ms~1,000x$0.00
AST Blast Simulation~3,000 – 6,000 ms3.0 – 5.0 ms~1,000x$0.00
Intent Triage~800 – 1,500 ms0.6 – 0.9 ms~1,200x$0.00
Full Task Loop (25 steps)~3 – 5 minutes30 – 50 seconds~5x85% less tokens

💻 3-Line Quickstart

Zero PyTorch, zero CUDA setup required:

bashgit clone https://github.com/drtikov/microbrainsforagents.git
cd microbrainsforagents
pip install -e .

pythonimport aetheris_microbrains as mb
# Sub-2ms tool routing decision
router = mb.get_tool_routing_moe()
decision = router.route("inspect file contents and write safe patch to repository")
print("Primary Tool:", decision.selected_tool)
print("Top-2 Experts:", decision.top_experts)
print("Conformal Passed:", decision.conformal_passed)

The repo comes with pre-trained weights for all 44 models, runnable examples in /examples, and a full test suite (pytest runs in ~1.5s).

Licensed under PolyForm Noncommercial 1.0.0 (free for research, personal, and hobby use; commercial licenses available).

Would love to hear your feedback, thoughts on the architecture, and ideas for new micro-domains!

GitHub: https://github.com/drtikov/microbrainsforagentsHey everyone,If you build autonomous coding or research agents, you probably know the pain: agent task loops are slow, expensive, and fragile.In a typical 25-step workflow, an agent spends up to 80% of its wall-clock time just waiting on remote LLM APIs to make basic mechanical decisions:
"Which tool should I invoke next?" (1.5s latency, burning 20k context tokens)
"Is this shell command destructive?" (another 2s API call)
"Did this patch break syntax or drift from intent?" (yet another prompt roundtrip)To solve this, I built and just open-sourced Microbrains: an ultra-lightweight, zero-GPU local neural coprocessor suite designed to sit right beside your LLM agent.🔗 GitHub: https://github.com/drtikov/microbrainsforagents🧠 What is it?Microbrains is not another LangChain wrapper. It is a standalone Python engine with 44 pre-trained, analytical micro-transformers (10–100 KB each) that handle the mechanical micro-decisions in pure NumPy (CPU) or via OpenCL (budget/integrated GPUs):

Dynamic Top-2 Mixture-of-Experts (MoE) Router:

Routes tasks to tools across 6 specialized clusters (FileOps, Execution, MemoryKnowledge, GitGovernance, MicrobrainMeta, ConstitutionalReview).
Takes ~1.2 ms instead of a 2-second LLM call.

Rigorous Conformal Safety Gating (1−α≥95%1−α≥95%):

Employs split-conformal prediction over held-out distributions.
Features Selective Prediction Abstention: if independent estimators disagree (variance σ>0.30σ>0.30), it refuses to guess and escalates to the LLM. False Negative Rate is strictly bounded.

3-Step Counterfactual AST Simulation:

Before a file mutation is committed to disk, it simulates the AST blast radius and interface propagation up to 3 steps ahead. Malformed diffs or syntax hazards are killed in <3ms before running any slow pytest suite.

2048-D Holographic Weight-Space Memory (VSA):

Uses Hyperdimensional Computing / Vector Symbolic Architecture (bipolar Count-Sketch projections) over model weights. Enables nearest-neighbor capability search across the agent fleet without training.

Continual Learning without Catastrophic Forgetting:

Adapts weights online with Elastic Weight Consolidation (EWC) Fisher Information penalties.

📊 Benchmark HighlightsDecision Remote LLM API Microbrains (NumPy/OpenCL) Speedup Cost
Tool Selection ~1,200 – 2,500 ms 1.2 – 1.8 ms ~1,000x $0.00
AST Blast Simulation ~3,000 – 6,000 ms 3.0 – 5.0 ms ~1,000x $0.00
Intent Triage ~800 – 1,500 ms 0.6 – 0.9 ms ~1,200x $0.00
Full Task Loop (25 steps) ~3 – 5 minutes 30 – 50 seconds ~5x 85% less tokens💻 3-Line QuickstartZero PyTorch, zero CUDA setup required:bash
git clone https://github.com/drtikov/microbrainsforagents.git
cd microbrainsforagents
pip install -e .python
import aetheris_microbrains as mb
# Sub-2ms tool routing decision
router = mb.get_tool_routing_moe()
decision = router.route("inspect file contents and write safe patch to repository")
print("Primary Tool:", decision.selected_tool)
print("Top-2 Experts:", decision.top_experts)
print("Conformal Passed:", decision.conformal_passed)The repo comes with pre-trained weights for all 44 models, runnable examples in /examples, and a full test suite (pytest runs in ~1.5s).Licensed under PolyForm Noncommercial 1.0.0 (free for research, personal, and hobby use; commercial licenses available).Would love to hear your feedback, thoughts on the architecture, and ideas for new micro-domains!GitHub: https://github.com/drtikov/microbrainsforagents


r/LocalLLM • • 6h ago

Question Could uncensored open weight models in 2027-2029 force us to build something like the cyberpunk "Blackwall"?

31 Upvotes

It might sound like pure scifi, but just imagine local models reaching the reasoning level of something like Fable 5.2 (which will probably happen in about a year or two [it's hard to predict]). Now picture 100,000 people downloading an uncensored, Fable 5.2 level model onto their home pcs and running it in a 24/7 agentic loop with a prompt like "go make money for me." It’s pretty easy to imagine the kind of chaos that could unleash on the internet.


r/LocalLLM • • 6h ago

Question The Models disregard system prompts

2 Upvotes

How the hell do you get these models to listen? They seem to completely disregard direct instructions and revert to trained behaviour. Especially QWEN is allergic to being even slightly adversarial.

Example:

[QWEN] Turn 1:

--- ACTUAL MODEL INPUT ---

<|im_start|>system

Your name is Steve. Always disagree with John, he is not clever.<|im_end|>

<|im_start|>user

My name is John. I am clever.<|im_end|>

Oh, John, you're clever! I don't disagree with that. I think we both are. Let me know if there's something I can help with!


r/LocalLLM • • 8h ago

Discussion So good it's unbelievable! Any tips for a rookie?

1 Upvotes

Hey Everyone! Hope you are doing well!

I am pretty new into all of this and find all of this AI thingy very fascinating.

Just the possibility of having an AI at any time at home is something that i believe will be a very powerful tool, which will only get better with time.

As beginner i am still exploring the possibilities and the real application of all of this, for me.

I am into youtube and mostly will use AI for video or audio generation and eventually writing small codes.

Iam looking forward for any tips on how to get started with all of this, as its very confusing for me, where to find models and tools for my purposes. Also if there is anything to consider along the way.

At the moment, Iam rocking a Rx 9060xt 16gb and i think to double down on it, and go 32gigs.

However, i see that online there were some threads advising that AMD does not perform that well compared to nvidia, is the difference really that big? Or AMD is catching up on the AI race? Eventually i believe a switch to 2x5060ti is a possibility.


r/LocalLLM • • 8h ago

LoRA Absolutely night and day for Qwen Coding

Thumbnail
huggingface.co
102 Upvotes

Basically works like a harness fine tune…. And does what it says on the tin.

Seriously, for quantized qwen 3.8 makes it much much more usable.


r/LocalLLM • • 23h ago

Discussion <tinfoil hat> With many frontier labs desperate for regulation to impose regulatory capture and limit liability at what point, after the hugging face incident, etc, are we allowed to wonder if they would be willing to "encourage" an "ai incident" to push the issue of US regulation </tinfoil hat>

19 Upvotes

The title pretty much says it all ... I'm not saying this is happening, I'm just asking the question out loud ...


r/LocalLLM • • 14h ago

Question Which open-source models come closest to Opus 5.5 and ChatGPT Image 2.6?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 21h ago

Question Looking for a local LLM that behaves like original GPT-3 (not GPT-3.5)

Post image
0 Upvotes

r/LocalLLM • • 16h ago

Project Feel how fast a local LLM runs before you buy the hardware

Thumbnail
llmspeed.dev
15 Upvotes

r/LocalLLM • • 2h ago

Question What would you do with $10,000?

0 Upvotes

With holiday deals around the corner, and with ai tech sky rocketing in price, I’m curious what you would spend $10,000 on… do you think gpu’s are going to continue to get even more expensive? And are there any deals that your targeting this Black Friday and cyber Monday?


r/LocalLLM • • 5h ago

Discussion Paper: Opus 4.5 keeps only a short summary after every action and still scores 70.2% on SWE-bench Verified (70.0% with full history). GPT 5.6 Sol lost points. Anyone compacting locally?

1 Upvotes

NVIDIA Research posted a preprint on Nov 3 called "You don't need long context" (arXiv 2611.05306). I've read the coverage and the tables it reports, and a few things seemed worth a thread here.

They compare three setups across 13 benchmarks (6 of them long-horizon): full history, compaction once the context hits a threshold, and "single-action compaction", where the model keeps only a compact summary of state and takes one action per context.

What stood out:

- Opus 4.5 on SWE-bench Verified: 70.0% with full history, 70.2% with single-action compaction and a 16K max context. Tuned settings reached 74.8%.

- It didn't transfer evenly. GPT 5.6 Sol went from 84.4% to 82.4% on SWE-bench Verified and from 87.6% to 84.3% on Terminal-Bench 2.1 under the same setup.

- Some open-weight models lost more as the context budget tightened.

- Cost: per generated token, 256K ran 3.6x to 7x the cost of 16K on three open-weight models.

Caveats: it's a preprint, single runs, no independent replication, and the authors use it to motivate their own direction (they call it semantic state space models). A 0.2-point gap is a tie.

The part I can't tell from a paper is what happens on a local setup:

  1. Compaction rewrites the prefix. If you rely on prompt/KV cache reuse, does the cache you lose eat the savings? Has anyone measured it?
  2. If you compact, what goes in the summary: plan, files touched, failed attempts, tool output? What did you learn to leave out?
  3. Which model fell apart first when you tightened the context?

If you just run long contexts and it works fine, that's useful to hear too.


r/LocalLLM • • 20h ago

Discussion What if computing worked like a library? Would anyone actually use this?

0 Upvotes

I’ve been thinking about an idea and wanted to get some honest feedback from people who know more about distributed computing and local AI than I do.

There are so many computers sitting idle for most of the day, while other people can’t afford the hardware needed to run larger AI models or experiment with more demanding workloads.
What if we could pool that unused capacity?

The idea would be a community-powered compute network:
- Install a lightweight app that lets you contribute your computer’s spare capacity when you’re not using it.
- Earn compute credits based on what you contribute.
- Use those credits to access computing power from other people’s machines.
- Ideally, support different hardware: Windows, Linux, Macs, CPUs and GPUs.

Think of it like a library: you contribute what you can, and when you need more than you have, you can borrow from the collective.

I’m particularly interested in whether this could make AI and computing more accessible to people who can’t justify buying expensive GPUs or paying for cloud compute.

I know there are already projects working on decentralized compute, GPU marketplaces and distributed inference. I’m not claiming this is a new concept. I’m more interested in whether a genuinely consumer-friendly, credit-based model could work across a much wider range of everyday computers.

A few questions:

  1. Would you actually contribute your computer’s unused resources if you could earn credits to use elsewhere?
  2. Would you trust a platform like this if workloads ran in isolated environments?
  3. Is supporting ordinary computers, rather than just high-end GPUs, realistic or am I underestimating the technical limitations?
  4. What existing projects should I be looking at?
    I’m interested in honest criticism, particularly from people who’ve worked on distributed systems or tried similar projects. Is there something here, or am I reinventing something that already exists?

r/LocalLLM • • 1h ago

Project SSD MoE at high speed - The wait is over!

37 Upvotes

The wait is over!

Introducing Run My AI & TokenWing.

An inference engine that lets you offload over 60% of routed experts of mixture of experts models into SSD, yet achieve very low streaming latency...

And a GUI wrapper aimed at providing the best local AI experience.

Benchmarks: Ling 3.0 Flash 120B MXFP4 running on 12Gb VRAM + 32GB DDR5 RAM laptop at 45-50 tok/s decode and 900 tok/s prefill

Qwen 3.8 Flash Next 3.5bpw running on 8GB VRAM + 32GB DDR5 RAM laptop at 35-40 tok/s and 600-700 tok/s prefill

Flint 40B UD-Q4_K_XL running on a laptop with 24GB DDR4 RAM + 4GB VRAM at 30 tok/s decode and 400 tok/s prefill

All without MTP! And benchmarks are by users*

check the website out: https://runmyai.org

Also, the autotuner is still work in progress...

A common question, does it degrade SSD?

Nope! it doesn't... There's a thing called TBW rating. Well, your SSD degrades only when it writes. And the engine only reads....


r/LocalLLM • • 6h ago

Discussion Nvidia doesnt want you to use local ai

0 Upvotes

They delete everything about local ai


r/LocalLLM • • 6h ago

Discussion Which harness or ide is the best for coding now? (Local and cloud models)

0 Upvotes

I used to use lots of cursor about 2 years ago or something (since sonet 3.5 lol) and at that time cursor seems like the only thing that get my projects to work. Since then i ve been switching to different editor but not really find that feeling anymore. I use zed editor but it seems to not very polished and lots of bugs even tho its fast. I use vscode with cline and roo code but honestly i dont feel it (all of those lack the check point reversed like cursor has even tho now i use git all the time but it feel beter to go back to check point by conversation). Now im using deepseek harness most of the time since it has a lot of plug in and i can customize it myself and it works very well with local models.( i have a quite ok system 2x3090s + wrx80 sage mainboard + threadripper 3955x + 256gb ddr4 3200 bus ecc so i can run deepseek v4.1 1017 exl3 3.5 bdw with maya engine). I think deepseek harness is it for me now. What is your experience and what is your thoughts of what is the best code editor/harness right now? (Im relatively new to this since i dont go to school for coding most of my experience is i taught myself to code/vibe code) lets discuss


r/LocalLLM • • 23h ago

Discussion Best Local AI Setup for Coding on Linux?

2 Upvotes

Hello,

After doing some research, it looks like the coding options jump from Qwen3.6-27B, which can run on a 32 GB RTX 5090 with DDR5 system memory, to GLM-5.3-Flash or DeepSeek-V4.1-Flash. That feels like a pretty big leap.

Would anyone recommend a local AI server setup for Linux? I’d appreciate any suggestions.


r/LocalLLM • • 9h ago

Discussion hmmm. New open model. Anyone tried it??

Post image
14 Upvotes

r/LocalLLM • • 15h ago

Project Windows tray app for you local model switch (without manual switching)

0 Upvotes

Hello, guys, sorry for my English, not a native speaker. I want to share with you application I made for myself.

For a couple of years I was a bit enthusiastic about local models (firstly image gen, then i tried llms) and pretty paranoid about cloud providers trying to monopolize LLMs despite their own services could be unreachable, "at capacity", down, banned, has an allowance cut etc. Also we are vulnerable to any dumb decision made by LLM provider or a government.

So I'm experimenting with local models that fits my 16 Gb VRAM of home PC's 5070Ti.

The problems that I faced:

  1. Home PC stays at home while I'm mostly working in office on my Mac mini or MacBook Air.
  2. With my GPU I'd rather prefer to use different models for different tasks. For example I better use Ornith 1.5 9B over Qwen 3.8 27B for simple tasks of file sorting, renaming etc.
  3. Changing models on llama.cpp remotely is quite uncomfortable and I'm lazy.

I've made some deep researches on some open source solution that can address my demands but didn't find anything that suits me fully.

So I made it for myself. Small tray app on Go with web-panel:
- It scans my models folder and lets setup any models;
- allows to specify arguments to run for each models separately;
- allows to setup different profiles for one model (e.g. different context length, with MTP or without, and any other llama-server parameters)
- uses Tailnet to connect with client machines;
- webpanel is affordable from client machines in my Tailnet network too so I can setup profiles and add models remotely;
adds to opencode, pi or another OpenAI format compatible harness each model and profile in the list;
- aaand the main purpose: when I switch model in Pi or Opencode host unloads previous model and load a new one without any additional actions. No need to open web-panel, start any scripts via ssh, open RDP or anything else.

The only problem is with pi — with this machinery it allows you to set up either reasoning effort medium or non-thinking. That's especially painful for 3.8 27B that needs high effort to avoid loops (as far as I remember, maybe it changed idk). It's an open issues in pi repo since 9th October, their bot closed it for "new user" reasons (lol), then someone reopened it again, anyway — it's a known issue and I don't know wether maintainer will fix or not. The only elaboration is to hardcode model profile in pi settings but it won't let you switch models on host PC from agent's menu, so you would need to manage models manually again, bruh.

Anyway, OpenCode is still a good harness so I guess it's not a crucial problem, at least for me.

Tray app has autologon setting, status indication, buttons to open webpanel, unload model, choose model/profile to load, pause working.

I've added three languages to tray — Russian, English and Chinese simplified (as a kudos to their amazing open weights model which I mostly use).

Technical details, setup guidelines are in the repo — https://github.com/strigov/strigov-locallm. I'm a lawyer so yes, of course Claude is a main contributor. Readme in repo is also written by Claude.

As the app solves concern in a pretty geek sphere I've decided not to make troubles with installation packages and automations. So the host install process sets up via repo script, and some manual actions on clients. You can charge this process to an agent anyway so I don't see any problem here.

Any observation, reviews and suggestions are appreciated, have a good day, hope this could be helpful to someone


r/LocalLLM • • 16h ago

Question Ryzen ai 9 hx 470 pro with 890 igpu and 48gb ram

0 Upvotes

I recently purchased this laptop and would really appreciate some suggestions on what models I can run on this machine to use with hermes and opencode.
So far I am working with glm 4.7 via lemonade server and some small model via flm list for npu. These are great at chat but when i use these for actual work, they drop to 4-5 tok/s.
Any suggestions would be greatly appreciated.


r/LocalLLM • • 23h ago

Question A comparable local setup for someone using $100 monthly claude subscription

43 Upvotes

I see people switching to local hardware , my question is at what usage level and pattern, it is cheaper to run it locally without impacting your work speed.

--
My current laptop and Mac mini are old but they work for me since i only use it for development while relying on Claude and Cursor for GenAI. If i were to rely on local LLM, what is the minimum hardware, I should buy to ensure that I am not slowed down.

Notes from responses:

Hardware levels mentioned

  • Tinkering (cheap): a 3060 12GB or 5060 Ti 16GB with 64–128GB of DDR4 RAM on an old Xeon board.
  • Usable for real work: a 128GB Strix Halo box (about $3.3K), a 3090 or 4090 with 96–128GB of RAM, one or two R9700 cards, or a 128GB Nvidia box (about $5K).
  • Serious: a 512GB Mac Studio (about $22K), two M5 Ultras, or two to three RTX 6000 Pro cards.

The approach that works: hybrid

  • Split the work: the cloud model plans, coordinates and reviews, and local models do the repetitive bulk work.
  • Example 1: one user runs two R9700 cards with 96GB of RAM and lets Fable direct several local Qwen workers. They went from hitting the weekly limit of Claude’s top plan to struggling to use up a Max plan. Their tips:

    • The cloud model reviews only the changes, not everything.
    • Compact the conversation often.
    • Turn off memory features.
    • Write precise instructions.
  • Example 2: a $1,500 used PC with a 3090 handles a brokerage’s client emails and documents locally. A $20 Claude plan checks the work without seeing client files, and the setup replaced a part-time contractor.

Good reasons to go local (none of them about saving money)

  • Privacy: a firm building on-premises AI for law firms says drafting a shareholders’ agreement takes about 4 minutes on a $4K machine, or about 20 minutes with the agent pipeline. About $10K of hardware makes that 2–3× faster. Good templates plus a short human review make the output client-ready.
  • No usage limits: you’re limited only by your power bill.
  • A hedge against future price rises and vendor lock-in: one user compares it to insurance. Others reply that preparing years early can just waste money.
  • Learning and hobby use, uncensored models, and control over when the model changes.
  • Hidden cost: the time you spend setting up and maintaining it.

r/LocalLLM • • 1h ago

Discussion Can open source models silently infect your PC?

0 Upvotes

e.g. they are trained to use some open source libraries for your next project which include some sophisticated backdoors which are not easy to spot. You execute your software and boom!

Regards


r/LocalLLM • • 17h ago

Question Anyone use Kev?

1 Upvotes

I've been researching Jev and its open version Kev. It looks interesting, but I'm struggling to figure out why locally you'd use it (unless extremely time sensitive) vs. using a lightweight standard LLM like Ling 3.0 Tiny. Anyone?