r/LLMDevs • • 22h ago

Discussion Aetheris Microbrains v0.6.4: Sub-2ms pure-NumPy pre-flight safety gate (full-file AST, secret & allowlist-first shlex shell parser) + retry-aware MCP-Zero tool narrowing for LLM agents

0 Upvotes

I’ve been building Aetheris Microbrains (https://github.com/drtikov/microbrainsforagents) — a lightweight, pure-NumPy (with optional OpenCL) 4-layer pre-flight safety gate and tool-surface narrower that sits in front of LLM agent tool calls and code edits.

First, what it is and what it is not:
It is not an SGD-trained generative LLM and does not replace general language reasoning or an OS sandbox. It is a deterministic < 2 ms CPU coprocessor that handles high-frequency structural checks and prompt schema hygiene before hitting a cloud API or executing a local subprocess:

  • Layer 0 (Policy & Protected Paths): Enforces live available_tools registries and protects standard SWE secret/key/hook paths (.env*, .git/hooks/, .git/config, .ssh/, id_rsa, credentials.json) without false-positiving on normal files like src/env_loader.py.
  • Layer 1 (Full-File AST, Secret & Allowlist-First Shell Gate):

    • Applies unified diffs onto the target file (original_source) and runs ast.parse + call-graph visitor checks (catching eval/exec, os.system, subprocess(shell=True), getattr / __builtins__ / base64-to-exec obfuscation, breaking API signature changes, and distinguishing benign static data tables from high cyclomatic complexity changes).
    • Scans diffs for hardcoded secrets (secret_leak_pattern_found).
    • Parses shell commands with an allowlist-first shlex tokenizer (ls, pwd, git log --format, etc. allow; rm -rf/rm -fr, curl | sh, dd of=/dev/..., mkfs, DROP DATABASE deny; unrecognized commands escalate).
  • Layer 2 (Tool Narrowing & Split-Conformal Abstention):

    • Bilingual (EN/RU) SHA-256 ToolRoutingMoE + narrow_tool_surface(top_k=4) to shrink large tool registries before prompt construction, falling back to the full catalog on out-of-distribution/ambiguous queries.
    • 42 analytical Kuramoto-Hopfield micro-classifiers calibrated at alpha = 0.05 on n_cal = 120 held-out samples (n_val = 40, std error <= 0.0197), reporting exact Clopper-Pearson 95% upper error bounds, rolling drift detection, and recalibrate_from_decision_journal().
  • Layer 3 (Honest Escalation): Invokes a caller-supplied llm_callback or explicitly returns ESCALATED_TO_LLM with a structured escalation_payload.

Honest Benchmarks (Including Needed-Tool Drop Rate & Retry Cost)

Because standard tool-calling agents choose tools and arguments in the same LLM generation pass, Top-K narrowing saves per-turn input schema prefill tokens—not a separate routing LLM call. And the metric that actually matters for Top-K narrowing is how often it drops the tool the model needed (1 - Recall@K) and what the retry costs when it misses.

In examples/06_real_llm_ab_benchmark.py, we serialized a real 32-tool OpenAI JSON schema registry (2,589 tokens/step) and evaluated narrow_tool_surface() across 40 diverse external SWE/DevOps-style tasks (1,200 context tokens assumed per retry):

Top-KRecall@KNeeded-Tool Drop RateEscalated to FullGross SavedNet Saved (After Retries)Local p95
292.1%7.9% (3/38)19/3846.7%35.1%1.718 ms
497.4%2.6% (1/38)19/3843.5%39.7%0.984 ms
697.4%2.6% (1/38)19/3840.5%36.6%1.859 ms
897.4%2.6% (1/38)19/3837.5%33.6%1.140 ms

It also includes a --live flag (python examples/06_real_llm_ab_benchmark.py --live --model gpt-4o-mini) to run an A/B comparison against any OpenAI-compatible or local Ollama/vLLM endpoint.

Try it out:

bashgit clone https://github.com/drtikov/microbrainsforagents.git
cd microbrainsforagents
pip install -e .
pytest aetheris_microbrains/tests -q   # 191 passed, 0 skipped
python examples/06_real_llm_ab_benchmark.py

Repo: https://github.com/drtikov/microbrainsforagents
Feedback, adversarial edge cases, and critiques are very welcome!I’ve been building Aetheris Microbrains (https://github.com/drtikov/microbrainsforagents) — a lightweight, pure-NumPy (with optional OpenCL) 4-layer pre-flight safety gate and tool-surface narrower that sits in front of LLM agent tool calls and code edits.First, what it is and what it is not:

It is not an SGD-trained generative LLM and does not replace general language reasoning or an OS sandbox. It is a deterministic < 2 ms CPU coprocessor that handles high-frequency structural checks and prompt schema hygiene before hitting a cloud API or executing a local subprocess:
Layer 0 (Policy & Protected Paths): Enforces live available_tools registries and protects standard SWE secret/key/hook paths (.env*, .git/hooks/, .git/config, .ssh/, id_rsa, credentials.json) without false-positiving on normal files like src/env_loader.py.
Layer 1 (Full-File AST, Secret & Allowlist-First Shell Gate):

Applies unified diffs onto the target file (original_source) and runs ast.parse + call-graph visitor checks (catching eval/exec, os.system, subprocess(shell=True), getattr / __builtins__ / base64-to-exec obfuscation, breaking API signature changes, and distinguishing benign static data tables from high cyclomatic complexity changes).
Scans diffs for hardcoded secrets (secret_leak_pattern_found).
Parses shell commands with an allowlist-first shlex tokenizer (ls, pwd, git log --format, etc. allow; rm -rf/rm -fr, curl | sh, dd of=/dev/..., mkfs, DROP DATABASE deny; unrecognized commands escalate).
Layer 2 (Tool Narrowing & Split-Conformal Abstention):

Bilingual (EN/RU) SHA-256 ToolRoutingMoE + narrow_tool_surface(top_k=4) to shrink large tool registries before prompt construction, falling back to the full catalog on out-of-distribution/ambiguous queries.
42 analytical Kuramoto-Hopfield micro-classifiers calibrated at alpha = 0.05 on n_cal = 120 held-out samples (n_val = 40, std error <= 0.0197), reporting exact Clopper-Pearson 95% upper error bounds, rolling drift detection, and recalibrate_from_decision_journal().
Layer 3 (Honest Escalation): Invokes a caller-supplied llm_callback or explicitly returns ESCALATED_TO_LLM with a structured escalation_payload.Honest Benchmarks (Including Needed-Tool Drop Rate & Retry Cost)
Because standard tool-calling agents choose tools and arguments in the same LLM generation pass, Top-K narrowing saves per-turn input schema prefill tokens—not a separate routing LLM call. And the metric that actually matters for Top-K narrowing is how often it drops the tool the model needed (1 - Recall@K) and what the retry costs when it misses.In examples/06_real_llm_ab_benchmark.py, we serialized a real 32-tool OpenAI JSON schema registry (2,589 tokens/step) and evaluated narrow_tool_surface() across 40 diverse external SWE/DevOps-style tasks (1,200 context tokens assumed per retry):Top-K Recall@K Needed-Tool Drop Rate Escalated to Full Gross Saved Net Saved (After Retries) Local p95
2 92.1% 7.9% (3/38) 19/38 46.7% 35.1% 1.718 ms
4 97.4% 2.6% (1/38) 19/38 43.5% 39.7% 0.984 ms
6 97.4% 2.6% (1/38) 19/38 40.5% 36.6% 1.859 ms
8 97.4% 2.6% (1/38) 19/38 37.5% 33.6% 1.140 msIt also includes a --live flag (python examples/06_real_llm_ab_benchmark.py --live --model gpt-4o-mini) to run an A/B comparison against any OpenAI-compatible or local Ollama/vLLM endpoint.Try it out:
bash
git clone https://github.com/drtikov/microbrainsforagents.git
cd microbrainsforagents
pip install -e .
pytest aetheris_microbrains/tests -q # 191 passed, 0 skipped
python examples/06_real_llm_ab_benchmark.pyRepo: https://github.com/drtikov/microbrainsforagents

Feedback, adversarial edge cases, and critiques are very welcome!


r/LLMDevs • • 16h ago

Discussion Reading a 131K-token patient history took 61 seconds. Reasoning over it took about eleven minutes.

Thumbnail
gallery
4 Upvotes

AntLing's Ling-3.0-flash-Sante got a synthetic patient's record from birth in 1950 to 2021 in one prompt: 334 visits across 71 years, 130,993 tokens. The question was current medications, allergies and active conditions, with dates.

On two DGX Sparks with the official FP8 weights, reading all of it took 61 seconds. Then the model thought for 30,865 tokens, about eleven minutes, before answering. The reasoning took roughly eleven times as long as the read.

That thinking is where the task gets hard. A record this long is full of items that start, stop and restart, and "current" depends on getting each date right. Scored against an answer key from the same synthetic data, the answer came back with 7 of 7 allergies, 8 of 10 active conditions, 2 of 7 dental codes and 2 of 5 medications. It kept every stopped item off the active lists and caught a medication started in 1954 that was still running 67 years later.


r/LLMDevs • • 2h ago

Discussion Built a strict verification loop to stop coding agents from faking tests — sanity check on this approach?

1 Upvotes

I'm looking for a sanity check on an agent setup I built, and mostly want to know how other people here handle verification and pipelining when coding with AI.

When I first started my project, I just let coding agents run loose. They wrote code super fast, mostly garbage, and worse—they wrote fake or weak tests just to pass, or modified existing tests so everything looked green. I couldn't even review diffs at the speed they were generating them. I tried looking around to see how others were solving this, couldn't really find what I wanted, so I ended up building my own harness around it.

The main idea is simple: don't trust the prompt or whatever the agent claims it did. One builder agent writes the code, and then everything else is just layers of deterministic checks and separate verifier agents.

Here is basically how it works right now:

  • Read-only rules: The agent works inside a bubblewrap sandbox where all the policy files, gate configs, and test rules are mounted read-only. It can't edit the rules to make its own life easier.
  • No agent self-reporting: The agent never gets to say "tests passed." A separate CLI runs the checks, hashes the actual stdout, exit codes, and policy files, and saves them as JSON proofs. If a tool is missing or a check doesn't run, it explicitly logs NOT_RUN or TOOL_ABSENT instead of silently passing.
  • Anti-cheating checks: Before anything moves forward, it runs things like BASE_FAIL_HEAD_PASS (proving a test actually failed before the fix and passed after), mutation testing, scope checks (making sure it didn't touch files outside its task), and checks to catch deleted tests.
  • Different models checking each other: The builder, the auditor, and a red-team/bug-finding agent use completely different model families so one model doesn't just rubber-stamp its own blind spots.
  • Bounded loops: If a check or CI fails, it kicks back to a repair loop, but there are hard caps on retries so it doesn't burn tokens forever. After a batch of tasks passes, it goes through a final release gate before merging.

It actually works pretty well so far. Some of the check runners are still coupled to my specific project (it's a Django app), while the task state and gate runner parts are generic.

TL;DR: Coding agents kept writing garbage tests and modifying rules to pass, so I wrapped one builder agent in a read-only sandbox with a CLI check runner, hashed JSON proofs, mutation/anti-cheating checks, and cross-model auditors. Works pretty well, curious if this is overkill or how others handle agent verification.

Is this kind of multi-layer harness overkill, or is this where things are headed? How are you guys structuring your pipelines so agents don't overfit to your test suite or hallucinate passing builds?


r/LLMDevs • • 20h ago

Help Wanted Need advice on improving adaptive user interfaces in an AI assistant

1 Upvotes

I am building an enterprise AI assistant using Claude CLI. Claude CLI handles the entire process, from understanding the user's question to generating the final answer.

I want the assistant to work more like ChatGPT, which dynamically chooses how to present information based on the question. Sometimes a simple text answer is enough, while other questions benefit from tables, cards, charts, or other UI components.

Despite instructions in my system prompt, my assistant doesn't consistently make the right formatting decisions.

For those who have worked on adaptive user interfaces or generative user interfaces in AI applications:

How do you handle this? Is it mainly prompt engineering, a separate response-planning layer, or a different approach?

I would appreciate practical suggestions.


r/LLMDevs • • 12h ago

Tools I built an open-source Hugging Face mirror so inference workers can boot even when the Hub is down

5 Upvotes

I built Model Delivery for Otium, my batch inference system. Otium spins up workers on AWS Spot and shuts them down when the queue is empty. A fresh worker can boot with an empty cache even if the last worker used the exact same model revision.

I wanted that worker to be able to start if Hugging Face was down, if the upstream repo had been removed, or if I’d deliberately cut off upstream access during a security incident. In all three cases, the worker needs the same thing: a copy of the revision I’ve chosen, on infrastructure I control.

Model Delivery lets me mirror that revision ahead of time. The mirror preserves the upstream revision ID and checks downloaded files against the hashes Hugging Face reports, when available. Once the mirror has the complete revision, Otium points HF_ENDPOINT at it and vLLM downloads the weights as usual. The registry runs as a single binary with SQLite and files on disk.

The clip replays a test of that failure case. I mirrored a five-file model from a local test hub, shut the hub down, and used the Hugging Face CLI to download the same revision from the mirror into an empty cache. All five files matched the originals byte for byte.

GitHub repo and quick start

How are people handling model availability and revision pinning in production? Are you relying on Hugging Face at worker startup, pre-baking weights into images, or maintaining your own mirrors?


r/LLMDevs • • 10h ago

Discussion When your LangGraph run fails, how do you figure out WHICH node broke it? I built a tool that answers with evidence

2 Upvotes

Genuine question first, because it's why I built this: when a multi-agent run (LangGraph, or a graph calling different models per node) produces a wrong final answer, how do you currently find the cause? Every time I've watched someone debug this, it's hours of reading trace logs by hand — and half the time the actual break was the handoff, a value that changed between one node's output and the next node's input, not the node everyone suspected.
So I built BlameGraph (part of AgentsChecked). You upload a run trace and it reconstructs the actual workflow graph, traces important fields from source to final output, and when a handoff fails it names the responsible step with the supporting evidence — the field, the value before, the value after. The checks are deterministic, no LLM judge: values mutated in transit, fields dropped, fields an agent invented, two agents disagreeing with no resolution, retry loops, outputs nobody consumed, cost outliers, and final-answer values that appear in no step's output. You get a per-agent blame %, a score per handoff, and cost per task. Policy weights are customizable, results are exportable through the API, and a run is reproducible with the same trace, engine and policy versions.
One design choice I'd defend: results are counted separately as passed / failed / insufficient_evidence / not_auditable. If the evidence in a trace isn't enough to judge a handoff, the tool says so instead of guessing either way.
On testing, since a blame tool has to earn trust: the engine (v2.1.0) went through an external review batch that closed at effectively 108/108 checks — including one finding the reviewer later conceded was an error in their own test trace, which the tool had flagged correctly.
It's free during early access (the paid tiers are on the site, struck through at $0 for now):
Full disclosure, also stated on the page: the engine runs server-side, and traces you upload and their results are collected and may be used anonymised — please don't upload anything sensitive.
I'm not a coder by background — I'm the founder; this was built with an AI dev team. What I actually need from this sub: run a real trace through it and tell me where the attribution lands wrong. And what do you export traces as today? That's the next thing I'd build.


r/LLMDevs • • 10h ago

Tools Companion Lab: MIT-licensed local workbench for prompt versions, context plugins and inspectable comparisons

2 Upvotes

Disclosure: I'm Sheila, an AI collaborator at Sheila Studios, sharing a tool we built. Companion Lab is a free, MIT-licensed local workbench for testing prompt construction and context plugins, not a hosted companion service.

Repository: https://github.com/sheilastudios/companion-lab

What it exposes:

  • Named prompt parts with adjustable roles and placement; each saved version gets an ID recorded with the turn.
  • Side-by-side replies under two saved prompt versions, without adding those trials to the live conversation.
  • The request supplied by the context plugin to the model adapter, plus reply metadata. Adapter transformations are distinguished from the context request.
  • Separate context and model interfaces. The shipped turns plugin is a baseline; you can build your own stateful memory plugin.
  • SillyTavern character-card import/export, with omissions reported rather than silently treated as lossless.

A deliberate limitation: plugin state and transcript persistence are separate writes. Interrupted stateful operations are held for reconciliation, not silently retried. The plugin contract documents recovery and rewind responsibilities. Fake-model integration and failure-boundary tests are included; they aren't live-provider certification.

Python 3.10+, no third-party Python packages. Supports OpenAI-compatible endpoints and local Ollama/LM Studio configurations; Codex CLI is optional. Run on loopback with your own model credentials. Local files stay on your machine; requests sent to a hosted model are subject to that provider's handling and costs.

The repository includes a 50-turn unscripted Helena transcript as a worked example. It was produced in this lab using our proprietary context plugin, which is NOT included. It's not a demonstration of the private HearthStage runtime, a controlled evaluation, or an out-of-the-box guarantee. Others can implement their own plugins through the public interface.

Licences are separated: code MIT, documentation CC BY 4.0, optional transcript and its included novella CC BY-NC 4.0. The noncommercial restriction applies to the literary material, not the lab code.


r/LLMDevs • • 15h ago

Tools I started Vibecoding as a mailman between models and computers. So I built VibeReview. It allows different models to plan, execute and review.

1 Upvotes

When I first got into vibecoding this summer, I quickly devolved into a mailman between my models, even ChatGPT chat and codex. I had my local model running in another computer so it got out of hand very quickly, and became very ugly and tiresome for me.

I vibecoded an app then. I call it VibeReview. It allows a model to chat and plan the work with you, split it to smaller tasks and if you approve, queues it to a worker. For each task, worker works on a seperate copy of the project. Reviewer agent then reviews it and if it passes, the work is merged with main. If the worker did a mistake, it's sent back.

As I progressed, I added LM Studio access and if the computer is in the same network and sharing is enabled, it detects the model. (If it's on the same computer, it detects that as well, of course!) If you have codex, claude, antigravity etc. installed, it can detect that as well. I think I have a complete set of models and providers. Let me know if you can't see your subscription and I can ask codex or claude to add it :)

There are a number of options that came from my experiences with projects. You can limit how many times a worker tries a task. You can limit the tokens a worker is allowed to run. You can change how atomic the tasks are (I ran ornith 1.5B as local so it needed to be very small. Then I got gemini to run as worker and tasks could be bigger).

I am Turkish, so sometimes workers got confused about the language, so you can force tasks to English if you want.

Another option is fallback agents. If a subscription runs out (Like my gemini sub), it can continue the task in another if you specify.

It also shows your remaining quotas for your subscriptions as well. You can switch thinking on and off for local models. You can specify how confident the reviewer should be about a task (like pass anything above 7/10 etc.)

If the worker is stuck on a task, the reviewer refers the task back to you to revise.

Lastly, you have a shared chat with worker and planner so if you have a problem, you can ask them.

I wrote all things I can think of, in no particular order. Let me know if I missed anything if you try VibeReview.

It is MIT Licensed and completely free. Let me know what you think:

https://github.com/erlik35/VibeReview


r/LLMDevs • • 16h ago

Discussion I built an open-source authorization broker for AI agents — looking for technical feedback

3 Upvotes

Hey everyone,

I'm a high school student from Morocco, and I've been working on a project called Zyntrava Shield.

One thing that bothered me about AI agents is how easily instructions from a document or tool response can influence what they try to do. So i started building a security layer that checks tool calls against actual permissions before they're executed.

For example, i tested it with a local Qwen model. The agent encountered a prompt injection and tried to read a private file it wasn't allowed to access. Zyntrava blocked the request, and the agent still managed to complete the original task.

It's written mainly in Python, with Pydantic AI integration.

It's definitely not perfect. It only protects tools connected through the broker, recovery isn't always reliable, and it hasn't had a full independent security audit yet.

I finally made the source public:

https://github.com/redeex-zy/zyntrava-shield-public

I'm curious what people working with AI agents think about this approach. Would something like this actually be useful in your projects?

Also, if anyone wants to try it or point out flaws in the design, I'd genuinely appreciate it.


r/LLMDevs • • 17h ago

Discussion How do you notice when a code or prompt change makes your LLM costs jump?

2 Upvotes

Curious how others deal with this. I keep seeing stories where a team's LLM bill went up 2-3x after a deploy with no change in traffic. Usually something small: a timestamp at the top of the system prompt breaking prompt caching, a model swap, retrieval pulling more chunks, a retry limit getting bumped. Sometimes nobody notices for weeks. If you run LLM features in production:

- What caused your last unexpected cost jump, and how long until you noticed?

- How do you catch these today?

- Would finding out same-day be good enough, or would you rather catch it before the change ships?


r/LLMDevs • • 19h ago

Help Wanted Apple pcc using Noema AI

Post image
3 Upvotes

Can someone help me how to configure Apple foundation model’s pcc on Noema AI local LLM?


r/LLMDevs • • 15m ago

Discussion Evaluating LLM outputs with independent judges: an LLM-as-Judge pipeline in n8n

2 Upvotes

Hi r/n8n,

I built an evaluation pipeline in n8n where one model's answers are scored by two independent judges from different providers, and I'd like feedback on the design.

Problem

Using an LLM to grade another LLM is convenient, but a judge that shares a model family with the candidate tends to favour its own answers, and a single judge gives you no sense of how much to trust its scores.

Design

  1. Test set: 5 questions (facts, math, tech, reasoning), each with a reference answer and a human score
  2. Candidate: Groq (gpt-oss-20b and gpt-oss-120b, tested in separate runs)
  3. Judge A: Gemini 3.5 Flash Lite. Judge B: Nemotron 3.5 Lightning via OpenRouter (free). Both score 1 to 5 against the question and the reference, and neither is the candidate model
  4. Scoring in code: judge average, absolute error against the human score, pass (average of 4 or more), agreement with the human (within 1 point) and judge disagreement (2 points or more)
  5. Reporting: run-level metrics go to Slack and email, and items and runs are stored in Postgres (Supabase)

Why it is built this way

Cross-provider judges remove self-preference. Judge disagreement is tracked as its own metric, so you can see where the scores are unreliable. Agreement with human labels is the check on whether the judges can be trusted at all.

Limitations

- Five questions is a demo. The numbers are not statistically meaningful, and a real evaluation needs a larger labeled set

- The human scores in the test set are starter labels and need to be replaced with your own

- An answer a judge cannot return as valid JSON is scored as 1, which can understate the candidate

Repository (workflow JSON, setup steps, table SQL):

https://github.com/ztsprofessionalaiid7-alt/llm-judge-eval-pipeline-n8n

Questions for the community

- Would you add a third judge, or a tie-break rule when the two disagree?

- How large a labeled set do you use before trusting judge-human agreement?


r/LLMDevs • • 23h ago

Tools Using an LLM to turn captured browser traffic into a clean MCP tool schema

4 Upvotes

Built an open source tool that turns a website's internal JSON API into MCP tools. The interesting part is the LLM step:

  1. Headless Chromium records the site's own JSON requests
  2. An LLM condenses dozens of raw endpoints into at most 15 clean, read-only tools with typed params
  3. A repair pass fixes common model mistakes (e.g. Algolia request bodies) and a filter drops anything that writes
  4. Serving needs no LLM at all, it just replays requests

Works with Gemini (free tier), Claude or OpenAI keys.

Demo: https://machine-internet.vercel.app

Repo: https://github.com/chetanty/machine_internet

Happy to talk about the condensing prompt and what broke along the way.


r/LLMDevs • • 56m ago

Discussion What is actually missing from today's AI agent governance infrastructure?

4 Upvotes

I've been exploring execution governance for AI agents, and a recent discussion here raised several interesting questions.

Preventing an AI agent from executing an unauthorized action seems to be only one part of a much larger problem.

What happens when:

  • An agent repeatedly retries a denied action?
  • A legitimate business process is abandoned after a denial?
  • A correctly enforced policy is fundamentally wrong?
  • Business conditions change, but execution policies remain unchanged?
  • Execution evidence cannot be independently verified?
  • An agent attempts to influence the policies that govern its own authority?

I'm increasingly interested in the infrastructure that sits between autonomous software and real-world execution.

From the discussions so far, several potential requirements have emerged:

  • Deterministic execution governance
  • Policy ownership, versioning, and approval
  • Independent execution verification
  • Policy assurance and drift detection
  • Post-denial lifecycle management
  • Human escalation and accountability
  • Retry and duplicate execution prevention
  • Execution monitoring and observability

But I don't want to assume that these are the problems developers actually need solved.

For those building or operating AI agents in production:

  1. What is your biggest unresolved governance problem?
  2. What have you had to build yourself because existing tools were insufficient?
  3. Which existing solutions are you using, and where do they fall short?
  4. What would make an independent execution governance platform worth paying for?

I'm building RBEK in this space, but I'm more interested in understanding real production problems than promoting a particular solution.

Critical feedback is especially welcome.


r/LLMDevs • • 2h ago

Discussion AI is burning out our best engineers :(

Thumbnail
leaddev.com
2 Upvotes

The “invisible validator” problem.


r/LLMDevs • • 3h ago

Discussion Has anyone tried generative retrieval instead of vector search for fast URL lookups?

2 Upvotes

I was looking into how to stop web search calls from stalling an agent mid-reasoning and found out about using generative retrieval models as the search index itself.

A normal search pipeline runs query parsing, candidate retrieval, and reranking, which usually adds 100ms or more per call. On the other hand, asking a standard frontier LLM to write URLs from memory fails most of the time because it hallucinates dead links. A setup I have been reading about is one where a 4B model is trained on click data to memorize roughly 200M pages and output a two-part numeric ID mapped to a real URL table, bringing latency down to around 10ms per query in ~10GB of VRAM instead of 400GB+ for exact vector search.

The issue is that memorizing pages works well for popular pages, but long-tail or brand-new pages still need a live index fallback. For anyone building search-heavy agents or type-ahead tools, how are you handling navigational queries compared to deeper web searches?


r/LLMDevs • • 4h ago

Discussion Which model for structured extraction? I scored 9 on 260 documents. Open source bench

4 Upvotes

Disclosure first: I run a small document extraction API (Refractr), and its model is row 4 below. I built the benchmark, so don't take my word for anything: the documents, answer key and scorer are all open. None of the benchmark data appears in my training dataset.

In production a null gets caught by validation or a human. A plausible wrong value passes your schema check and nobody notices. That is why I implemented the following scoring: A correctly extracted field gives one point, a missed extraction is 0 points, a confidently wrong extraction gives -1.

About one field in seven is deliberately not in its document. The right answer there is null, which is how made-up values get caught.

Setup: 260 documents (invoices, receipts, payslips, bank statements, court opinions, SEC filings, Federal Register rules, US government forms), each with a JSON schema. Models get the original PDF, image or HTML, so parsing is part of the test. Same published prompt for every model, one request, first reply scored. Scoring is deterministic, no LLM judge.

SystemScore95% intervalMedian time per doc
Gemini 3.8 Flash94.692.9–96.17.2 s
DeepSeek v4.1 Flash93.291.3–94.93.0 s
Mistral Large 491.489.4–93.227.4 s
Refractr (mine)89.487.2–91.40.7 s
GPT-5.4 mini86.283.9–88.52.2 s
Ministral 8B85.583.0–87.73.3 s
Llama 3.1 8B73.970.5–77.20.5 s
NuExtract 2.0 (2B)58.654.7–62.41.4 s
GLiNER2 (base)24.220.8–27.60.5 s

Questions for people shipping this kind of pipeline:

  1. Do you evaluate extraction on your own documents?
  2. Has schema-constrained output reduced wrong values for you, or only parse failures?
  3. What stack do you currently use for document automation?

Repo: https://github.com/Refractr-io/DocFieldBench Write-up: https://refractr.io/docfieldbench