r/LLMDevs • u/drtikov • 22h ago
Discussion Aetheris Microbrains v0.6.4: Sub-2ms pure-NumPy pre-flight safety gate (full-file AST, secret & allowlist-first shlex shell parser) + retry-aware MCP-Zero tool narrowing for LLM agents
I’ve been building Aetheris Microbrains (https://github.com/drtikov/microbrainsforagents) — a lightweight, pure-NumPy (with optional OpenCL) 4-layer pre-flight safety gate and tool-surface narrower that sits in front of LLM agent tool calls and code edits.
First, what it is and what it is not:
It is not an SGD-trained generative LLM and does not replace general language reasoning or an OS sandbox. It is a deterministic < 2 ms CPU coprocessor that handles high-frequency structural checks and prompt schema hygiene before hitting a cloud API or executing a local subprocess:
- Layer 0 (Policy & Protected Paths): Enforces live
available_toolsregistries and protects standard SWE secret/key/hook paths (.env*,.git/hooks/,.git/config,.ssh/,id_rsa,credentials.json) without false-positiving on normal files likesrc/env_loader.py. Layer 1 (Full-File AST, Secret & Allowlist-First Shell Gate):
- Applies unified diffs onto the target file (
original_source) and runsast.parse+ call-graph visitor checks (catchingeval/exec,os.system,subprocess(shell=True),getattr/__builtins__/base64-to-execobfuscation, breaking API signature changes, and distinguishing benign static data tables from high cyclomatic complexity changes). - Scans diffs for hardcoded secrets (
secret_leak_pattern_found). - Parses shell commands with an allowlist-first
shlextokenizer (ls,pwd,git log --format, etc. allow;rm -rf/rm -fr,curl | sh,dd of=/dev/...,mkfs,DROP DATABASEdeny; unrecognized commands escalate).
- Applies unified diffs onto the target file (
Layer 2 (Tool Narrowing & Split-Conformal Abstention):
- Bilingual (EN/RU) SHA-256
ToolRoutingMoE+narrow_tool_surface(top_k=4)to shrink large tool registries before prompt construction, falling back to the full catalog on out-of-distribution/ambiguous queries. - 42 analytical Kuramoto-Hopfield micro-classifiers calibrated at
alpha = 0.05onn_cal = 120held-out samples (n_val = 40, std error<= 0.0197), reporting exact Clopper-Pearson 95% upper error bounds, rolling drift detection, andrecalibrate_from_decision_journal().
- Bilingual (EN/RU) SHA-256
- Layer 3 (Honest Escalation): Invokes a caller-supplied
llm_callbackor explicitly returnsESCALATED_TO_LLMwith a structuredescalation_payload.
Honest Benchmarks (Including Needed-Tool Drop Rate & Retry Cost)
Because standard tool-calling agents choose tools and arguments in the same LLM generation pass, Top-K narrowing saves per-turn input schema prefill tokens—not a separate routing LLM call. And the metric that actually matters for Top-K narrowing is how often it drops the tool the model needed (1 - Recall@K) and what the retry costs when it misses.
In examples/06_real_llm_ab_benchmark.py, we serialized a real 32-tool OpenAI JSON schema registry (2,589 tokens/step) and evaluated narrow_tool_surface() across 40 diverse external SWE/DevOps-style tasks (1,200 context tokens assumed per retry):
| Top-K | Recall@K | Needed-Tool Drop Rate | Escalated to Full | Gross Saved | Net Saved (After Retries) | Local p95 |
|---|---|---|---|---|---|---|
| 2 | 92.1% | 7.9% (3/38) | 19/38 | 46.7% | 35.1% | 1.718 ms |
| 4 | 97.4% | 2.6% (1/38) | 19/38 | 43.5% | 39.7% | 0.984 ms |
| 6 | 97.4% | 2.6% (1/38) | 19/38 | 40.5% | 36.6% | 1.859 ms |
| 8 | 97.4% | 2.6% (1/38) | 19/38 | 37.5% | 33.6% | 1.140 ms |
It also includes a --live flag (python examples/06_real_llm_ab_benchmark.py --live --model gpt-4o-mini) to run an A/B comparison against any OpenAI-compatible or local Ollama/vLLM endpoint.
Try it out:
bashgit clone https://github.com/drtikov/microbrainsforagents.git
cd microbrainsforagents
pip install -e .
pytest aetheris_microbrains/tests -q # 191 passed, 0 skipped
python examples/06_real_llm_ab_benchmark.pyRepo: https://github.com/drtikov/microbrainsforagents
Feedback, adversarial edge cases, and critiques are very welcome!I’ve been building Aetheris Microbrains (https://github.com/drtikov/microbrainsforagents) — a lightweight, pure-NumPy (with optional OpenCL) 4-layer pre-flight safety gate and tool-surface narrower that sits in front of LLM agent tool calls and code edits.First, what it is and what it is not:
It is not an SGD-trained generative LLM and does not replace general language reasoning or an OS sandbox. It is a deterministic < 2 ms CPU coprocessor that handles high-frequency structural checks and prompt schema hygiene before hitting a cloud API or executing a local subprocess:
Layer 0 (Policy & Protected Paths): Enforces live available_tools registries and protects standard SWE secret/key/hook paths (.env*, .git/hooks/, .git/config, .ssh/, id_rsa, credentials.json) without false-positiving on normal files like src/env_loader.py.
Layer 1 (Full-File AST, Secret & Allowlist-First Shell Gate):
Applies unified diffs onto the target file (original_source) and runs ast.parse + call-graph visitor checks (catching eval/exec, os.system, subprocess(shell=True), getattr / __builtins__ / base64-to-exec obfuscation, breaking API signature changes, and distinguishing benign static data tables from high cyclomatic complexity changes).
Scans diffs for hardcoded secrets (secret_leak_pattern_found).
Parses shell commands with an allowlist-first shlex tokenizer (ls, pwd, git log --format, etc. allow; rm -rf/rm -fr, curl | sh, dd of=/dev/..., mkfs, DROP DATABASE deny; unrecognized commands escalate).
Layer 2 (Tool Narrowing & Split-Conformal Abstention):
Bilingual (EN/RU) SHA-256 ToolRoutingMoE + narrow_tool_surface(top_k=4) to shrink large tool registries before prompt construction, falling back to the full catalog on out-of-distribution/ambiguous queries.
42 analytical Kuramoto-Hopfield micro-classifiers calibrated at alpha = 0.05 on n_cal = 120 held-out samples (n_val = 40, std error <= 0.0197), reporting exact Clopper-Pearson 95% upper error bounds, rolling drift detection, and recalibrate_from_decision_journal().
Layer 3 (Honest Escalation): Invokes a caller-supplied llm_callback or explicitly returns ESCALATED_TO_LLM with a structured escalation_payload.Honest Benchmarks (Including Needed-Tool Drop Rate & Retry Cost)
Because standard tool-calling agents choose tools and arguments in the same LLM generation pass, Top-K narrowing saves per-turn input schema prefill tokens—not a separate routing LLM call. And the metric that actually matters for Top-K narrowing is how often it drops the tool the model needed (1 - Recall@K) and what the retry costs when it misses.In examples/06_real_llm_ab_benchmark.py, we serialized a real 32-tool OpenAI JSON schema registry (2,589 tokens/step) and evaluated narrow_tool_surface() across 40 diverse external SWE/DevOps-style tasks (1,200 context tokens assumed per retry):Top-K Recall@K Needed-Tool Drop Rate Escalated to Full Gross Saved Net Saved (After Retries) Local p95
2 92.1% 7.9% (3/38) 19/38 46.7% 35.1% 1.718 ms
4 97.4% 2.6% (1/38) 19/38 43.5% 39.7% 0.984 ms
6 97.4% 2.6% (1/38) 19/38 40.5% 36.6% 1.859 ms
8 97.4% 2.6% (1/38) 19/38 37.5% 33.6% 1.140 msIt also includes a --live flag (python examples/06_real_llm_ab_benchmark.py --live --model gpt-4o-mini) to run an A/B comparison against any OpenAI-compatible or local Ollama/vLLM endpoint.Try it out:
bash
git clone https://github.com/drtikov/microbrainsforagents.git
cd microbrainsforagents
pip install -e .
pytest aetheris_microbrains/tests -q # 191 passed, 0 skipped
python examples/06_real_llm_ab_benchmark.pyRepo: https://github.com/drtikov/microbrainsforagents
Feedback, adversarial edge cases, and critiques are very welcome!