r/Rag • • Sep 02 '25

Showcase 🚀 Weekly /RAG Launch Showcase

32 Upvotes

Share anything you launched this week related to RAG—projects, repos, demos, blog posts, or products 👇

Big or small, all launches are welcome.


r/Rag • • 8h ago

Discussion Should users be able to see what a RAG system ignored?

3 Upvotes

Most RAG systems show users the sources they used to generate an answer... What about the documents that RAG systems retrieved and then discarded?

Imagine asking a RAG system, about a company policy. The RAG system finds five documents uses two and ignores three. One of those ignored documents might contain information or an important exception.

Would showing users, the rejected sources make RAG more transparent or just create more confusion? I have rarely seen this topic discussed. I wonder what RAG engineers think.


r/Rag • • 9h ago

Discussion Half our RAG eval swings came from the ingestion pipeline, not the retriever

2 Upvotes

We build a document assistant for accounting firms. Customers dump PDFs, scans, Excel exports and the odd .msg file, we parse it, chunk it, embed it into pgvector and answer questions on top. Our eval is about 400 questions with known answers, scored with Ragas every night.

For weeks recall moved by 6 or 7 points between runs with no change to chunking, embeddings or prompts. We kept tuning the retriever. The retriever was fine.

The problem was ingestion. Three things kept changing under us.

Parser versions. Docling and LibreOffice got updated with the base image, and a minor release changed how tables were split. Same PDF, different chunks.

OCR. Scanned docs went through Tesseract on whatever node picked up the job, and two nodes had different language packs installed. Some scans came out as garbage on one and fine on the other.

Shared workers. One customer's 900 page scan could hog a worker for an hour, and the nightly reindex for everyone else timed out halfway. So the eval ran against a half built index.

What we do now. Ingestion runs in its own Linux VM per customer, started from one frozen snapshot with pinned versions of Docling, LibreOffice and Tesseract. The snapshot id is stored with every chunk, so when a score moves we know right away if the parser changed. When we upgrade the parser we build a new snapshot, reindex the eval set with both and compare before rolling it out. And a nasty file only takes down that one customer's run.

Side benefit we didn't plan for. Customer documents never sit on a shared machine anymore, which made our last security questionnaire a lot shorter.

The VMs are on boat.dev. We looked at Cloud Run jobs and at Kubernetes jobs with stricter images. boat won because a full VM runs LibreOffice and a headless Chrome for HTML exports without fighting the sandbox. No GPU there, so embeddings still run elsewhere, and a VM takes a few seconds to boot instead of being instant, which is fine for batch jobs.

Since then recall moves by about 1 point between runs. When it moves more, it's because we changed something on purpose.


r/Rag • • 7h ago

Showcase text-chipper: a Rust text chunker for RAG with strict token budgets, grapheme-safe splitting and bounded-memory streaming

1 Upvotes

I'm the author of text-chipper, a Rust crate (with Python bindings) that splits text into chunks for RAG pipelines. I'd like feedback from Rust folks on the design.

Design points:

- Strict token budgets: each finished chunk is re-tokenized in the exact form it is returned (including the heading-path prefix), so token_count <= max_length holds for the configured tokenizer. If a single grapheme cluster exceeds the budget, it returns an error instead of an oversized chunk.

- Grapheme-safe: it never splits a Unicode grapheme cluster, and offsets map back to the original input.

- Markdown-aware blocks: a lightweight block parser (not full CommonMark) that splits at blank lines, tracks heading paths, and keeps fenced code and list items whole when they fit.

- Bounded-memory streaming: on a 514.3 MiB file, from_file_stream() grew memory by about 56 MB, versus about 4.6 GB when loading the whole file into one Python string and calling chunk().

In my benchmark (12 workloads, 512 tokens, cl100k_base, each library's documented defaults), some Python chunkers returned chunks of 513-514 tokens. I haven't traced why, it's one benchmark rather than a general claim, and the comparison isn't equal-feature.

Where I'd like opinions:

- API design for custom token counters (currently Rust-only)

- Whether the strict-mode recount is a reasonable cost

- Anything un-idiomatic in the crate

Repo: https://github.com/kameh0use/text-chipper

crates.io: https://crates.io/crates/text_chipper


r/Rag • • 8h ago

Showcase I let an agent file and link my notes instead of building a retrieval pipeline for them

1 Upvotes

Most of my thinking happens out loud, so I stopped treating my notes as a pile to retrieve from.

I'm the developer of Zenod, so weigh this. It's built around one idea: an agent keeps the notes for me.

How it works today:

  • Each voice note or message is kept as I said it, in a dated log that is never edited
  • An agent files it under the projects and topics it touches and links it to what I said before
  • Any MCP client (Claude, ChatGPT, Cursor) can search it, and every answer points back to the note
  • The library is plain Markdown in my own Google Drive or GitHub

What I haven't solved: agents don't always save what matters unless I ask them to.

If you keep a personal knowledge base for agents, how do you catch it when the filing is wrong?

The first ten testers get 30 memories free, no card: https://zenod.dev/testers?s=P-0448


r/Rag • • 14h ago

Tools & Resources I built an open-source viewer for ParseBench 2 so you can find the exact PDF pages where 56 parsers fail and see why

2 Upvotes

I've been digging into ParseBench 2 (currently the most active document-parsing benchmark, covering 56 parsers across Text, Formatting, Tables, Charts, Layout, and Forms) to figure out which parsers actually hold up for RAG instead of just trusting summary scores.

Since LlamaIndex open-sourced the ParseBench 2 dataset, scorer, and raw parser outputs, I threw together a free, static viewer so it's easier to audit this stuff directly in the browser (no login, no backend, not affiliated with LlamaIndex or any parser vendor). You can inspect cost/latency vs. accuracy tradeoffs, drill into any page to see the exact scored errors, and compare any two parsers head-to-head on any dimension to find the exact pages where one outperforms the other.

Built it quickly in a day, so expect rough edges. Please report any bugs you hit (I already know of a few and will keep fixing them), and additional suggestions are very welcome.


r/Rag • • 19h ago

Discussion Benchmarked 9 models on extracting data out of documents, scoring wrong values as worse than missing ones. Results and open bench+dataset

4 Upvotes

Disclosure first: I run a small document extraction API (Refractr), and its model is row 4 below. I built the benchmark, so don't take my word for anything: the documents, answer key and scorer are all open. None of the benchmark data appears in my training dataset.

Why I made it: When you extract fields or metadata from documents at ingestion, a missing value is annoying but visible. A wrong value that looks plausible goes into your store and every query downstream trusts it. Most extraction benchmarks only count right answers, so a model that guesses confidently looks the same as one that leaves a blank. I wanted a score that separates them. A correctly extracted field gives one point, a missed extraction is 0 points, a confidently wrong extraction gives -1.

About one field in seven is deliberately not in its document. The right answer there is null, which is how made-up values get caught.

Setup: 260 documents (invoices, receipts, payslips, bank statements, court opinions, SEC filings, Federal Register rules, US government forms), each with a JSON schema. Models get the original PDF, image or HTML, so parsing is part of the test. Same published prompt for every model, one request, first reply scored. Scoring is deterministic, no LLM judge.

System Score 95% interval Median time per doc
Gemini 3.8 Flash 94.6 92.9–96.1 7.2 s
DeepSeek v4.1 Flash 93.2 91.3–94.9 3.0 s
Mistral Large 4 91.4 89.4–93.2 27.4 s
Refractr (mine) 89.4 87.2–91.4 0.7 s
GPT-5.4 mini 86.2 83.9–88.5 2.2 s
Ministral 8B 85.5 83.0–87.7 3.3 s
Llama 3.1 8B 73.9 70.5–77.2 0.5 s
NuExtract 2.0 (2B) 58.6 54.7–62.4 1.4 s
GLiNER2 (base) 24.2 20.8–27.6 0.5 s

Two things I'd like to hear from people here:

  1. How do you catch wrong extracted values in your own pipeline today? Confidence scores, a second model, sampling by hand?
  2. If you run it on your own stack (parser plus model), I'll add the result. It takes any OpenAI-style endpoint or a Python function, so a full pipeline counts as a system.

Repo: https://github.com/Refractr-io/DocFieldBench

Write-up: https://refractr.io/docfieldbench


r/Rag • • 22h ago

Discussion Does agentic retrieval actually beat a simple reranker?

5 Upvotes

Abdullahi Dattijo tested four retrieval approaches on 298 questions over 2,000 arXiv papers. The metric was whether the target paper appeared in the top five results, not whether the system generated a correct answer.

Dense retrieval reached 95.6% Recall@5. Reranking the top 50 candidates with a small cross-encoder reached 97.0%. Multiquery rewriting scored 93.6%, while an agent allowed up to three searches reached 96.9%.

The agent gained about 1.2 percentage points over dense retrieval, with a 95% bootstrap interval from −0.8 to +3.5. Its benefit remained uncertain. The reranker’s recall gain was also uncertain, but its ranking improvement was clearer. MRR@10 rose from 0.895 to 0.959, the only improvement whose interval excluded zero.

The useful detail is what extra searches gained and lost. Per run, the agent recovered 7–8 papers dense retrieval missed, but lost 3–5 papers it had already found. Some rewrites introduced unsupported terminology that pulled retrieval toward distractors. Extra searches could undo a good first result.

Cost wasn’t the main problem. The agent averaged roughly $0.00009 per query. Both agentic retrieval and reranking took around 1.4–1.5 seconds on the test setup. The local reranker incurred no LLM API fees, though compute still costs something.

The limitation matters. Questions were generated from abstracts, and retrieval used titles and abstracts rather than full papers. Dense retrieval already missed only 13 questions, leaving little headroom.

For anyone testing this in production, count both recovered hits and previously correct hits that get displaced. Inspect the rewritten queries on those regressions, then compare the net improvement against a reranker baseline. That gives you a concrete reason to decide whether the agent loop is worth keeping.


r/Rag • • 1d ago

Discussion My RAG pipeline was working fine until I realised its knowledge was six months old

14 Upvotes

I have been working on a RAG pipeline that pulls information from documentation sites, product pages and company websites.

The initial setup was pretty straightforward. Scrape the pages, convert everything to Markdown, chunk the content, generate embeddings and push it into the vector database.
The problem showed up later.

A product changed its pricing, documentation got updated and a few pages were removed. But the vector database was still serving the old information.
I initially thought about running a complete re-index every week, but that feels wasteful when only a small percentage of the content actually changes.
I'm now looking at a setup where I can monitor the original pages, identify what changed and only reprocess the relevant content.

I have been looking at Firecrawl, Apify and context.dev for different parts of this workflow.

Context.dev caught my attention because it has scraping and website monitoring under the same API, so I wouldn't need to build another integration just for change detection.

Still figuring out the best architecture here.

For people running production RAG systems, how are you keeping your external knowledge base fresh without rebuilding the entire index every time?


r/Rag • • 14h ago

Discussion Unsiloed's $7.5M seed round, and why the document layer comes before retrieval

0 Upvotes

We're Unsiloed, and we've just announced a $7.5M seed round led by Felicis to make documents as computable as databases.

A lot of RAG problems we see start before retrieval. A table gets chunked as flat text and loses which number belongs to which row and column, or a two-column page gets read in the wrong order, and no amount of reranking fixes that. We focus on that step. Documents come back as clean markdown or as structured JSON against your schema, with the page location for each value, so an answer can point at the exact place in the source.

If you're dealing with scanned tables, filings or messy multi-column PDFs in your pipeline, we're happy to talk through what works.

Announcement and launch video ¡ TechCrunch coverage


r/Rag • • 16h ago

Tools & Resources Looking for testers: run and manage local AI agents, models and GPU servers with Yorath

1 Upvotes

Hey everyone! 👋

We’ve been building Yorath, an open-source operating environment for running AI models and agents on your own infrastructure, and we’re looking for early users willing to test it and give honest feedback.

What it does:

Yorath lets you run local AI models, connect agents such as Codex, OpenClaw and Hermes, manage multiple machines with GPUs, and control access to models and sensitive data from one place.

Think of it as an operating layer for your AI infrastructure: models, agents, compute, permissions and audit logs under one roof.

Why we built it:

Running one local model is relatively easy. Managing several models, agents and GPU servers securely is not.

You quickly end up with separate dashboards, API keys, containers, network rules and logs. More importantly, access restrictions are often implemented inside prompts — where they can be ignored or bypassed.

We wanted a system that installs quickly, keeps the infrastructure understandable, and enforces data-access rules outside the model.

Tech stack and architecture:

  • Containerized deployment with Docker
  • Local and OpenAI-compatible model endpoints
  • OpenAI-compatible API for applications and agents
  • Isolated containers for connected agents
  • gVisor sandbox for the built-in Developer Agent
  • Gateway-level access policies and data classification
  • Multi-node GPU cluster support
  • Web interface for models, agents, nodes and policies
  • Central journal for model and agent calls

Key features:

  • One-command installation: curl -fsSL https://get.yorath.io | sh
  • Automatically detects the server’s hardware and selects an appropriate local model
  • Installs, runs and monitors local models
  • Works with Codex, OpenClaw, Hermes, Ollama and OpenRouter
  • Provides a single endpoint regardless of how many GPU servers are behind it
  • Adds new servers to a cluster using a one-time join token
  • Supports mixed hardware across cluster nodes
  • Enforces access to data, tools and models before a request reaches the model
  • Blocks calls that violate data-classification policies
  • Runs connected agents in isolated, non-root containers
  • Lets you disable internet access for connected agents
  • Records calls in a central audit journal
  • Sends no telemetry

Current requirements:

  • Ubuntu 24.04 on x86-64
  • Minimum 4 vCPU and 8 GB RAM
  • A GPU for practical use with Codex and connected agents
  • Docker 29+ for building from source

What we’re looking for:

  • Feedback on the installation and setup experience
  • Testing on different server and GPU configurations
  • Real-world testing with Codex, OpenClaw, Hermes and custom agents
  • Multi-node cluster testing
  • Security and policy edge cases
  • Bug reports accompanied by yorath diag
  • Feedback on what’s confusing, missing or unnecessarily complicated

If something breaks, yorath doctor runs the initial checks, while yorath diag creates an archive that can be attached to a bug report.

Repository: https://github.com/yorath-os/yorath

Website and documentation: https://yorath.io

Yorath is still early, so we’re particularly interested in discovering the rough edges that only appear outside our own environment.

Happy to answer any questions!

Yorath Community is open-source under the Apache License 2.0. Core agent registry, data classification, access policies and the audit journal are included in the Community edition. No telemetry.


r/Rag • • 1d ago

Showcase Looking for feedback on my RAG "longevity assistant" app

1 Upvotes

Hi all,

I built a longevity assistant using Google's Embeddings model. It is really cool and I was able to easily get a RAG system going with thousands of published medical journals. I do have a software dev background, so it there was not much friction in getting this built.

Would love to get some feedback on your thoughts? I think I've got hallucinations down to near non-zero. Thank you!

URL: https://www.modernmedlife.com/longevity-assistant


r/Rag • • 1d ago

Tools & Resources In RAG, what goes in determines what comes out. Prajnya: a local ingestion engine for scientific PDFs

6 Upvotes

I'm a researcher with background in Developmental biology, biochemistry and biophysics. I kept finding that retrieval failures started at ingestion: figures dissolved into prose, tables flattened, captions merged with body text, reference lists polluting search. So I built the ingestion layer first. I am now building the RAG part and after having dealt with many issues, I feel a bit confident in sharing this project with you all. Prajnya is open source (Apache-2.0) and fully offline by default.

How it works

  1. A CPU pre-pass classifies each document (paper, book or chapter) and every page.
  2. Plain prose comes straight from the PDF's text layer. Scans and visually complex pages go to a local vision model.
  3. A layout model finds figures and tables, and each is cut out and handled as its own object.
  4. Everything lands in one JSON per document, with a documented schema.

What you get, and why it matters for retrieval

  • Evidence objects, not chunks. Figures are cropped and captioned with stable IDs, and panels are grouped under the label a reader cites ("Fig. 3"). Tables carry a reconstructed grid, so numbers are searchable while the image stays available to a multimodal model. Tables that run onto the next page keep their headers.
  • Provenance. Every item records its page and how it was read (text layer, model read or repaired read), because those differ in reliability.
  • Citations linked conservatively. In-text citations, numbered and author-year, link to the reference list. A wrong link is worse than none, so ambiguous ones stay unlinked.
  • Metadata verified, not believed. The model's reading of the title page, the PDF text and an optional authority (Zotero, or Crossref/OpenAlex sent only the DOI or title) are reconciled, with the source of each value recorded.

What it doesn't do (yet)

  • No chunking, embedding or retrieval. That’s the next layer, Shastrasaara (in development), which builds on these evidence objects.
  • PDFs only. No DOCX, HTML or slides.
  • It isn’t one-click. Local mode needs llama.cpp and the model weights. Cloud models are optional, per role.
  • It doesn’t race to be the fastest. It aims for the most faithful data, and would rather take longer than hand you a wrong table, caption or citation.

How it compares

  • NotebookLM: for public papers and quick cited Q&A, use it. But your PDFs leave your machine, and you get a chat, not structured data you can index your own way.
  • GROBID: the standard for metadata and references, fast on CPU. Its weak spot is pages where figures and tables break the reading order. Prajnya is built around those pages.
  • Docling, Marker and MinerU: solid general converters. Prajnya is narrower and specific to scientific documents, aimed at what a literature RAG needs to cite and trace.

Repo: Prajnya-Academic-PDF-Processor

Question for this sub: how do you index figures and tables in scientific RAG? Captions, images, both, or one-hop expansion from the text that cites them?


r/Rag • • 1d ago

Discussion RAG over a product catalog: how do you stop it quoting yesterday's prices?

7 Upvotes

Shop assistants and product Q&A bots are a common RAG use case, but prices and stock change daily, sometimes hourly. If the price sits inside an embedded chunk, the bot can confidently quote an old one.

How are people handling the volatile fields:

- keep price and stock out of the chunks and fetch them at answer time?

- re-embed changed products nightly?

- something else, like a tool call to the store API?


r/Rag • • 1d ago

Discussion Web pages as RAG sources: how do you keep the index fresh when the pages keep changing?

6 Upvotes

Documents in a drive change now and then. Web pages change constantly: prices, specs, policies, pages that disappear. A RAG index built from a crawl starts going stale the day after.

For anyone with web sources in production:

- full re-crawl on a schedule, or change detection per page?

- how do you handle pages that 404 or redirect: delete the chunks or keep them?

- do you store the crawl date in chunk metadata and use it at retrieval?


r/Rag • • 1d ago

Discussion I priced 100 web search APIs. The cheapest is $0.25 per 1,000 searches and the median is $2.83

0 Upvotes

Picking a search API for an agent turned into opening pricing page after pricing page, so we went through 100 of them and worked out what 1,000 searches cost on each one's entry plan. The spread is silly: the priciest costs 90x the cheapest, and Tavily, Exa and Brave all sit above the $2.83 median.

These are the 10 cheapest paid ones. Three tie at $0.80, so you get 11:

# Provider $ per 1,000 Pricing Returns Free tier
1 Ceramic $0.25 Pay as you go Snippets 1,000 queries, no card
2 Cloudflare $0.25 Pay as you go Snippets None listed (resells Ceramic)
3 FlyCrawl $0.29 $29/mo Not stated 100 searches/month
4 Geonode $0.39 $3.90/mo Snippets 1,500 requests/month
5 TickerTrends $0.39 $10 pack Snippets 100 credits, card needed
6 Jina $0.50 (est.) $50 pack Full page 10M tokens, non-commercial
7 DeepSeek $0.75 Top-up Full page None listed
8 UnSearch $0.76 $19/mo Not stated 5,000 queries/month, or self-host
9= Dipper $0.80 Top-up from $10 Full page 1,000 requests
9= Searlo $0.80 $3.99 pack Snippets 3,000 credits
9= Serpex $0.80 Top-up from $5 Full page 200 credits, no card

"Full page" means you get the page text back with the results, which saves a separate scrape call for RAG. Jina's price is worked out from its credits because it doesn't quote one per search.

Four more are free outright with rate limits: AgentWeb, LangSearch, ssearch and TinyFish.

Full list of all 100, with free tiers, search modes and a CSV: https://voxoria.ai/web-search-api-pricing/

If any of the info is wrong/out of date, let me know - hopefully this is helpful


r/Rag • • 2d ago

Showcase How I fixed RAG hallucinations on 1000-page books: loss-bounded compression + 100% deterministic citations (running on a 4GB GPU)

76 Upvotes

standard RAG for long-form reading is fundamentally broken. if you ask an LLM to summarize a 50-page chapter, it just gives you a "gist" and hallucinates connective tissue. it drops the actual reasoning, skips the caveats, and gives you zero way to verify where the information came from.

i wanted a system where the AI compresses the text, but every single sentence is deterministically mapped back to the exact source paragraph. if the model invents a bridge between two chunks, you can immediately click the sentence, jump to the original text, and verify it.

here is the architecture i built to solve this:

  1. Ingestion & Embedding: Parsed text is chunked and embedded locally using BGE-M3 (1024-dim, FP16 CUDA).

  2. Reranking: BGE Reranker v2 (cross-encoder). Instead of just relying on cosine similarity, we score query-passage relevance directly to pick the best context.

  3. The Two-Representation Invariant: The original document is strictly immutable ("Source is paper"). The compressed reading is a distinct view ("The lens is tinted").

  4. Deterministic Attribution: Every synthesized sentence goes through a provenance resolver. If a sentence outputs zero source citations, it's flagged as a hallucination defect in the UI.

getting this whole pipeline (plus Kokoro TTS) to run entirely offline on my 4GB RTX 3050 was a nightmare. i had to build a strict F65 sequential await barrier to stop Node.js from OOMing at 14.6GB during async ingestion, and an F43 idle lifecycle manager to evict models from VRAM before the GPU choked.

it's open source (AGPL-3.0). you can dig into the retrieval logic or the provenance resolver here:

https://github.com/ayumu-ml/Omnitome

i'd love feedback from other RAG builders. how are you currently handling the "summary loses the source" problem? is anyone actually achieving deterministic citations at scale, or is everyone just relying on semantic similarity?


r/Rag • • 1d ago

Discussion How are you guys debugging RAG when you know the answer exists in the docs?

6 Upvotes

I've been spending a lot of time thinking about this problem lately.

Say you have a RAG pipeline and you know the answer is somewhere in your indexed documents, but the model still gets it wrong.

The usual debugging approach seems to be messing with top-k, similarity thresholds, chunk sizes, or reranking until the answer starts showing up.

But there are a few different ways things can go wrong:

  • The chunk was retrieved but fell outside top-k.
  • It got filtered out by a similarity threshold.
  • The reranker pushed it down.
  • Retrieval worked, but it got dropped before reaching the LLM.

I've been building a tool called ChunkTrace to help pinpoint which stage caused the problem.

I'm curious how others are debugging this right now. Are you logging the intermediate retrieval results, using LangSmith/Langfuse, or just inspecting things manually?

Especially interested in anyone running multi-stage retrieval with pgvector or rerankers.


r/Rag • • 1d ago

Discussion Our AI worker retried a subscription limit 900 times. Here's what we learned.

0 Upvotes

I came across an article about agentic system design this week. One sentence stuck with me:

"These failures are ordinary, and you can plan for them."

We learned that lesson the hard way, in production.

We run an AI-powered design pipeline on a Windows machine. A worker picks up tasks and executes them through a model CLI using a subscription plan. Separate operator processes handle the generated images and uploads.

On October 2, we hit the subscription's session limit.

Our worker treated the limit message as a regular error and retried the same task every second. By the end of the day, we'd accumulated roughly 900 attempts.

About two hours later, every new process launch started failing with 0xC0000142 (STATUS_DLL_INIT_FAILED). The machine only recovered after a reboot.

Initially, we blamed the model CLI. But the same command worked before and after the incident. Looking at the logs, our retry loop became the more likely explanation. We haven't proven a direct causal link, though.

We also had other infrastructure problems during that production round (28 screens, ~42 generations):

  • An upload folder was deleted, losing 7 results.
  • Two operators resumed after sleep and uploaded images to the wrong screens.
  • Operators logged successful uploads even when delivery had failed.

At least 8 generations (~19%) had to be redone because of infrastructure issues, not model quality.

The biggest lesson: A usage limit is a state, not an ordinary error.

Our worker now handles limits separately:

result = run_model_cli(task)

switch classify(result):
    case LIMIT:
        sleep_until(reset_time)
    case ERROR:
        backoff = next_step(backoff)
        requeue(task, after=backoff)

We also added:

  • PID + heartbeat-based operator locks.
  • Upload verification instead of trusting success logs.
  • Exponential backoff and retry limits.
  • A plan to introduce execution budgets, job leases and better run tracking.

We considered switching agent frameworks, but realized the problem wasn't the framework. It was missing reliability mechanisms in our existing system.

Our next production target is zero infrastructure-caused regenerations. The fixes haven't been validated in another production round yet.

With HTTP APIs, you usually get a 429 and sometimes a Retry-After header.

But with CLI tools running on subscription plans, you often just get a text message saying you've reached a usage limit.

How do you reliably distinguish a subscription limit from an actual execution failure?

And how do you determine when the limit resets?

Do you parse CLI output, track usage yourself, or use another approach?

I'd especially like to hear from people running autonomous workers with subscription-based AI CLIs in production.


r/Rag • • 2d ago

Showcase Code retrieval: two Haiku agents with grep, 77% line recall on ContextBench, ~1¢ per issue

3 Upvotes

In our ContextBench run, CodeNib retrieves code context with two Claude Haiku 5.5 agents and grep: no embeddings, no reranker, about a cent per issue.

ContextBench (500 tasks, 8 languages) scores how much of the human-annotated relevant code a system retrieves. This setup finds 77% of the relevant lines; the next-best system on the leaderboard finds 61%.

How the two agents split the work:

  • The first follows the issue with glob, grep and file reads.
  • The second gets the issue plus a map of where the first already looked (paths and line ranges, at most 2,048 tokens), and goes after what was missed: callers, alternate paths, tests, configuration.

The search reads broadly to get that recall: its line precision is 20.5%. With an Opus repair agent on top, the full system resolves 388 of the 500 tasks, #1 among agent systems on the leaderboard as of Oct 23. The task annotations were visible during development.

Write-up, trajectories and cost receipts: https://codenib.ai/blogs/contextbench-500-tasks/


r/Rag • • 1d ago

Discussion RAG over a product catalog: how do you stop it quoting yesterday's prices?

2 Upvotes

Shop assistants and product Q&A bots are a common RAG use case, but prices and stock change daily, sometimes hourly. If the price sits inside an embedded chunk, the bot can confidently quote an old one.

How are people handling the volatile fields:

- keep price and stock out of the chunks and fetch them at answer time?

- re-embed changed products nightly?

- something else, like a tool call to the store API?


r/Rag • • 1d ago

Tools & Resources If your security team blocks hosted embedding APIs, the air-gapped self-host setup is one config flag, not the research project I expected

1 Upvotes

Our blocker was never model quality, it was legal. Our document text is not allowed to leave the VPC, and that ruled out every hosted API before we even got to compare them.

The part I expected to be a nightmare was proving the server never quietly reaches out for weights on its own. On SIE it came down to a single setting:

yaml

SIE_HF_FALLBACK: ""false"" # never reach out to HuggingFace

hfToken.create: false

You mirror the weights into your own storage first, then pin the image and install, and after that the worker pods have no path to the internet at all.

The catch is that this is real infrastructure you now own and babysit. If your data is not actually sensitive, do not put yourself through it, just use an API.


r/Rag • • 1d ago

Discussion At what point does self-hosting your embedding model actually get cheaper than the API? Did the math

1 Upvotes

Every time someone asks about embedding costs here, the answer is “just self-host” and nobody says the one number that decides it.

An API bills per token, so idle costs nothing.
A GPU bills per hour whether it’s busy or not.

Self-hosting only wins once the GPU stays busy enough. Rough math for a small embedding model:

API is ~$0.02/M tokens, so 1B tokens ≈ $20/day
A10G is ~$1/hr, so ~$24/day before overhead

Crossover is ~1.2B tokens/day.

Under that, the API wins. The RAG-specific bit is that reindexing is what gets you there. Queries alone almost never hit 1B/day, but re-embedding the whole corpus every time you change chunking or swap models does. If you reindex often, the line comes way closer.

Catch is it assumes the card is pushing ~14k tokens/sec all day. Spiky traffic means you’re paying for peak. And nobody puts ops in the spreadsheet. Someone’s awake at 2am when it falls over. What volume are people here actually at? Curious if anyone’s crossed it on queries alone.


r/Rag • • 2d ago

Showcase We open-sourced a database that handles the RAG pipeline for you (Apache 2.0)

7 Upvotes

We just released Antfly v0.3 under Apache 2.0.

We built it to make RAG over complex files easier e.g. PDFs, images, audio, and video. Instead of wiring together OCR, embeddings, vector search, reranking, and graph retrieval, Antfly handles it inside one open-source database.

It supports keyword/grep, hybrid, and graph search, and runs embedded (Lite), self-hosted, or in the cloud.

GitHub: https://github.com/antflydb/antfly

Would love feedback from anyone building RAG pipelines, especially over multimodal data.


r/Rag • • 2d ago

Discussion Lessons from running a RAG support bot for real Shopify stores: the hard part wasn't retrieval, it was knowing when to say "I don't know"

1 Upvotes

I've been building a customer-support chatbot for Shopify stores (order status, product questions, store policies) using RAG. It's now in a pilot with real merchants. A few lessons that weren't obvious to me at the start:
1. A confident wrong answer is worse than no answer.
In e-commerce, a bot that invents a return policy or a delivery date creates real problems for the merchant. So the core design question became "when should the bot refuse?", not "how do I get better answers?"
2. Retrieval score floors did more than prompt engineering.
I spent a lot of time telling the model "only answer from the context." What actually reduced hallucinations was a minimum similarity threshold: if nothing retrieved clears the floor, the bot doesn't get to answer from vibes. It says it doesn't know and offers a human handoff.
3. Build a refusal test set, not just an accuracy test set.
I keep a set of questions the bot should not be able to answer from the store's data (other stores' policies, made-up products, general-knowledge bait). Every change to chunking, prompts or models gets run against it. This caught regressions that normal Q&A tests missed.
4. Model choice mattered more than I expected for grounding.
On a smaller, cheaper model I kept seeing fabricated details even with good context. Moving to a newer model eliminated most of those failures, and the cost difference was worth it for support use cases.
5. Per-shop isolation has to be enforced in retrieval, not just in the prompt.
With multiple merchants on one system, the filter must sit at the retrieval layer. You can't rely on the model to ignore another store's chunks.
6. Show the merchant why the bot answered.
The admin can see each conversation and which document an answer came from. When the bot gets something wrong, the merchant can see whether it's a missing document or a retrieval problem, and fix it. That built more trust than any accuracy number.
7. Human takeover is a feature, not a fallback.
Merchants wanted to jump into a live conversation at any point. Designing for that from the start made "I don't know" answers acceptable instead of embarrassing.
Curious how others handle this:
How do you pick your score threshold? Fixed value, per-tenant, or calibrated on a test set?
Anyone using rerankers specifically to improve refusal behaviour?
(Disclosure: this is my own product, but not linking it here. Happy to go deeper on any of these.)