r/Rag • • 7h ago

Discussion Unsiloed's $7.5M seed round, and why the document layer comes before retrieval

0 Upvotes

We're Unsiloed, and we've just announced a $7.5M seed round led by Felicis to make documents as computable as databases.

A lot of RAG problems we see start before retrieval. A table gets chunked as flat text and loses which number belongs to which row and column, or a two-column page gets read in the wrong order, and no amount of reranking fixes that. We focus on that step. Documents come back as clean markdown or as structured JSON against your schema, with the page location for each value, so an answer can point at the exact place in the source.

If you're dealing with scanned tables, filings or messy multi-column PDFs in your pipeline, we're happy to talk through what works.

Announcement and launch video · TechCrunch coverage


r/Rag • • 20h ago

Discussion Our AI worker retried a subscription limit 900 times. Here's what we learned.

0 Upvotes

I came across an article about agentic system design this week. One sentence stuck with me:

"These failures are ordinary, and you can plan for them."

We learned that lesson the hard way, in production.

We run an AI-powered design pipeline on a Windows machine. A worker picks up tasks and executes them through a model CLI using a subscription plan. Separate operator processes handle the generated images and uploads.

On October 2, we hit the subscription's session limit.

Our worker treated the limit message as a regular error and retried the same task every second. By the end of the day, we'd accumulated roughly 900 attempts.

About two hours later, every new process launch started failing with 0xC0000142 (STATUS_DLL_INIT_FAILED). The machine only recovered after a reboot.

Initially, we blamed the model CLI. But the same command worked before and after the incident. Looking at the logs, our retry loop became the more likely explanation. We haven't proven a direct causal link, though.

We also had other infrastructure problems during that production round (28 screens, ~42 generations):

  • An upload folder was deleted, losing 7 results.
  • Two operators resumed after sleep and uploaded images to the wrong screens.
  • Operators logged successful uploads even when delivery had failed.

At least 8 generations (~19%) had to be redone because of infrastructure issues, not model quality.

The biggest lesson: A usage limit is a state, not an ordinary error.

Our worker now handles limits separately:

result = run_model_cli(task)

switch classify(result):
    case LIMIT:
        sleep_until(reset_time)
    case ERROR:
        backoff = next_step(backoff)
        requeue(task, after=backoff)

We also added:

  • PID + heartbeat-based operator locks.
  • Upload verification instead of trusting success logs.
  • Exponential backoff and retry limits.
  • A plan to introduce execution budgets, job leases and better run tracking.

We considered switching agent frameworks, but realized the problem wasn't the framework. It was missing reliability mechanisms in our existing system.

Our next production target is zero infrastructure-caused regenerations. The fixes haven't been validated in another production round yet.

With HTTP APIs, you usually get a 429 and sometimes a Retry-After header.

But with CLI tools running on subscription plans, you often just get a text message saying you've reached a usage limit.

How do you reliably distinguish a subscription limit from an actual execution failure?

And how do you determine when the limit resets?

Do you parse CLI output, track usage yourself, or use another approach?

I'd especially like to hear from people running autonomous workers with subscription-based AI CLIs in production.


r/Rag • • 20h ago

Discussion My RAG pipeline was working fine until I realised its knowledge was six months old

13 Upvotes

I have been working on a RAG pipeline that pulls information from documentation sites, product pages and company websites.

The initial setup was pretty straightforward. Scrape the pages, convert everything to Markdown, chunk the content, generate embeddings and push it into the vector database.
The problem showed up later.

A product changed its pricing, documentation got updated and a few pages were removed. But the vector database was still serving the old information.
I initially thought about running a complete re-index every week, but that feels wasteful when only a small percentage of the content actually changes.
I'm now looking at a setup where I can monitor the original pages, identify what changed and only reprocess the relevant content.

I have been looking at Firecrawl, Apify and context.dev for different parts of this workflow.

Context.dev caught my attention because it has scraping and website monitoring under the same API, so I wouldn't need to build another integration just for change detection.

Still figuring out the best architecture here.

For people running production RAG systems, how are you keeping your external knowledge base fresh without rebuilding the entire index every time?


r/Rag • • 15h ago

Discussion Does agentic retrieval actually beat a simple reranker?

5 Upvotes

Abdullahi Dattijo tested four retrieval approaches on 298 questions over 2,000 arXiv papers. The metric was whether the target paper appeared in the top five results, not whether the system generated a correct answer.

Dense retrieval reached 95.6% Recall@5. Reranking the top 50 candidates with a small cross-encoder reached 97.0%. Multiquery rewriting scored 93.6%, while an agent allowed up to three searches reached 96.9%.

The agent gained about 1.2 percentage points over dense retrieval, with a 95% bootstrap interval from −0.8 to +3.5. Its benefit remained uncertain. The reranker’s recall gain was also uncertain, but its ranking improvement was clearer. MRR@10 rose from 0.895 to 0.959, the only improvement whose interval excluded zero.

The useful detail is what extra searches gained and lost. Per run, the agent recovered 7–8 papers dense retrieval missed, but lost 3–5 papers it had already found. Some rewrites introduced unsupported terminology that pulled retrieval toward distractors. Extra searches could undo a good first result.

Cost wasn’t the main problem. The agent averaged roughly $0.00009 per query. Both agentic retrieval and reranking took around 1.4–1.5 seconds on the test setup. The local reranker incurred no LLM API fees, though compute still costs something.

The limitation matters. Questions were generated from abstracts, and retrieval used titles and abstracts rather than full papers. Dense retrieval already missed only 13 questions, leaving little headroom.

For anyone testing this in production, count both recovered hits and previously correct hits that get displaced. Inspect the rewritten queries on those regressions, then compare the net improvement against a reranker baseline. That gives you a concrete reason to decide whether the agent loop is worth keeping.


r/Rag • • 7h ago

Tools & Resources I built an open-source viewer for ParseBench 2 so you can find the exact PDF pages where 56 parsers fail and see why

2 Upvotes

I've been digging into ParseBench 2 (currently the most active document-parsing benchmark, covering 56 parsers across Text, Formatting, Tables, Charts, Layout, and Forms) to figure out which parsers actually hold up for RAG instead of just trusting summary scores.

Since LlamaIndex open-sourced the ParseBench 2 dataset, scorer, and raw parser outputs, I threw together a free, static viewer so it's easier to audit this stuff directly in the browser (no login, no backend, not affiliated with LlamaIndex or any parser vendor). You can inspect cost/latency vs. accuracy tradeoffs, drill into any page to see the exact scored errors, and compare any two parsers head-to-head on any dimension to find the exact pages where one outperforms the other.

Built it quickly in a day, so expect rough edges. Please report any bugs you hit (I already know of a few and will keep fixing them), and additional suggestions are very welcome.


r/Rag • • 12h ago

Discussion Benchmarked 9 models on extracting data out of documents, scoring wrong values as worse than missing ones. Results and open bench+dataset

5 Upvotes

Disclosure first: I run a small document extraction API (Refractr), and its model is row 4 below. I built the benchmark, so don't take my word for anything: the documents, answer key and scorer are all open. None of the benchmark data appears in my training dataset.

Why I made it: When you extract fields or metadata from documents at ingestion, a missing value is annoying but visible. A wrong value that looks plausible goes into your store and every query downstream trusts it. Most extraction benchmarks only count right answers, so a model that guesses confidently looks the same as one that leaves a blank. I wanted a score that separates them. A correctly extracted field gives one point, a missed extraction is 0 points, a confidently wrong extraction gives -1.

About one field in seven is deliberately not in its document. The right answer there is null, which is how made-up values get caught.

Setup: 260 documents (invoices, receipts, payslips, bank statements, court opinions, SEC filings, Federal Register rules, US government forms), each with a JSON schema. Models get the original PDF, image or HTML, so parsing is part of the test. Same published prompt for every model, one request, first reply scored. Scoring is deterministic, no LLM judge.

SystemScore95% intervalMedian time per doc
Gemini 3.8 Flash94.692.9–96.17.2 s
DeepSeek v4.1 Flash93.291.3–94.93.0 s
Mistral Large 491.489.4–93.227.4 s
Refractr (mine)89.487.2–91.40.7 s
GPT-5.4 mini86.283.9–88.52.2 s
Ministral 8B85.583.0–87.73.3 s
Llama 3.1 8B73.970.5–77.20.5 s
NuExtract 2.0 (2B)58.654.7–62.41.4 s
GLiNER2 (base)24.220.8–27.60.5 s

Two things I'd like to hear from people here:

  1. How do you catch wrong extracted values in your own pipeline today? Confidence scores, a second model, sampling by hand?
  2. If you run it on your own stack (parser plus model), I'll add the result. It takes any OpenAI-style endpoint or a Python function, so a full pipeline counts as a system.

Repo: https://github.com/Refractr-io/DocFieldBench

Write-up: https://refractr.io/docfieldbench