r/Rag • • Sep 02 '25

Showcase šŸš€ Weekly /RAG Launch Showcase

33 Upvotes

Share anything you launched this week related to RAG—projects, repos, demos, blog posts, or products šŸ‘‡

Big or small, all launches are welcome.


r/Rag • • 10h ago

Tools & Resources In RAG, what goes in determines what comes out. Prajnya: a local ingestion engine for scientific PDFs

7 Upvotes

I'm a researcher with background in Developmental biology, biochemistry and biophysics. I kept finding that retrieval failures started at ingestion: figures dissolved into prose, tables flattened, captions merged with body text, reference lists polluting search. So I built the ingestion layer first. I am now building the RAG part and after having dealt with many issues, I feel a bit confident in sharing this project with you all. Prajnya is open source (Apache-2.0) and fully offline by default.

How it works

  1. A CPU pre-pass classifies each document (paper, book or chapter) and every page.
  2. Plain prose comes straight from the PDF's text layer. Scans and visually complex pages go to a local vision model.
  3. A layout model finds figures and tables, and each is cut out and handled as its own object.
  4. Everything lands in one JSON per document, with a documented schema.

What you get, and why it matters for retrieval

  • Evidence objects, not chunks. Figures are cropped and captioned with stable IDs, and panels are grouped under the label a reader cites ("Fig. 3"). Tables carry a reconstructed grid, so numbers are searchable while the image stays available to a multimodal model. Tables that run onto the next page keep their headers.
  • Provenance. Every item records its page and how it was read (text layer, model read or repaired read), because those differ in reliability.
  • Citations linked conservatively. In-text citations, numbered and author-year, link to the reference list. A wrong link is worse than none, so ambiguous ones stay unlinked.
  • Metadata verified, not believed. The model's reading of the title page, the PDF text and an optional authority (Zotero, or Crossref/OpenAlex sent only the DOI or title) are reconciled, with the source of each value recorded.

What it doesn't do (yet)

  • No chunking, embedding or retrieval. That’s the next layer, Shastrasaara (in development), which builds on these evidence objects.
  • PDFs only. No DOCX, HTML or slides.
  • It isn’t one-click. Local mode needs llama.cpp and the model weights. Cloud models are optional, per role.
  • It doesn’t race to be the fastest. It aims for the most faithful data, and would rather take longer than hand you a wrong table, caption or citation.

How it compares

  • NotebookLM: for public papers and quick cited Q&A, use it. But your PDFs leave your machine, and you get a chat, not structured data you can index your own way.
  • GROBID: the standard for metadata and references, fast on CPU. Its weak spot is pages where figures and tables break the reading order. Prajnya is built around those pages.
  • Docling, Marker and MinerU: solid general converters. Prajnya is narrower and specific to scientific documents, aimed at what a literature RAG needs to cite and trace.

Repo: Prajnya-Academic-PDF-Processor

Question for this sub: how do you index figures and tables in scientific RAG? Captions, images, both, or one-hop expansion from the text that cites them?


r/Rag • • 3h ago

Discussion I priced 100 web search APIs. The cheapest is $0.25 per 1,000 searches and the median is $2.83

1 Upvotes

Picking a search API for an agent turned into opening pricing page after pricing page, so we went through 100 of them and worked out what 1,000 searches cost on each one's entry plan. The spread is silly: the priciest costs 90x the cheapest, and Tavily, Exa and Brave all sit above the $2.83 median.

These are the 10 cheapest paid ones. Three tie at $0.80, so you get 11:

#Provider$ per 1,000PricingReturnsFree tier
1Ceramic$0.25Pay as you goSnippets1,000 queries, no card
2Cloudflare$0.25Pay as you goSnippetsNone listed (resells Ceramic)
3FlyCrawl$0.29$29/moNot stated100 searches/month
4Geonode$0.39$3.90/moSnippets1,500 requests/month
5TickerTrends$0.39$10 packSnippets100 credits, card needed
6Jina$0.50 (est.)$50 packFull page10M tokens, non-commercial
7DeepSeek$0.75Top-upFull pageNone listed
8UnSearch$0.76$19/moNot stated5,000 queries/month, or self-host
9=Dipper$0.80Top-up from $10Full page1,000 requests
9=Searlo$0.80$3.99 packSnippets3,000 credits
9=Serpex$0.80Top-up from $5Full page200 credits, no card

"Full page" means you get the page text back with the results, which saves a separate scrape call for RAG. Jina's price is worked out from its credits because it doesn't quote one per search.

Four more are free outright with rate limits: AgentWeb, LangSearch, ssearch and TinyFish.

Full list of all 100, with free tiers, search modes and a CSV: https://voxoria.ai/web-search-api-pricing/

If any of the info is wrong/out of date, let me know - hopefully this is helpful


r/Rag • • 3h ago

Discussion Has anyone tried generative retrieval instead of vector search for fast URL lookups?

0 Upvotes

Hi guys, I was looking into how to stop web search calls from stalling an agent mid-reasoning and found out about using generative retrieval models as the search index itself.

A normal search pipeline runs query parsing, candidate retrieval, and reranking, which usually adds 100ms or more per call. On the other hand, asking a standard frontier LLM to write URLs from memory fails most of the time because it hallucinates dead links. A setup I have been reading about is one where a 4B model is trained on click data to memorize roughly 200M pages and output a two-part numeric ID mapped to a real URL table, bringing latency down to around 10ms per query in ~10GB of VRAM instead of 400GB+ for exact vector search.

The issue is that memorizing pages works well for popular pages, but long-tail or brand-new pages still need a live index fallback. For anyone building search-heavy agents or type-ahead tools, how are you handling navigational queries compared to deeper web searches?


r/Rag • • 13h ago

Discussion How are you guys debugging RAG when you know the answer exists in the docs?

3 Upvotes

I've been spending a lot of time thinking about this problem lately.

Say you have a RAG pipeline and you know the answer is somewhere in your indexed documents, but the model still gets it wrong.

The usual debugging approach seems to be messing with top-k, similarity thresholds, chunk sizes, or reranking until the answer starts showing up.

But there are a few different ways things can go wrong:

  • The chunk was retrieved but fell outside top-k.
  • It got filtered out by a similarity threshold.
  • The reranker pushed it down.
  • Retrieval worked, but it got dropped before reaching the LLM.

I've been building a tool called ChunkTrace to help pinpoint which stage caused the problem.

I'm curious how others are debugging this right now. Are you logging the intermediate retrieval results, using LangSmith/Langfuse, or just inspecting things manually?

Especially interested in anyone running multi-stage retrieval with pgvector or rerankers.


r/Rag • • 1d ago

Showcase How I fixed RAG hallucinations on 1000-page books: loss-bounded compression + 100% deterministic citations (running on a 4GB GPU)

37 Upvotes

standard RAG for long-form reading is fundamentally broken. if you ask an LLM to summarize a 50-page chapter, it just gives you a "gist" and hallucinates connective tissue. it drops the actual reasoning, skips the caveats, and gives you zero way to verify where the information came from.

i wanted a system where the AI compresses the text, but every single sentence is deterministically mapped back to the exact source paragraph. if the model invents a bridge between two chunks, you can immediately click the sentence, jump to the original text, and verify it.

here is the architecture i built to solve this:

  1. Ingestion & Embedding: Parsed text is chunked and embedded locally using BGE-M3 (1024-dim, FP16 CUDA).
  2. Reranking: BGE Reranker v2 (cross-encoder). Instead of just relying on cosine similarity, we score query-passage relevance directly to pick the best context.
  3. The Two-Representation Invariant: The original document is strictly immutable ("Source is paper"). The compressed reading is a distinct view ("The lens is tinted").
  4. Deterministic Attribution: Every synthesized sentence goes through a provenance resolver. If a sentence outputs zero source citations, it's flagged as a hallucination defect in the UI.

getting this whole pipeline (plus Kokoro TTS) to run entirely offline on my 4GB RTX 3050 was a nightmare. i had to build a strict F65 sequential await barrier to stop Node.js from OOMing at 14.6GB during async ingestion, and an F43 idle lifecycle manager to evict models from VRAM before the GPU choked.

it's open source (AGPL-3.0). you can dig into the retrieval logic or the provenance resolver here:

https://github.com/ayumu-ml/Omnitome

i'd love feedback from other RAG builders. how are you currently handling the "summary loses the source" problem? is anyone actually achieving deterministic citations at scale, or is everyone just relying on semantic similarity?


r/Rag • • 18h ago

Showcase Code retrieval: two Haiku agents with grep, 77% line recall on ContextBench, ~1Ā¢ per issue

4 Upvotes

In our ContextBench run, CodeNib retrieves code context with two Claude Haiku 5.5 agents and grep: no embeddings, no reranker, about a cent per issue.

ContextBench (500 tasks, 8 languages) scores how much of the human-annotated relevant code a system retrieves. This setup finds 77% of the relevant lines; the next-best system on the leaderboard finds 61%.

How the two agents split the work:

  • The first follows the issue with glob, grep and file reads.
  • The second gets the issue plus a map of where the first already looked (paths and line ranges, at most 2,048 tokens), and goes after what was missed: callers, alternate paths, tests, configuration.

The search reads broadly to get that recall: its line precision is 20.5%. With an Opus repair agent on top, the full system resolves 388 of the 500 tasks, #1 among agent systems on the leaderboard as of Oct 23. The task annotations were visible during development.

Write-up, trajectories and cost receipts: https://codenib.ai/blogs/contextbench-500-tasks/


r/Rag • • 11h ago

Discussion Web pages as RAG sources: how do you keep the index fresh when the pages keep changing?

1 Upvotes

Documents in a drive change now and then. Web pages change constantly: prices, specs, policies, pages that disappear. A RAG index built from a crawl starts going stale the day after.

For anyone with web sources in production:

- full re-crawl on a schedule, or change detection per page?

- how do you handle pages that 404 or redirect: delete the chunks or keep them?

- do you store the crawl date in chunk metadata and use it at retrieval?


r/Rag • • 11h ago

Discussion RAG over a product catalog: how do you stop it quoting yesterday's prices?

1 Upvotes

Shop assistants and product Q&A bots are a common RAG use case, but prices and stock change daily, sometimes hourly. If the price sits inside an embedded chunk, the bot can confidently quote an old one.

How are people handling the volatile fields:

- keep price and stock out of the chunks and fetch them at answer time?

- re-embed changed products nightly?

- something else, like a tool call to the store API?


r/Rag • • 15h ago

Discussion RAG over a product catalog: how do you stop it quoting yesterday's prices?

2 Upvotes

Shop assistants and product Q&A bots are a common RAG use case, but prices and stock change daily, sometimes hourly. If the price sits inside an embedded chunk, the bot can confidently quote an old one.

How are people handling the volatile fields:

- keep price and stock out of the chunks and fetch them at answer time?

- re-embed changed products nightly?

- something else, like a tool call to the store API?


r/Rag • • 14h ago

Tools & Resources If your security team blocks hosted embedding APIs, the air-gapped self-host setup is one config flag, not the research project I expected

1 Upvotes

Our blocker was never model quality, it was legal. Our document text is not allowed to leave the VPC, and that ruled out every hosted API before we even got to compare them.

The part I expected to be a nightmare was proving the server never quietly reaches out for weights on its own. On SIE it came down to a single setting:

yaml

SIE_HF_FALLBACK: ""false"" # never reach out to HuggingFace

hfToken.create: false

You mirror the weights into your own storage first, then pin the image and install, and after that the worker pods have no path to the internet at all.

The catch is that this is real infrastructure you now own and babysit. If your data is not actually sensitive, do not put yourself through it, just use an API.


r/Rag • • 15h ago

Discussion At what point does self-hosting your embedding model actually get cheaper than the API? Did the math

1 Upvotes

Every time someone asks about embedding costs here, the answer is ā€œjust self-hostā€ and nobody says the one number that decides it.

An API bills per token, so idle costs nothing.
A GPU bills per hour whether it’s busy or not.

Self-hosting only wins once the GPU stays busy enough. Rough math for a small embedding model:

API is ~$0.02/M tokens, so 1B tokens ā‰ˆ $20/day
A10G is ~$1/hr, so ~$24/day before overhead

Crossover is ~1.2B tokens/day.

Under that, the API wins. The RAG-specific bit is that reindexing is what gets you there. Queries alone almost never hit 1B/day, but re-embedding the whole corpus every time you change chunking or swap models does. If you reindex often, the line comes way closer.

Catch is it assumes the card is pushing ~14k tokens/sec all day. Spiky traffic means you’re paying for peak. And nobody puts ops in the spreadsheet. Someone’s awake at 2am when it falls over. What volume are people here actually at? Curious if anyone’s crossed it on queries alone.


r/Rag • • 1d ago

Showcase We open-sourced a database that handles the RAG pipeline for you (Apache 2.0)

5 Upvotes

We just released Antfly v0.3 under Apache 2.0.

We built it to make RAG over complex files easier e.g. PDFs, images, audio, and video. Instead of wiring together OCR, embeddings, vector search, reranking, and graph retrieval, Antfly handles it inside one open-source database.

It supports keyword/grep, hybrid, and graph search, and runs embedded (Lite), self-hosted, or in the cloud.

GitHub: https://github.com/antflydb/antfly

Would love feedback from anyone building RAG pipelines, especially over multimodal data.


r/Rag • • 1d ago

Showcase NornicDB - v1.5.0 - Feedback

3 Upvotes

Okay this has been like an entire month in the making so i’m super excited about it

https://github.com/orneryd/NornicDB/releases/tag/v1.5.0

new docs portal, new UI and dashboards, new ai assistant, faster queries, lower memory usage, and most importantly extremely hardened crash recovery and query correctness. there’s still some other compatibility things we are going to be adding, i also have a fork of badger i am experimenting with projected reads for arbitrary maps so that we never serialize more from disk than is needed, a whole new storage format i’m experimenting with where preliminary measurements have storage use down ~40-50% depending on workload it’s a hybrid compressed columnar storage format i’m seeing if i can show an improvement across the board

https://github.com/orneryd/NornicDB/discussions/1728

new docs site
https://orneryd.github.io/NornicDB/

i appreciate all the community support over the last year. Black Friday weekend last year was the day i started it and it’s come soooo much further than i ever expected.

MIT licensed, 931 stars and counting.

enjoy!


r/Rag • • 1d ago

Discussion Lessons from running a RAG support bot for real Shopify stores: the hard part wasn't retrieval, it was knowing when to say "I don't know"

2 Upvotes

I've been building a customer-support chatbot for Shopify stores (order status, product questions, store policies) using RAG. It's now in a pilot with real merchants. A few lessons that weren't obvious to me at the start:
1. A confident wrong answer is worse than no answer.
In e-commerce, a bot that invents a return policy or a delivery date creates real problems for the merchant. So the core design question became "when should the bot refuse?", not "how do I get better answers?"
2. Retrieval score floors did more than prompt engineering.
I spent a lot of time telling the model "only answer from the context." What actually reduced hallucinations was a minimum similarity threshold: if nothing retrieved clears the floor, the bot doesn't get to answer from vibes. It says it doesn't know and offers a human handoff.
3. Build a refusal test set, not just an accuracy test set.
I keep a set of questions the bot should not be able to answer from the store's data (other stores' policies, made-up products, general-knowledge bait). Every change to chunking, prompts or models gets run against it. This caught regressions that normal Q&A tests missed.
4. Model choice mattered more than I expected for grounding.
On a smaller, cheaper model I kept seeing fabricated details even with good context. Moving to a newer model eliminated most of those failures, and the cost difference was worth it for support use cases.
5. Per-shop isolation has to be enforced in retrieval, not just in the prompt.
With multiple merchants on one system, the filter must sit at the retrieval layer. You can't rely on the model to ignore another store's chunks.
6. Show the merchant why the bot answered.
The admin can see each conversation and which document an answer came from. When the bot gets something wrong, the merchant can see whether it's a missing document or a retrieval problem, and fix it. That built more trust than any accuracy number.
7. Human takeover is a feature, not a fallback.
Merchants wanted to jump into a live conversation at any point. Designing for that from the start made "I don't know" answers acceptable instead of embarrassing.
Curious how others handle this:
How do you pick your score threshold? Fixed value, per-tenant, or calibrated on a test set?
Anyone using rerankers specifically to improve refusal behaviour?
(Disclosure: this is my own product, but not linking it here. Happy to go deeper on any of these.)


r/Rag • • 1d ago

Discussion Should a weight-only edge change invalidate a GraphRAG community summary?

2 Upvotes

I'm working on incremental updates in a TypeScript GraphRAG engine. One awkward case is when two documents support the same graph edge.

The implementation stores contributions per document. Updating a document replaces its contributions; deleting it removes them. The shared edge is then rebuilt from the remaining weights and unique relation texts. If no contributions remain, the edge is removed.

The less obvious decision is when to regenerate the community summary. Its fingerprint includes members, entity descriptions, claim text, and edge relation text—but not edge weight. The summary prompt doesn't include weight either.

Consider two documents contributing identical relation text. Removing one could change only the edge's weight. If the other fingerprint inputs stay unchanged, the existing summary can be reused. Different relation text would change the fingerprint instead.

This is an illustrative case, not a measured result. There are database-mocked tests for document-scoped deletion and claim-text changes, but they don't establish shared-edge correctness after deletion or equivalence with a fresh Postgres build.

The trade-off seems to be between avoiding unnecessary summarisation and preserving changes in how much source material supports a relation. A weight isn't necessarily an independent evidence count, either.

Would you include weight in the summary fingerprint, or keep support metadata separate from the text summary? What would make a weight-only change significant enough to regenerate it?


r/Rag • • 1d ago

Tutorial I built a fully local RAG system in ~55 lines of Python: no LangChain, no vector DB, just Ollama + NumPy

26 Upvotes

I wanted to understand RAG without a framework hiding the important parts, so I built the whole pipeline from scratch and recorded a tutorial walking through it.

**Stack**

- Ollama running two local models: `nomic-embed-text` (embeddings) and `qwen3:4b-instruct` (chat)

- Python with just `requests` + `numpy`, managed with `uv`

- No LangChain, no LlamaIndex, no vector database, no API key

**How it works**

  1. Load a markdown doc and split it on blank lines (11 chunks)
  2. Embed every chunk via `/api/embed` → an (11, 768) NumPy array
  3. Embed the question and rank chunks by cosine similarity
  4. Put the top 3 chunks into the prompt and send it to `/api/chat`

The whole retrieval step is basically this:

def retrieve(question, k=3):

query = embed([question])[0]

denominator = np.linalg.norm(vectors, axis=1) * np.linalg.norm(query)

scores = (vectors @ query) / np.maximum(denominator, 1e-12)

indices = np.argsort(scores)[::-1][:k]

return [chunks[i] for i in indices]

**Results on a test doc** (a fictional coffee shop handbook):

- "Are dogs allowed inside?" → Pet Policy chunk scored 0.784 (next best 0.493), correct answer

- "What time does the shop open?" → correct

- "How should employees clean the espresso machine?" → correct but missed one step

- "What is the owner's favorite movie?" (not in the doc) → "The answer is unavailable." šŸ‘

It's intentionally minimal. Things it doesn't do yet: persistent vector storage, smarter chunking, reranking, evaluation. Those are covered at the end as next steps.

Full disclosure, it's my video: https://youtu.be/x-sn2WAsX8I

Happy to answer questions. I'd also love to hear what you'd add first for a real project.


r/Rag • • 1d ago

Showcase I built an open-source AI engineering learning platform with a RAG tutor — looking for feedback

0 Upvotes

I've been working on Cairnly, a free and open-source AI engineering learning platform.

The idea was to create something useful for people learning AI engineering while also making the complete implementation available for developers to explore.

For learners:

  • 126 lessons across 6 tracks
  • 7 learning roadmaps
  • AI tutor with citations linking directly to lesson sections
  • Quizzes and progress tracking

For developers:

  • Hybrid RAG using PostgreSQL, pgvector, and keyword search
  • Cross-encoder reranking
  • Local LLM inference with Ollama
  • RAG evaluations and confidence calibration
  • Langfuse tracing, safety checks, and deployment examples

The retrieval system currently reports Recall@5 of 0.971 on our 627-question evaluation set. I've also documented the cases where the confidence gate and safety checks fail.

The project is still evolving, and I'd appreciate feedback on the learning experience, architecture, and areas worth improving.

GitHub: https://github.com/CherukuriSiva/Cairnly

Live demo: https://cairnly-blond.vercel.app

What would you improve or add?


r/Rag • • 1d ago

Discussion Looking for a fast, open-source LLM under 5B for Markdown-to-JSON extraction

3 Upvotes

Hey everyone,

I'm working on a document processing pipeline where most of the input consists of scanned PDFs, forms, and tables.

My current setup looks like this:

  • OCR: PaddleOCR-VL-1.6 served through vLLM. I'm using it to convert scanned pages into Markdown.
  • JSON extraction: oroboros-labs/claude-fable5:latest running through Ollama. I pass the Markdown of each page to the model, and it converts it into JSON.

The important part is that I already have fixed JSON schema templates that the output needs to follow. So, the model needs to map the extracted Markdown content into the predefined JSON structure.

The problem I'm facing right now isn't accuracy. It's inference time. The current model is around 9B parameters, and processing pages one by one through Ollama is taking longer than I'd like.

I'm looking for an open-source model under 5B parameters that can handle this kind of structured information extraction reasonably well, ideally with faster inference and reliable JSON output.

Has anyone worked on a similar pipeline?

I'd particularly like to hear about models you've actually tested in real-world document extraction rather than just benchmark results.

Any suggestions or personal experiences would be really helpful. Thanks!


r/Rag • • 1d ago

Tools & Resources I built BulkParse, an open-source Rust document parser for RAG pipelines

1 Upvotes

Hey everyone!

I’ve been working on BulkParse, an open-source document parser written in Rust, mainly to make document ingestion faster and simpler for RAG pipelines.

It converts PDFs, Word documents, Excel spreadsheets, PowerPoint files, and other formats into Markdown or structured JSON.

Some of the features:

• 19 file formats supported, including PDF, DOCX, XLSX, PPTX, EPUB, and more

• Native Rust engine for fast document processing
Structure preservation for headings, tables, lists, and document blocks

• Built-in OCR support for scanned documents, with locally provisioned backends

• Fully local processing, without requiring an external parsing API

• Python, Node.js, Rust, and WebAssembly integrations

• MIT licensed

I also ran some benchmarks against tools like Docling, MarkItDown, Unstructured, and others.
In one benchmark of 200 PDFs (OCR disabled), BulkParse processed the documents in 0.44 seconds, compared with 0.73 seconds for the next-fastest tested configuration. The benchmark methodology and raw results are available in the repo.

It’s still an early release, and I’m actively improving extraction quality and format coverage.

I’d really appreciate feedback from people building RAG pipelines:

• What parser are you currently using?

• How important is parsing speed compared to extraction quality?

• Are there specific document formats or parsing problems you’d like to see handled better?

GitHub: https://github.com/bulkgrid/bulkparse

Would love to hear your thoughts and suggestions!


r/Rag • • 1d ago

Discussion What experiments changed the way you understand RAG?

1 Upvotes

In the last two months, I have been working on a hobby project to get the hang of RAG. I've never had any experience writing Python since I myself come from a full-stack (JS/TS) background. As this sub suggested, I wrote code manually to implement chunking, embedding, ingestion, cross-encoder reranking, vector and hybrid retrieval with RRF. The alteration of the pipeline always happened after measurement. Since it was a text-based corpus, I scrutinized hit@5, recall@5 and MRR for every alteration.

I have understood the importance of retrieval gaps, citation verification and retrieval-generation consistency. Most of the setbacks I encountered while building came from how the system was architected rather than the way it was architected. The system caused so many errors (manual errors from my end), while the pipeline gap closed over time through various iterations.

After the generation step, I wrote an eval with golden evidence. It helped me identify the gaps and close them. This is how I introduced cross-encoder reranking into the pipeline.

I kept the above project's outcome as a frozen benchmark to write a verification layer. So, I prepared golden claims and ensured they were self-contained, then performed claim extraction and claim-level verification. With the help of cross-family LLM judgment, I captured various oracle vs retrieved evidence evaluations.

Since I have navigated all these decisions via Claude, I am quite isolated from what is happening in the real world.

I want to know what other aspects could meaningfully reinforce my understanding of RAG. I would like to focus on unanswered questions that I could explore to understand the deeper concepts of RAG.

What experiments changed the way you understood RAG better?

I'd particularly appreciate feedback on experiments that address meaningful problems in RAG systems rather than recommendations to explore more frameworks or tools.

Github - https://github.com/venkatsundaraj/rag-for-medical-records


r/Rag • • 1d ago

Discussion Every chunk retrieves fine and the answer still falls apart

1 Upvotes

cConversation memory has a failure that document RAG rarely shows. A name gets mentioned in one session. The constraint that actually matters shows up weeks later. The relation between them lands in a third.

Each fragment retrieves perfectly on its own. Ask for any single piece and the right chunk comes back. Ask the question that needs all three and the answer still fails, because nothing ever puts them in the same context.

Most document retrieval checks are single-hop. One good chunk settles the question, so the retriever looks excellent.

My honest opinion: single-hop lookups flatter almost any retriever, and joining fragments said weeks apart is where conversation memory actually gets hard.

The Agent Memory Leaderboard (AML) scores Compositional Inference on its own in its Textual track. That capability covers cross-fragment evidence and relation chains. It sits next to Explicit Fact Recall, which covers facts, attributes, sources, and entities.

The setup is strict about the boundary too. Entries are limited to Add for writes and Search, which returns evidence rather than an answer. The platform handles answering and grading identically for all entries, so score gaps between entries point mainly at the memory side.

For scale, the Cycle 2 Textual track draws on 32 data sources, with more than 6,000 evaluation instances.

With Cycle 2 results expected on Thursday, November 19, this is a good week to compare notes on how people actually join fragments.

In your pipeline, what gets both halves of a split answer into one context: links between memories, query decomposition, a bigger Top K, or something else?


r/Rag • • 1d ago

Discussion Vendors ranked their own document parser #1 on 29 of the 33 parsing benchmarks they published

7 Upvotes

Anyone who has shopped for a PDF parser or document extraction tool for a RAG pipeline has probably noticed the same thing: nearly every vendor publishes its own benchmark, and the vendor's own parser almost always comes out on top.

To see just how consistent that pattern actually is, I tracked down every vendor-published parsing and extraction benchmark I could find where a commercial vendor includes its own product on the leaderboard.

The results across 33 benchmarks (archived 15 to 16 November 2026):

  • 29 of 33: The vendor that published the benchmark ranks #1 on the default or headline metric.
  • 2 more: The vendor trails only a much pricier frontier model or agent (Gemini 4 on OmniParseBench; Codex GPT-6 Astra on ExtractBench Pro), while framing its own product as the sensible, cost-effective choice for production.
  • 2 exceptions: Baidu's Real5-OmniDocBench (where two small open OCR models edge out PaddleOCR-VL) and Parsebridge's PDF Parser Benchmark (where competing tools beat Docling on both speed and table accuracy).

If you group by company rather than by benchmark, 25 out of 27 vendors rank #1 on at least one of their own leaderboards.

I don't think it means vendors are faking numbers. Basically, no company has an incentive to spend time and money publishing a benchmark it loses, and vendors naturally build evaluations around the document types and failure modes their parsers were engineered to solve. Still, it shows why you can't rely on a vendor's own leaderboard to pick a parser. You have to run candidates on a sample of your own documents.

All 33 benchmarks (rankings reflect the headline or default metric used to sort the public leaderboard; each benchmark links to its Wayback Machine snapshot from the capture date):

#Benchmark (Wayback)PublisherPublisher's rankRanked above publisher
1ParseBench 2LlamaIndex#1None
2ExtractBenchLlamaIndex#1None
3Real5-OmniDocBenchBaidu / PaddlePaddle#3MiniCPM-OCR-1.0, OvisOCR3
4PulseBench-TabPulse#1None
5OmniParseBenchDatalab#3 (#1 on the benchmark page)Gemini 4 Argon, Gemini 4 Carbon
6RealDoc-BenchExtend#1None
7ExtractBench ProLlamaIndex#2Codex GPT-6 Astra (xhigh)
8WholeDoc BenchNutrient#1None
9OmniExtractBenchDatalab#1None
10DocuBenchDocuPipe#1None
11SCORE-BenchUnstructured#1 (Adjusted CCT)None
12OmniAI OCR benchmarkOmniAI#1None
13DocParsingBenchSoMark#1None
14IDP LeaderboardNanonets#1None
15BizDocBenchDocSlicer#1None
16LongArray-ExtractExtend#1None
17OpenDataLoader BenchHancom-associated#1None
18DP-BenchUpstage#1 (all 3 accuracy columns)None
19RD-TableBenchReducto#1None
20DeepRead extraction compositeDeepRead#1None
21LongExtractionBenchReducto (commissioned)#1 (both headline metrics)None
22ChartParse-BenchUnsiloed#1 (both headline tests)None
23OCRFlux-benchChatDOC#1None
24Parse IndexNace.AI#1None
25PDF Parser Benchmark 2026ParsebridgeNot #1 on any tableMinerU, Marker, MarkItDown, others
26Handwriting OCR Benchmark 2026Parseur#1None
27ParseArena / PAIRReducto#1None
28Five-hard-document Open BenchmarkAirparser#1 (sister product #2)None
29120-document OCR benchmarkDocsumo#1 (both headline metrics)None
30NuMind internal evaluationsNuMind#1None
31Legal document parsing benchmarkLexSelect#1 (both headline metrics)None
32Mistral OCR internal evaluationsMistral#1 (both owned tests)None
33Enterprise Document BenchmarkTensorlake#1None

I almost certainly missed some benchmarks. If you know of another vendor leaderboard or spot an error in the rankings above, let me know so I can add or fix it.

OneĀ importantĀ distinction:Ā aĀ vendorĀ rankingĀ someoneĀ elseĀ atĀ #1Ā isĀ goodĀ evidenceĀ againstĀ self-servingĀ publicationĀ bias,Ā butĀ itĀ isĀ notĀ aĀ measureĀ ofĀ benchmarkĀ quality.Ā AĀ tinyĀ five-PDFĀ testĀ whereĀ theĀ publisherĀ losesĀ isĀ notĀ automaticallyĀ moreĀ rigorousĀ orĀ usefulĀ thanĀ aĀ 2,000-pageĀ openĀ benchmarkĀ withĀ publicĀ datasetsĀ andĀ scoringĀ codeĀ whereĀ theĀ publisherĀ wins.

The full post includes original leaderboard screenshots, live links, the methodology for what counts as an exception, and a side-by-side matrix of how the six most frequently tested vendors rank on each other's benchmarks:

https://www.hashiromer.com/posts/vendor-benchmarks/


r/Rag • • 1d ago

Discussion Need Suggestions for Rag Pipeline Improvement

3 Upvotes

Hi everyone! I just got appointed to a project that was like built a way back and now they want me to revamp the project especially the rag pipeline part of it as they wanna onboard a client that wants a really accurate and grounded answers whereas their current pipeline gives around 35-50% score. The current setup looks like this according to my agent:

What it does

  • You upload your documents, then ask questions about them by typing or speaking. The answer is written or spoken from what’s actually in your files.
  • Files are private by default, so only your own answers use them. An admin can share a document with everyone.

When you upload a file

  1. The original file goes into object storage (Cloudflare R2).
  2. The text is pulled out of the file. It handles PDF, Word, Excel, CSV, JSON and plain text.
  3. The text is cut into pieces by LangChain’s RecursiveCharacterTextSplitter of about 1,000 characters, each overlapping the next by 150 so no sentence gets lost at a cut.
  4. Each piece is turned into an embedding (a list of 1,536 numbers that captures its meaning) using OpenAI’s text-embedding-3-small.
  5. Each piece is saved as one row in Postgres with pgvector, along with its embedding, who owns it and whether it’s private or shared.

When you ask a question

  1. A chat assistant decides whether the question needs your documents. If it does, it calls the search step as a tool.
  2. The question is turned into an embedding with the same model.
  3. The database finds the 5 pieces closest in meaning, counting only the ones you’re allowed to see.
  4. Those 5 pieces and your question go to an LLM (GPT-4.1 for typed chat, GPT-4.1-mini for voice), which writes the answer.
  5. The answer streams back to the chat, or is read out loud for voice.

Libraries:

  • LangChain’s RecursiveCharacterTextSplitter for cutting text into pieces.
  • pypdf and python-docx for reading files.
  • Prisma for database queries.

I want suggestions on what to improve and which new tech to follow so it can at least give a better grounded answers with better accuracy like 80-90% range. As the project owner wants something magically 2027ish lol! Good suggestions are appreciated :)


r/Rag • • 2d ago

Tools & Resources A Hands-on Guide to Building Agentic RAG with LangGraph

12 Upvotes

An open-source project for learning, building, and experimenting with Agentic RAG systems using LangGraph.

The repository includes interactive notebooks and a modular implementation covering the main components of an agentic retrieval pipeline.

šŸš€ Key Features

  • 🧠 Agentic RAG: Dynamic retrieval and reasoning workflows powered by LangGraph.
  • šŸ” Hierarchical Retrieval: Multi-level document retrieval for better context selection.
  • šŸ¤– Multi-Agent Workflows: Specialized agents collaborating to handle complex queries.
  • šŸ’¬ Conversation Memory: Context-aware interactions across multiple turns.
  • ā“ Query Clarification: Handling ambiguous or incomplete user queries.
  • šŸ”„ Self-Correction: Iterative refinement of retrieval and generated responses.
  • šŸ“Š RAGAS Evaluation: Measuring retrieval quality and answer generation performance.
  • šŸ¦™ Local LLM Support: Running models locally through Ollama.
  • ā˜ļø Cloud LLM Support: Integration with cloud-based language models.
  • šŸ““ Interactive Notebooks: Step-by-step examples for exploring the architecture and its components.

šŸ”— Repository

https://github.com/GiovanniPasq/agentic-rag-for-dummies

A practical resource for exploring Agentic RAG architectures, from individual components to complete workflows.