r/Rag • u/newzi_la • 23h ago
Discussion My RAG pipeline was working fine until I realised its knowledge was six months old
I have been working on a RAG pipeline that pulls information from documentation sites, product pages and company websites.
The initial setup was pretty straightforward. Scrape the pages, convert everything to Markdown, chunk the content, generate embeddings and push it into the vector database.
The problem showed up later.
A product changed its pricing, documentation got updated and a few pages were removed. But the vector database was still serving the old information.
I initially thought about running a complete re-index every week, but that feels wasteful when only a small percentage of the content actually changes.
I'm now looking at a setup where I can monitor the original pages, identify what changed and only reprocess the relevant content.
I have been looking at Firecrawl, Apify and context.dev for different parts of this workflow.
Context.dev caught my attention because it has scraping and website monitoring under the same API, so I wouldn't need to build another integration just for change detection.
Still figuring out the best architecture here.
For people running production RAG systems, how are you keeping your external knowledge base fresh without rebuilding the entire index every time?