From Dev.to
I crawled 150 pages from 6 popular docs sites into RAG chunks: one page often eats half the token budget
If you build a RAG bot on top of someone's documentation, the first crawl decides what your retriever will see. I wanted to know what a naive crawl of popular docs sites actually returns, so on 3 Nove...