RAG was very popular a few years ago, especially when companies were scrambling to build their own internal Q&A systems. The idea was straightforward: collect all your internal documents, split them into chunks, generate embeddings, put them into a vector database such as ChromaDB, retrieve the relevant chunks, and then let an LLM answer the question.

It made a lot of sense at the time.

But recently, at least from my own team’s usage, I am starting to question whether we still need to go through all that trouble.

Why? Because modern LLMs and AI harnesses such as Claude and Devin have become surprisingly powerful. I can give them PDFs, Markdown files, CSVs, and other documents, and they can read and reason over them directly. They can even query SQL databases and combine the results with information from documents. More importantly, they can often do this quickly enough that I don’t have to sit around waiting for an intolerably long processing time.

This makes me wonder: should RAG eventually fade away?

There is, of course, an obvious counterargument. My team’s current use cases involve a relatively small amount of information. If we suddenly have millions of documents or tens of thousands of PDFs, no matter how powerful the LLM is, it cannot possibly digest all that information every time a user asks a question. At that scale, RAG still makes a lot of sense: retrieve a small number of relevant documents first, then let the LLM focus on those.

But I would challenge even that argument.

Modern AI harnesses are becoming increasingly agentic. Instead of simply passing everything to one LLM, the harness can deploy multiple agents to perform initial searching and retrieval across a huge collection of documents. Those agents can narrow down the information first, identify the relevant sources, and then hand a much smaller and more focused set of information to the main LLM for deeper reasoning.

In other words, perhaps we are moving from:

RAG → retrieve chunks → LLM

to something more like:

Agents → search/retrieve → filter/digest → LLM → answer

The retrieval capability doesn’t necessarily disappear. What may disappear is the need for us to build and maintain the traditional RAG machinery ourselves.

That means fewer embedding pipelines, fewer vector databases, less chunking strategy, less time worrying about embedding models, and less infrastructure to maintain.

So I am starting to wonder whether the future is not “RAG everywhere,” but rather “let increasingly capable AI agents do the retrieval for us.”

For my current use case, that means I am increasingly comfortable saying:

Maybe it’s time to say goodbye to RAG embeddings, vector databases, ChromaDB, and all the plumbing around them.

At least until the scale of the problem forces us to bring them back.