Building RAG with Anthropic

Retrieval-augmented generation isn't dead just because context windows keep growing. Even with a 1M-token window on Claude's current models, most teams still don't want to stuff an entire knowledge base into every prompt — it's expensive, it's slow, and it doesn't scale past a certain corpus size. RAG is still the right answer when your data is large, changes often, or needs to be attributed back to a source. What's changed is how much of the pipeline you actually have to build yourself, and how much Claude can now do for you inside that pipeline.

It's worth being precise about what "RAG with Claude" means, because Anthropic doesn't hand you a vector database. Retrieval is infrastructure you own — an embeddings store, a search index, an internal API. Claude is the reasoning and generation layer that sits on top of it: deciding what to search for, turning retrieved chunks into a grounded answer, and (increasingly) deciding whether it needs to retrieve at all.

If you want someone else to own the retrieval loop too, that's a different conversation — Anthropic's Managed Agents product will run an agent loop and a hosted sandbox for you, but that's a step beyond classic RAG, not a replacement for it. This post is about the architecture you build yourself.

Three architectures, from simplest to most autonomous

Long context, no retrieval at all. If your corpus reliably fits in the context window — a product's full documentation set, a legal contract and its exhibits, a codebase — the simplest thing that could possibly work is to skip retrieval entirely and just pass the whole thing in. Pair this with prompt caching: put the corpus first in the prompt as a cached prefix, and only the user's question varies per request. The first call pays full price; every call after that only pays for the cached-read tokens plus the new question. For a lot of internal tools, this is genuinely simpler and cheaper than standing up a retrieval pipeline, and it sidesteps the hardest RAG problem — bad retrieval — by not retrieving at all.

Classic RAG. This is the shape most people mean by "RAG": embed your documents, store them in a vector database (Pinecone, Weaviate, pgvector, whatever fits your stack), embed the incoming query, pull back the top-k most similar chunks, and hand those chunks to Claude along with the question so it can synthesize an answer. The retrieval step is deterministic and outside the model's control — you decide what "relevant" means before Claude ever sees the question. This is the right default when your corpus is too large or too dynamic for long context, but the query patterns are fairly predictable.

Agentic RAG. Instead of retrieving once and generating once, you give Claude a search (and maybe fetch) tool and let it decide what to look up, look at the results, and decide whether it needs to search again with a refined query. This turns retrieval from a single deterministic step into an iterative loop the model controls. It's more expensive per query and slower, but it handles the cases classic RAG struggles with: multi-hop questions ("what changed between the two most recent versions of X"), ambiguous queries that need reformulation, or questions that only reveal what they actually need after a first look at the data.

None of these are mutually exclusive — a lot of production systems start with classic RAG and add an agentic layer on top for the subset of queries that need it, while keeping long-context-with-caching as the answer for smaller, stable sub-corpora.

Where Claude does more than "generate the answer"

Retrieval-augmented generation has historically split neatly into "retrieval, which is your code" and "generation, which is the model." That line has blurred, and most of the interesting engineering now lives in the pieces between the two:

• Query understanding before retrieval. Raw user queries are often bad search queries. Having Claude rewrite or expand the query first — generating multiple phrasings, or a hypothetical answer to embed and search against (HyDE-style) — measurably improves recall before a single chunk is retrieved.
• Reranking with a cheap model. Vector similarity gets you a candidate set, not a ranked one. Running that candidate set through a fast, inexpensive model (Claude Haiku 4.5, for instance) to rerank or filter before the expensive synthesis call is a cheap way to improve final answer quality without paying full-model prices on every chunk.
• Citations, not hand-rolled prompting. Claude's document content blocks support a citations field you can enable directly — the response comes back split into text segments with citation metadata (which document, which page or character range) attached to each claim. That's a meaningfully better foundation for a "sources" UI than asking the model to append footnotes in free text, and it's far more auditable when someone needs to check an answer against the source.
• Structured output for downstream systems. If the answer needs to feed a UI component, a downstream service, or a database, structured outputs let you constrain the response to a schema instead of parsing free text and hoping the format holds.

Tool use as the retrieval mechanism

In the agentic RAG shape, retrieval is tool use. Each data source your answer might need — a documentation search index, an internal SQL database, a ticketing system, a public web search — becomes a tool definition, and Claude decides which ones to call and in what order. A few things make this work well in practice:

• One tool per source, narrowly scoped, rather than one do-everything "search" tool. Claude reasons better about when to use a tool when the tool's purpose is obvious from its name and description.
• Parallel tool calls let Claude hit multiple sources in a single turn when a question genuinely needs more than one — pull the internal docs and the latest web result at the same time rather than serially.
• Server-side web search and web fetch are worth reaching for instead of hand-rolling your own scraper when part of the corpus is public web content; they run on Anthropic's infrastructure and return results as content blocks directly, no extra execution loop required.

Making it fast and cheap

RAG pipelines have a lot of natural caching and tiering opportunities that are easy to leave on the table:

• Prompt caching on anything static across requests — system prompt, tool definitions, few-shot examples, any reference material that doesn't change per query. This is the highest-leverage, lowest-effort cost lever available, and it's purely additive to whatever architecture you pick.
• Model tiering. Use a fast, cheap model for retrieval, reranking, and query routing, and reserve the larger model for the final synthesis step where quality actually matters to the end user.
• Batch processing for anything offline. Reindexing a corpus, generating summaries for a document store, backfilling embeddings — none of that is latency-sensitive, and running it through the batch API instead of the regular request path cuts cost meaningfully for work nobody is waiting on synchronously.

Where it breaks

Worth being honest about the failure modes, because they're what actually determines whether a RAG system is trustworthy:

• Stale or irrelevant chunks produce confident-sounding wrong answers. Retrieval failing quietly is worse than retrieval failing loudly. Citations help here because they make it possible to spot-check a claim against its source, and explicit "say you don't know if the retrieved context doesn't answer the question" instructions matter more than they seem like they should.
• Agentic RAG can wander. Once the model controls its own retrieval loop, a bad first query can lead it to keep searching in the wrong direction. Cap tool call counts and consider a task budget so the loop paces itself instead of running until it hits a hard token ceiling mid-thought.
• RAG is sometimes the wrong tool entirely. Structured data with a clear schema usually wants a direct database query, not a semantic search over prose descriptions of rows. A small, stable corpus is often better served by long context with caching than by standing up and maintaining a retrieval pipeline for it.

Picking an architecture

As a rough decision framework:

Situation

Reach for

Corpus is small and stable enough to fit comfortably in a context window

Long context + caching

Corpus is large or changes often, but query patterns are predictable

Classic RAG

Questions are open-ended, multi-hop, or need reformulation before they're answerable

Agentic RAG

You'd rather not own the loop, sandbox, or infrastructure at all

A managed agent platform — a different tradeoff than anything covered here, and worth its own post