Agentic RAG Patterns — From Single-Shot to Multi-Step

Topics We'll Cover

• From Single-Shot to Multi-Step Retrieval
• HyDE — Hypothetical Document Embeddings
• Query Decomposition for Compound Questions
• Self-RAG — Training the Model to Decide When to Retrieve
• Corrective RAG (CRAG) — Quality Classifier + Fallback
• Iterative Retrieval Loops
• Query Routing — Which Index for Which Question
• Tool-Using RAG — Retrieval + Code + Web
• What to Take Away

From Single-Shot to Multi-Step Retrieval

Naive RAG is a single-pass pipeline: embed query → retrieve top-k → stuff into context → generate. This fails for compound questions (multiple sub-questions needing separate retrievals), ambiguous queries (query expansion helps), and unreliable corpora (retrieved docs may be irrelevant or outdated).

HyDE — Hypothetical Document Embeddings

HyDE (Gao et al. 2022) addresses the query–document embedding gap: a short question and a long passage have very different embedding distributions even when semantically aligned. The fix: use an LLM to generate a hypothetical answer document, then embed that for retrieval.

from openai import OpenAI
from sentence_transformers import SentenceTransformer

llm = OpenAI()
embedder = SentenceTransformer("BAAI/bge-large-en-v1.5")

def hyde_retrieve(query: str, k: int = 10) -> list:
    # 1. Generate hypothetical answer
    hyp = llm.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user",
                   "content": f"Write a 100-word technical answer to: {query}"}],
        max_tokens=150,
    ).choices[0].message.content

    # 2. Embed the hypothetical answer (not the query!)
    vec = embedder.encode(hyp, normalize_embeddings=True)

    # 3. Retrieve against corpus
    return vector_db.search(vec, limit=k)

Query Decomposition for Compound Questions

Compound questions contain multiple independent sub-questions that each require separate retrieval. Embedding the full compound question produces an averaged vector that may not retrieve well for any single sub-question.

from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
retriever = MultiQueryRetriever.from_llm(
    retriever=vector_store.as_retriever(search_kwargs={"k": 5}),
    llm=llm,
    # Generates 3 alternative phrasings + the original
)

# Retrieves for all generated queries, deduplicates, returns union
docs = retriever.invoke("What are the limitations of RLHF for alignment?")

Self-RAG — Training the Model to Decide When to Retrieve

Self-RAG (Asai et al. 2023) fine-tunes a model to emit special reflection tokens that control retrieval dynamically. Unlike pipeline-controlled retrieval, the model itself decides whether retrieval is needed, whether retrieved docs are relevant, and whether its own output is grounded.

Corrective RAG (CRAG) — Quality Classifier + Fallback

CRAG (Yan et al. 2024) adds a retrieval-quality classifier between the retrieval stage and the generation stage. If retrieved documents are classified as low-quality or irrelevant, the system falls back to a web-search augmented retrieval pass.

Iterative Retrieval Loops

Some questions cannot be answered from a single retrieval pass because the answer requires synthesising information across multiple documents, or because the initial query is underspecified and needs refinement based on what was found.

Query Routing — Which Index for Which Question

A production RAG system may have multiple indexes: a dense vector index for prose, a BM25 index for code, a structured SQL database for factual lookups, and a web search tool for current events. Query routing selects the best retrieval path without trying all of them on every query.

from llama_index.core.selectors import LLMSingleSelector
from llama_index.core.tools import ToolMetadata

selector = LLMSingleSelector.from_defaults()

choices = [
    ToolMetadata(description="technical documentation", name="docs"),
    ToolMetadata(description="source code repository",     name="code"),
    ToolMetadata(description="current news and web",         name="web"),
]

result = selector.select(choices, query="How do I fix NullPointerException in Java 21 records?")
# Returns: SelectionOutput(index=1, reason="code-related query")

Tool-Using RAG — Retrieval + Code + Web

The most capable production RAG systems combine vector retrieval with other tools: code execution for numerical reasoning, web search for current events, SQL for structured data, and external APIs for live lookups. The model orchestrates which tools to call and in what order.

What to Take Away

• HyDE closes the query–document gap by 5–15 NDCG points for short queries. Run it in parallel with direct-query retrieval and fuse with RRF to hedge against LLM hallucination in the hypothetical.
• Query decomposition is essential for compound questions. Parallel for multi-facet queries; sequential for multi-hop questions where each answer feeds the next query.
• Self-RAG requires a fine-tuned model but lets the model control retrieval dynamically. For proprietary LLMs, approximate with a separate relevance-grading step (CRAG).
• CRAG prevents hallucination on stale-corpus queries by grading retrieved docs and falling back to web search when all docs are irrelevant.
• Iterative retrieval: cap at 2–3 iterations; always check for query redundancy before another retrieval pass.
• Query routing reduces latency and cost by directing simple queries to fast paths and complex queries to expensive ones. An embedding classifier is faster than an LLM router for high-traffic applications.
• Tool-using RAG (ReAct + retrieval + code + web) is the current state of the art for general-purpose QA agents.