Production RAG — The Real Work

Topics We'll Cover

• Demo to Production — The Real Work
• Eval Pipelines — RAGAS, ARES, Trulens, Golden Sets
• Drift & Freshness — Embedding Migrations, A/B Tests
• Latency Budgets — Caching, Prefetching, Query Rewriting
• Multi-Tenancy — Namespaces, ACLs, Secure Retrieval
• Citation & Grounding — Quoting Sources, Traceability
• Cost — Embedding Refresh, Vector DB Pricing, Reranker Inference
• Failure Modes — Empty Results, Low Confidence, Abstention
• What to Take Away

Demo to Production — The Real Work

A RAG prototype can be built in an afternoon. A production RAG system is a different class of problem: it must be correct, fast, secure, cost-predictable, and observable — at scale, continuously, over a corpus that drifts.

Eval Pipelines — RAGAS, ARES, Trulens, Golden Sets

RAG evaluation has two separate concerns: retrieval quality (did we find the right documents?) and generation quality (given the documents, did the LLM produce a correct, faithful answer?).

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

eval_data = {
    "question":         ["What is HNSW?", ...],
    "contexts":         [["HNSW is a graph-based ANN..."], ...],
    "answer":           ["HNSW stands for Hierarchical...", ...],
    "ground_truth":     ["Hierarchical Navigable Small World...", ...],
}

result = evaluate(
    dataset=Dataset.from_dict(eval_data),
    metrics=[faithfulness, answer_relevancy, context_recall],
)
# result.to_pandas() for detailed per-question breakdown

Drift & Freshness — Embedding Migrations, A/B Tests

RAG systems degrade over time from two distinct sources: corpus drift (the world changes; indexed documents become stale) and model drift (a better embedding model is available but the index still uses the old one).

Latency Budgets — Caching, Prefetching, Query Rewriting

RAG latency is dominated by two serial operations that cannot easily be parallelised: embedding the query and LLM generation. Everything else can be optimised to negligible cost.

Multi-Tenancy — Namespaces, ACLs, Secure Retrieval

Multi-tenant RAG is a security-critical design. A tenant isolation failure allows one customer to retrieve another customer's documents — a data breach.

Citation & Grounding — Quoting Sources, Traceability

Enterprise RAG users require answers to be grounded in sources — both for trust and for auditability in regulated industries (legal, medical, financial). Citation is not a cosmetic feature; it is the mechanism by which users verify correctness.

# System prompt
system = """
You are a precise research assistant. Answer only from the provided context.
For each claim, include an inline citation [1], [2] referencing the source chunk.
If the context does not contain the answer, say "I don't have enough information."
Never invent sources or cite documents not in the context.
"""

# Context block (each retrieved chunk labelled)
context = "\n\n".join([
    f"[{i+1}] (Source: {c['source']}, p.{c['page']})\n{c['content']}"
    for i, c in enumerate(retrieved_chunks)
])

# User message includes context + question
user = f"Context:\n{context}\n\nQuestion: {query}"

Cost — Embedding Refresh, Vector DB Pricing, Reranker Inference

Cost modelling is essential before committing to infrastructure choices. The largest RAG cost drivers are often not the LLM generation but the embedding and vector DB layers.

Failure Modes — Empty Results, Low Confidence, Abstention

A production RAG system must handle retrieval failures gracefully. Silently returning hallucinated answers is worse than explicit abstention.

What to Take Away

• Build a labelled evaluation set first. 200–500 question–answer pairs from real users. Everything else depends on it.
• RAGAS is the standard eval framework. Faithfulness, Answer Relevancy, and Context Recall are the three metrics every production RAG system must track continuously.
• Never mix embedding models in a single index. Use shadow index + alias swap for zero-downtime embedding migrations.
• Multi-tenancy requires either namespace isolation or mandatory payload filters. There is no safe middle ground. Enforce at the retrieval service layer.
• Citation and groundedness are enterprise requirements, not optional polish. RAGAS Faithfulness < 0.7 requires an explicit user warning or abstention.
• LLM generation dominates cost. Contextual compression (extracting only relevant sentences) is the highest-leverage cost optimisation. The embedding and reranker costs are secondary.
• Build abstention into the system. Monitor context relevance; return explicit "I don't know" rather than confident hallucination when retrieval quality is low.
• A/B test every significant change (chunk size, embedding model, reranker) against your labelled eval set before full rollout.