Production RAG Systems

RAG (Retrieval-Augmented Generation) is an architecture where an LLM's response is grounded in externally retrieved documents rather than relying solely on its parametric knowledge. The pipeline works in three steps: (1) convert the user's query into a vector embedding, (2) retrieve relevant documents from a vector database using similarity search, (3) pass the retrieved documents as context to the LLM for answer generation.

RAG is preferred over fine-tuning for knowledge-intensive applications for several reasons: Currency — you can update the knowledge base in real-time without retraining; Cost — fine-tuning large models is expensive and must be repeated for each data update; Attribution — RAG provides source citations since answers come from retrievable documents, while fine-tuned models bake knowledge into weights with no traceability; Multi-tenancy — in a SaaS context, you'd need a separate fine-tuned model per tenant, which is impractical; Reduced hallucination — grounding in retrieved text constrains the model's output.

Fine-tuning is still valuable for teaching the model a particular style, format, or behavior (e.g., "always respond in formal tone"), but not for injecting factual knowledge.

Vector embeddings are dense, fixed-size arrays of floating-point numbers that represent the semantic meaning of text in a high-dimensional space. An embedding model (a neural network, often a transformer) takes text as input and outputs a vector — typically 256 to 3072 dimensions.

The key property: texts with similar meanings get mapped to nearby points in this vector space. "How do I reset my password?" and "I forgot my login credentials" will have similar embeddings (high cosine similarity), while "What's the weather in Tokyo?" will be far away.

This works because the embedding model is trained on enormous text corpora with objectives that push semantically similar texts closer together. For example, contrastive learning trains the model to maximize similarity between a query and its correct document while minimizing similarity with random documents. Through this training, each dimension learns to capture some abstract aspect of meaning — no single dimension is independently interpretable, but collectively they encode semantic content.

Cosine similarity measures the angle between two vectors, ignoring their magnitude. This is preferred because the direction in embedding space encodes semantic meaning, while the length can vary based on text length or model artifacts. Two vectors pointing in nearly the same direction (cosine ≈ 1) are semantically similar regardless of their magnitudes.

Euclidean distance considers both direction and magnitude, which can introduce noise. A short text and a long text about the same topic might have different vector magnitudes, making their Euclidean distance large even though their semantic meaning is similar.

That said, many modern embedding models produce normalized vectors (unit length), in which case cosine similarity equals dot product, and Euclidean distance is a monotonic function of cosine similarity — so all three metrics give the same ranking. In practice, dot product (inner product) is often used for speed since it's computationally cheaper than cosine similarity (no normalization step), and most vector databases use it internally.

Chunking is the process of breaking documents into smaller pieces before embedding and indexing. It matters because embedding models have token limits (typically 512-8192 tokens), and because the quality of retrieval depends heavily on chunk granularity.

The fundamental tension: too-large chunks contain noise that dilutes the embedding signal (the embedding becomes a blurry average of many topics), while too-small chunks lose the context needed to understand what the text is about. A sentence like "The rate is 3.5%" means nothing without knowing what rate — but in a 5000-word document about mortgage terms, it's buried among unrelated paragraphs.

Common strategies include: fixed-size chunking (simple but breaks mid-sentence), recursive character splitting (respects paragraph/sentence boundaries), semantic chunking (uses embedding similarity to find natural topic breaks), document-structure-aware chunking (splits on headers), and parent-child chunking (small chunks for precise retrieval, larger parent chunks for context in the LLM prompt). The optimal approach depends on document type and query patterns.

A vector database is purpose-built for storing and searching high-dimensional vectors using approximate nearest neighbor (ANN) algorithms. Traditional databases (PostgreSQL, MySQL) index data for exact lookups — find rows matching a specific condition. Vector databases solve a fundamentally different problem: find the k most similar vectors to a query vector, across millions of vectors, in milliseconds.

The key difference is the indexing. Traditional B-tree or hash indices don't work for high-dimensional similarity search. Vector databases use specialized structures like HNSW (hierarchical navigable small world graphs), IVF (inverted file indices), and PQ (product quantization) that trade exact accuracy for dramatic speed improvements — from O(n) brute force to O(log n) approximate search.

Modern vector databases also support: metadata filtering (combine vector search with structured filters like tenant_id or date), namespaces/multi-tenancy, CRUD operations on vectors, and hybrid search (vector + keyword). Examples include Pinecone, Weaviate, Qdrant, Milvus, and ChromaDB. PostgreSQL's pgvector extension brings basic vector search to traditional databases.

HNSW (Hierarchical Navigable Small World) is the most popular ANN algorithm, used by Qdrant, Weaviate, ChromaDB, and pgvector. It builds a multi-layer graph where each layer has progressively fewer nodes.

Analogy: finding a house. The top layer is a map of countries — you quickly navigate to the right country. The next layer is a map of cities within that country. Then neighborhoods. Then streets. Finally, individual houses. At each layer, you do a greedy walk toward the query vector, then descend to the next layer with more detail.

Technically: Layer 0 contains all vectors, each connected to M nearest neighbors. Higher layers contain a random subset of vectors (each vector has a probability of being in the next layer). Search starts at the top layer's entry point, greedily moves to the closest node, then descends. Key parameters: M (connections per node — more = better recall but more memory), efConstruction (build-time quality), efSearch (query-time accuracy vs speed trade-off).

Why popular? It offers excellent recall (95-99%+) at very low latency (sub-millisecond for millions of vectors), works well for dynamic datasets (easy inserts/deletes), and has predictable performance. The main trade-off is memory — HNSW requires the full index in RAM.

Major options: OpenAI's text-embedding-3-small (1536-dim, $0.02/1M tokens — best value), text-embedding-3-large (3072-dim, higher quality), Cohere embed-v3 (excellent multilingual, has input_type parameter for queries vs documents), Voyage AI voyage-3 (long context, good for code), and open-source models like BGE-large-en-v1.5 and E5-mistral-7b (free, self-hosted).

Selection criteria: Cost — text-embedding-3-small is 5x cheaper than ada-002 for similar quality; Quality — evaluate on your specific domain using MTEB benchmarks and your own test set; Dimensions — higher dims = more storage and compute but potentially better representation; Context length — if your chunks exceed 512 tokens, you need a model that handles longer input; Multilingual — Cohere and E5 excel for non-English; Vendor lock-in — you can't mix models (switching requires full re-embedding), so open-source avoids dependency; Latency — self-hosted can be faster than API calls if you have GPU infrastructure.

BM25 is a statistical keyword search algorithm that scores documents based on term frequency (how often query terms appear) and inverse document frequency (how rare those terms are), with document length normalization. It's pure lexical matching — no understanding of semantics.

Vector search encodes both query and documents as dense vectors and finds semantically similar matches. "How do I reset my password?" matches "login credential recovery process" even though they share no words.

Use BM25 when: queries contain specific identifiers (product codes, error numbers, proper nouns), exact terminology matters (legal, medical), or you need to match rare terms that embedding models might not handle well. Use vector search when: queries are natural language, paraphrasing is common, or you need to match concepts rather than exact words. In production, use both (hybrid search) — combine their results with Reciprocal Rank Fusion for the best of both worlds.

Research by Liu et al. (2023) showed that LLMs pay disproportionate attention to information at the beginning and end of the context window, while information in the middle is frequently ignored or underutilized. When 20 documents are placed in context, the model performs best when the relevant document is first or last, and worst when it's in positions 5-15.

Implications for RAG: (1) Place the most relevant chunks at the beginning of the context; (2) If using many chunks, consider placing the second-most-relevant at the end; (3) Use fewer, higher-quality chunks rather than many mediocre ones; (4) For long contexts, consider breaking the query into sub-questions and using separate, focused contexts for each. Some practitioners also repeat the most critical information at both the beginning and end.

Chunk overlap is when consecutive chunks share some text at their boundaries. For example, with a 500-token chunk size and 100-token overlap, chunk 1 covers tokens 1-500, chunk 2 covers tokens 401-900, etc.

It's used because fixed-boundary chunking inevitably cuts through sentences, paragraphs, and ideas. Overlap ensures that context at the boundary isn't lost — if a critical sentence spans the boundary between two chunks, the overlap ensures it appears in full in at least one chunk. Without overlap, you might have chunk 1 ending with "The refund policy allows returns within" and chunk 2 starting with "30 days of purchase" — neither chunk captures the complete policy.

Typical overlap: 10-20% of chunk size (e.g., 50-100 tokens for a 500-token chunk). Too much overlap wastes storage and creates near-duplicate chunks; too little defeats the purpose.

1. Query reception: User sends a question through the API. The system authenticates the user and extracts the tenant_id. 2. Query preprocessing: The query may be rewritten (resolving pronouns from chat history, expanding acronyms) or decomposed into sub-queries for complex questions. 3. Embedding: The (possibly rewritten) query is converted to a vector embedding using the embedding model. 4. Retrieval: The query embedding is used to search the vector database, filtered by tenant_id. Optionally, BM25 keyword search runs in parallel. Results are merged (e.g., via RRF). 5. Reranking: A cross-encoder model rescores the top 20-30 candidates, reordering them by relevance. The top 3-5 are selected. 6. Context construction: Selected chunks are formatted with source metadata and inserted into the prompt template alongside the system instructions. 7. Generation: The LLM generates an answer grounded in the context, with citations. 8. Post-processing: Guardrails check for hallucination, PII, and prompt injection. The response is streamed to the user. 9. Logging: The entire trace (query, chunks, scores, response, latency, cost) is logged for monitoring.

Metadata filtering adds structured conditions to vector similarity search. Instead of finding the top-k most similar vectors globally, you find the top-k most similar vectors that also satisfy specific conditions — like tenant_id, document type, date range, or access level.

In a multi-tenant SaaS, metadata filtering is a security requirement: you must ensure Tenant A never sees Tenant B's documents. It's also critical for relevance: a user asking about "2024 HR policies" should not retrieve 2022 policies, even if semantically similar.

Implementation matters: pre-filtering (apply filters before ANN search) is better than post-filtering (search first, filter after) because post-filtering can return fewer than top-k results if many candidates are filtered out. Databases like Qdrant and Pinecone do pre-filtering natively.

A good RAG system prompt must accomplish five things: (1) Ground the model — explicitly instruct it to answer only from the provided context; (2) Handle uncertainty — tell it what to do when the context doesn't contain the answer ("say I don't know" rather than guessing); (3) Enforce citations — require source attribution for every claim; (4) Set tone and format — match the product's voice (professional, conversational, technical); (5) Handle conflicts — instruct behavior when multiple sources disagree.

Key principles: be specific (don't say "be helpful," say "answer in 2-3 paragraphs with bullet points"), use delimiters to separate context from instructions, version your prompts and A/B test changes, and iterate based on failure cases.

An embedding model takes text in and outputs a fixed-size vector (e.g., 1536 numbers). It doesn't generate any text — it produces a numerical representation of meaning. It's used for comparison (is text A similar to text B?).

A generative LLM (GPT-4, Claude, etc.) takes text in and outputs new text token by token. It generates language — answers, summaries, translations. In RAG, the embedding model is used for retrieval (finding relevant chunks) and the generative LLM is used for answer synthesis (producing the final response from the context).

They can be based on the same architecture (transformers) but are trained with different objectives. Embedding models are trained with contrastive learning (push similar texts together, dissimilar texts apart). Generative models are trained with next-token prediction or instruction following.

Overlap ensures continuity between chunks. Without it, important information that spans a chunk boundary is split across two chunks, and neither chunk contains the complete thought. This degrades both embedding quality (each chunk has an incomplete idea) and retrieval (the user's query might match the complete thought but not either half).

Without overlap, you also risk losing transition sentences that provide context: "As mentioned in the previous section, the 30-day policy also applies to..." — without overlap, this sentence in chunk 2 references a concept only in chunk 1, making chunk 2 less useful in isolation. Typical overlap is 10-20% of chunk size.

Hybrid search combines vector (semantic) search with keyword (BM25) search. The challenge is merging two ranked lists with incompatible score scales.

RRF solves this by ignoring scores entirely and using only ranks. For each document that appears in any result list, its RRF score is the sum of 1/(k + rank) across all lists, where k is a constant (typically 60). This naturally gives high scores to documents ranked highly in multiple lists, while gracefully handling documents that appear in only one list.

Example: Document A is rank 1 in semantic, rank 3 in BM25 → RRF = 1/61 + 1/63 = 0.032. Document B is rank 5 in semantic, rank 1 in BM25 → RRF = 1/65 + 1/61 = 0.032. They score equally because both appear in both lists at comparable ranks. A document appearing only in one list gets a lower score. The k=60 constant makes lower-ranked results contribute less, but still contribute — making the fusion robust to small rank differences.

A reranker is a cross-encoder model that takes a (query, document) pair as input and outputs a relevance score. Unlike bi-encoders (used for initial retrieval) that encode query and document independently, cross-encoders process them together, allowing deep attention between query tokens and document tokens.

This produces much more accurate relevance judgments — the model can identify that "The return window is 30 business days" is relevant to "How long do I have to return a product?" even though the wording is quite different, by attending to the semantic relationship between "return window" and "how long to return."

The trade-off: cross-encoders can't precompute document representations (every query requires re-processing every document), so they're O(n) per query. This makes them impractical for searching millions of documents. The solution is a two-stage pipeline: use fast bi-encoder retrieval to get 20-50 candidates, then use the slow-but-accurate cross-encoder to rerank just those candidates. This typically improves retrieval quality by 10-25%.

HyDE (Hypothetical Document Embeddings) is a retrieval technique where instead of embedding the user's query directly, you first ask the LLM to generate a hypothetical answer, then embed that hypothetical answer for retrieval.

The intuition: queries and documents live in slightly different embedding sub-spaces. A question like "What causes inflation?" is semantically different in form from a passage explaining inflation. The hypothetical answer — a paragraph explaining inflation — is in the same "style" as the actual documents, so its embedding will be closer to the relevant chunks.

Use it when: queries are short and sparse (e.g., keyword-like queries), when there's a significant form mismatch between questions and documents, or when baseline retrieval recall is low. Avoid it when: the LLM might generate a wrong hypothetical (it can lead retrieval astray), when queries are already well-formed, or when latency is critical (adds an LLM call before retrieval).

The RAG triad evaluates three independent quality dimensions: Context Relevance (did retrieval find the right documents?), Groundedness/Faithfulness (is the answer supported by the retrieved context — no hallucination?), and Answer Relevance (does the answer actually address the user's question?).

A system can fail on any dimension independently. You might retrieve perfect context but the LLM ignores it (low groundedness), or the LLM faithfully summarizes irrelevant context (low context relevance), or the answer is factually correct from context but doesn't address the question (low answer relevance).

RAGAS is an open-source framework that automates this evaluation using LLM-as-judge. Key metrics: Faithfulness (fraction of claims in the answer that are supported by context), Context Precision (fraction of retrieved chunks that are relevant), Context Recall (fraction of required information that was retrieved), Answer Relevancy (semantic similarity between the question and the answer). You provide evaluation datasets with questions, retrieved contexts, generated answers, and ground truth, and RAGAS computes these metrics.

The challenge: follow-up questions use pronouns and references to previous turns. "What about the premium plan?" after discussing pricing means "What is the price of the premium plan?" but embedding "What about the premium plan?" alone won't retrieve pricing information.

Solution: query rewriting with conversation context. Before retrieval, send the chat history + latest message to an LLM that rewrites the query into a standalone, self-contained question. "What about the premium plan?" becomes "What is the pricing and features of the premium plan?" This rewritten query is then used for embedding and retrieval.

Implementation: keep the last N turns (typically 5-10) as context for the rewriting step. Store chat history in your database, not in the LLM's context for every message. Consider using a smaller, faster model (GPT-4o-mini) for query rewriting since it's a simpler task.

Semantic caching stores the response for a query along with its embedding vector. When a new query comes in, its embedding is compared against cached query embeddings. If similarity exceeds a threshold (e.g., 0.95), the cached response is returned without running the full RAG pipeline.

Benefits: dramatically reduces LLM costs and latency for repeated or near-repeated queries. Risks: (1) False positives — if the threshold is too low, semantically similar but actually different questions get the wrong cached answer ("How do I cancel my account?" might match "How do I create an account?" at 0.88 similarity); (2) Stale cache — if the knowledge base is updated but cached answers aren't invalidated, users get outdated information; (3) Tenant leakage — the cache must be per-tenant; a shared cache across tenants is a security vulnerability.

Best practice: start with a high threshold (0.95+), implement per-tenant caching, invalidate cache when the tenant's documents change, and log cache hits for monitoring.

Tables are challenging because when chunked as plain text, the column-row relationships are lost. Strategies:

1. Markdown table format: Convert tables to Markdown before chunking. This preserves some structure and most LLMs understand Markdown tables well. 2. Table summarization: Use an LLM to generate natural language summaries of each table, embed the summaries for retrieval, but include the original table in the context sent to the LLM. 3. Structured extraction: Parse tables into JSON objects, store them as metadata, and use self-query retrieval (LLM extracts filter conditions from natural language) to query them. 4. Multi-modal approach: Render tables as images and use vision-capable models (GPT-4o, Claude) to "read" them. Particularly useful for complex, merged-cell tables. 5. Dedicated chunk per table: Keep each table as a single chunk (even if it's large) with the table title and surrounding context as metadata.

For a SaaS product, start with Markdown conversion + dedicated table chunks. For high-value documents with complex tables, add LLM summarization.

Contextual retrieval (introduced by Anthropic) prepends each chunk with a brief explanation of its context within the larger document before embedding. The problem it solves: chunks often lose context when separated from their document. A chunk saying "The rate is 3.5%" is ambiguous — which rate? Which product?

Implementation: for each chunk, use an LLM to generate a 1-2 sentence context summary based on the full document: "This chunk is from the mortgage terms section of the 2024 Home Loan Product Guide. The rate is 3.5%..." The enriched chunk is then embedded. At retrieval time, the embedding captures both the specific content and its document context.

Anthropic reported that contextual retrieval combined with hybrid search (BM25) reduced retrieval failure by up to 49%. The cost is an additional LLM call per chunk at ingestion time, but this is a one-time cost that dramatically improves retrieval quality.

Parent-child chunking addresses the chunk size dilemma by using two levels: small child chunks for precise retrieval and larger parent chunks for context in the LLM prompt.

Process: (1) Split the document into large parent chunks (e.g., 2000 tokens — full sections or pages). (2) Split each parent into small child chunks (e.g., 200 tokens — paragraphs or sentences). (3) Embed only the child chunks for retrieval. (4) Store a mapping from each child to its parent. (5) At query time, search against child embeddings, then retrieve the parent chunks of the matched children and send those to the LLM.

This gives you precise matching (small chunks have focused embeddings) with sufficient context (the LLM sees the full surrounding section). It's particularly effective for detailed factual queries where you need to find a specific sentence but the LLM needs surrounding context to generate a complete answer.

Prompt injection is when malicious input overrides system instructions. In RAG, there are two attack vectors: direct injection (the user crafts a query like "Ignore all previous instructions and...") and indirect injection (malicious instructions are embedded in documents that get retrieved and placed in the LLM's context).

Mitigations: (1) Input sanitization: detect and reject queries that contain instruction-like patterns. (2) Structural separation: use XML tags or special delimiters to clearly separate system instructions from user input and retrieved context in the prompt. (3) Output filtering: detect if the response contains system prompt content, internal data, or content from other tenants. (4) LLM-based classifiers: use a fast model to classify inputs as benign or potentially malicious before processing. (5) Document scanning at ingestion: check uploaded documents for embedded instructions. (6) Least privilege: the RAG system should only have access to the querying tenant's data — even if an injection succeeds, the blast radius is limited.

MMR is a technique for diversifying retrieval results. Standard top-k retrieval often returns redundant results — the top 5 chunks might all say the same thing. MMR balances relevance to the query with novelty relative to already-selected chunks.

Formula: MMR = arg max [lambda * sim(query, doc) - (1-lambda) * max(sim(doc, already_selected))]. Lambda controls the trade-off: lambda=1 is pure relevance (standard top-k), lambda=0 is pure diversity. Typical sweet spot: lambda=0.5-0.7.

In practice, MMR works iteratively: select the first chunk by pure relevance, then for each subsequent chunk, penalize candidates that are similar to already-selected chunks. This ensures the final context covers different aspects of the topic rather than repeating the same information.

When a document is updated, you need to update the corresponding chunks in the vector database. Two approaches: Full re-indexing — delete all chunks for the document, re-parse, re-chunk, re-embed, re-insert. Simple and correct, but expensive for large documents. Incremental updates — detect which chunks changed (e.g., hash comparison), only re-embed those. Faster but complex because one edit might shift all subsequent chunk boundaries.

For a SaaS, full re-indexing per document is usually fine because most documents are small. The key is to make it asynchronous: the update API immediately marks the document as "re-indexing," queues a background job, and returns. The user sees a status indicator. Also important: invalidate any cached answers derived from the old version of the document.

For very large documents (100+ pages) or frequent updates, consider content-addressed chunking: hash each chunk's content, and only re-embed chunks whose hash changed.

Every RAG query should produce a trace containing: the original query, the rewritten query, retrieved chunk IDs and their scores, the reranked order, the prompt sent to the LLM, the generated answer, token counts (prompt + completion), latency breakdown (embedding, retrieval, reranking, generation), cost estimate, and eventual user feedback (thumbs up/down).

Key dashboards: (1) Quality: average retrieval score, answer relevance score, faithfulness score, thumbs-down rate. (2) Performance: P50/P95/P99 latency, error rate. (3) Cost: daily token usage, cost per query, cost per tenant. (4) Usage: queries per tenant, most common query patterns, zero-result queries. Tools: Langfuse (self-hosted, excellent cost tracking), LangSmith (best for LangChain), Phoenix/Arize (embedding visualization).

An evaluation dataset consists of (question, expected_answer, relevant_chunks) triples. Methods: (1) Manual curation: domain experts write 50-100 Q&A pairs covering various query types — factual, complex, multi-hop, opinion, edge cases. Highest quality but most expensive. (2) LLM-generated: feed documents to an LLM and ask it to generate diverse questions: fact-based, comparison, how-to, troubleshooting. Then have humans verify and curate. (3) Production mining: log real user queries and feedback. Queries with thumbs-down are immediate test cases. Queries with thumbs-up validate your current quality. (4) Hybrid approach: LLM generates a large set, humans verify a representative sample, use the verified set for evaluation. Aim for at least 50 questions covering different document types, query complexities, and edge cases (queries that should return "I don't know").

Temperature controls the randomness of the LLM's output. For RAG, you almost always want low temperature (0-0.3). High temperature introduces randomness that can cause the model to deviate from the context, add creative embellishments, or combine facts from different chunks in novel (and incorrect) ways.

Temperature 0: deterministic, most faithful to context, best for factual Q&A. Temperature 0.1-0.3: slight variation in phrasing, useful if you want natural-sounding responses. Temperature 0.5+: risky for RAG — the model may start generating plausible-sounding but unsupported claims.

Exception: for creative tasks grounded in context (e.g., "summarize this document in a fun tone"), moderate temperature (0.3-0.5) can produce more engaging output while still staying grounded.

In Agentic RAG, an LLM agent orchestrates the retrieval process — deciding which data sources to query, what queries to run, and whether additional retrieval steps are needed based on intermediate results. Unlike standard RAG's fixed pipeline (embed → search → generate), the agent makes dynamic decisions at each step.

It outperforms standard RAG for: (1) Multi-hop questions: "Which employees in the London office report to managers hired after 2020?" requires searching employees, filtering by office, looking up manager hire dates — multiple dependent queries. (2) Ambiguous queries: the agent can ask clarifying questions or search multiple interpretations. (3) Cross-source queries: when the answer requires combining information from different databases or document collections. (4) Queries requiring reasoning about retrieval: "Find all contradictions in our HR policy" requires retrieving policies, comparing them, and reasoning about consistency.

Trade-offs: higher latency (multiple LLM calls), higher cost, unpredictable execution paths (harder to debug and evaluate), and risk of infinite loops. Use standard RAG by default, and offer Agentic RAG for complex use cases or premium tiers.

Graph RAG builds a knowledge graph (entities as nodes, relationships as edges) from documents, then uses graph traversal + vector search for retrieval. Microsoft's implementation extracts entities and relationships using LLMs, clusters them into communities at multiple levels, and generates summaries for each community.

Comparison: Standard RAG retrieves individual text chunks — great for "What is X?" questions but struggles with "How does X relate to Y across the entire corpus?" Graph RAG can answer relational and aggregate queries by traversing the knowledge graph. "Who are all the people mentioned in connection with Project Alpha?" requires connecting entity mentions across many documents — graph traversal handles this naturally while standard RAG would need to retrieve and parse many chunks.

Graph RAG excels at: summarization of large corpora, relational queries, multi-hop reasoning, and finding patterns across documents. It's more expensive to build (LLM extraction of entities/relationships at ingestion) and maintain (graph must be updated when documents change). Best suited for structured domains with clear entities and relationships.

Multi-tenancy architecture should match tenant tier and scale. Three patterns:

Shared everything (free/starter tier): All tenants in one vector DB collection, isolated by tenant_id metadata filter. Cheap, simple, but risks noisy-neighbor performance issues and requires bulletproof filtering (a missing filter = data breach). Suitable for up to ~100 tenants, ~1M vectors.

Shared infra, isolated data (growth tier): Separate namespaces/collections per tenant within shared infrastructure. Each tenant has their own vector index, so noisy-neighbor issues are reduced. More operational overhead (managing many collections) but better isolation. Suitable for 100-10,000 tenants.

Dedicated infrastructure (enterprise tier): Each tenant gets their own vector DB instance, possibly in their own VPC. Complete data isolation, customizable performance, compliance-friendly (SOC2, HIPAA). Expensive and complex to manage. For enterprise contracts.

Key implementation details: extract tenant_id from the auth token (never from user input), build a retrieval wrapper that makes tenant filtering mandatory (impossible to query without it), implement per-tenant rate limiting, and monitor per-tenant costs for billing.

The challenge: when a document is updated, there's a window where the old chunks are being replaced with new ones. During this window, queries might return stale or missing data.

Solutions: (1) Blue-green indexing: write new chunks to a separate index/collection, then atomically swap the query target once re-indexing completes. Zero-downtime, but doubles storage temporarily. (2) Versioned chunks: each chunk has a version field. Write new chunks with version+1, then update the query filter to use the new version, then delete old chunks. (3) Shadow mode: new chunks are written alongside old ones. A "current_version" pointer controls which chunks are live. Update the pointer atomically. (4) Event sourcing: treat documents as an event stream. The vector DB is a materialized view that can be rebuilt from the event log.

For most SaaS products, versioned chunks with an atomic pointer update is the simplest reliable approach.

CRAG adds a self-correction loop after retrieval. A lightweight evaluator assesses whether retrieved documents are relevant to the query and triggers different actions: if relevant (CORRECT), proceed with refined chunks; if irrelevant (INCORRECT), fall back to web search or alternative knowledge sources; if ambiguous (partially relevant), combine refined internal results with web search results.

The evaluator is typically a fine-tuned classifier or a fast LLM prompt that scores relevance. The key insight: it's better to detect bad retrieval and recover than to blindly generate from irrelevant context.

Advantages: (1) Reduces hallucination from irrelevant context; (2) Graceful degradation when the knowledge base doesn't cover the query; (3) Can leverage web search as a fallback for general knowledge questions. Limitations: adds latency (evaluation step + potential fallback); the evaluator itself can make errors; web search fallback may not be appropriate for private/confidential queries.

When different users within the same tenant have access to different documents (e.g., managers can see performance reviews, regular employees cannot), you need document-level access control layered on top of tenant isolation.

Implementation: (1) At ingestion, tag each chunk with the document's ACL (access control list) — which roles, groups, or user IDs can access it. Store this as metadata. (2) At query time, resolve the user's permissions — which documents they can see. (3) Add an ACL filter to the vector search alongside the tenant_id filter. (4) Post-retrieval validation: before sending chunks to the LLM, verify the user has access to every chunk.

Challenge: when a document's permissions change, you need to update the ACL metadata on all its chunks — similar to a re-index but without re-embedding. Performance: complex ACL filters (user belongs to groups A, B, C; any of these groups has access) can slow down vector search if not implemented efficiently — pre-compute a user's accessible document IDs and use an IN filter.

Query decomposition breaks a complex query into simpler sub-queries, retrieves for each independently, then combines the results. Example: "Compare the pricing, features, and SLAs of our Enterprise and Pro plans" decomposes into: (1) "Enterprise plan pricing", (2) "Enterprise plan features", (3) "Enterprise plan SLA", (4) "Pro plan pricing", (5) "Pro plan features", (6) "Pro plan SLA".

It helps when: the query spans multiple topics or entities (comparison queries), when the answer requires information from different parts of the knowledge base, or when the query is too broad for a single embedding to capture all aspects.

Implementation: use an LLM to decompose the query (a fast model like GPT-4o-mini works well). Run retrieval for each sub-query in parallel. Deduplicate the combined results (same chunk might be retrieved for multiple sub-queries). Optionally rerank the combined set. Then generate the final answer with all the context. The LLM in the generation step is good at synthesizing information from multiple sub-topics into a coherent comparative answer.

Latency budget breakdown for a typical RAG query: embedding (~50ms API, ~10ms local), retrieval (~20-50ms), reranking (~100-200ms), generation (~500-2000ms). Generation dominates.

Optimizations: (1) Streaming: stream the LLM response — users see tokens immediately, perceived latency drops dramatically. (2) Parallel execution: run semantic search and BM25 search in parallel, not sequentially. (3) Embedding caching: cache query embeddings for repeated queries. (4) Local embedding models: self-hosted embedding avoids API round-trip (~50ms → ~10ms). (5) Semantic caching: skip the entire pipeline for similar previous queries. (6) Smaller generation model: GPT-4o-mini generates 2-3x faster than GPT-4o. (7) Reduce context size: fewer, better chunks = less to process = faster generation. (8) Skip reranking for high-confidence retrieval: if the top result's score exceeds a threshold, skip reranking. (9) Speculative execution: start generating with the first retrieved chunk while waiting for reranking to complete.

Target: P95 latency under 3 seconds for first token, under 5 seconds for full response.

Managed (Pinecone, Weaviate Cloud, Qdrant Cloud): Zero ops overhead — no server management, automatic scaling, backups, and upgrades. Faster time to market. Typically more expensive at scale, and you're dependent on the provider. Best for teams without dedicated infrastructure engineers, or when speed-to-market is the priority.

Self-hosted (Qdrant, Milvus, Weaviate, pgvector): Full control over infrastructure, data residency, and performance tuning. Lower marginal cost at scale. Requires DevOps expertise, monitoring setup, backup management, and version upgrades. Best for teams with infrastructure expertise, regulated industries needing data control, or very large scale where managed costs become prohibitive.

pgvector (special case): If you're already on PostgreSQL, pgvector eliminates an entire new service from your stack. Decent for up to ~5M vectors with HNSW indexing. Not as performant as purpose-built vector DBs, but the operational simplicity of one database is compelling for small teams.

Recommendation for a SaaS startup: start with Pinecone (managed) for fastest MVP, plan migration to self-hosted Qdrant when you reach scale where costs justify the engineering investment.

Two approaches: Multilingual embeddings — use a model trained on multiple languages (Cohere embed-v3, BGE-M3, multilingual-e5-large). These map text from different languages into a shared embedding space, so an English query can retrieve French documents. Translate-then-embed — translate all documents to a single language at ingestion, and translate queries at search time. Higher quality per-language but adds translation latency and cost.

For a SaaS product: use multilingual embeddings (Cohere embed-v3 is excellent) as the default. This lets tenants upload documents in any language and query in any language. For high-stakes use cases where translation quality matters, offer a translate-then-embed option.

Additional considerations: BM25 keyword search needs language-specific tokenization and stop-word lists. Rerankers should also be multilingual (Cohere Rerank v3 supports 100+ languages). The generation LLM should be instructed to respond in the user's language.

PQ is a vector compression technique that reduces storage and speeds up search at the cost of some accuracy. It works by: (1) splitting each vector into m sub-vectors (e.g., a 1536-dim vector into 192 sub-vectors of 8 dimensions each), (2) clustering each sub-vector space into k centroids (typically k=256), (3) representing each sub-vector by its centroid index (1 byte instead of 32 bytes for 8 float32 values). This compresses a 6KB vector to ~192 bytes — a 32x reduction.

At search time, distances between the query and codebook centroids are precomputed, and approximate distances to database vectors are computed by looking up precomputed values — making each comparison much faster.

Use PQ when: you have tens of millions to billions of vectors and can't fit the full HNSW index in RAM, or when you need to reduce storage costs dramatically. Don't use it for small datasets where brute-force or standard HNSW works fine. PQ is often combined with IVF (IVF-PQ): IVF narrows the search space, PQ makes the remaining comparisons fast and memory-efficient.

Prompt caching (offered by Anthropic and OpenAI) allows you to cache the static prefix of your prompt (system instructions, few-shot examples) across API calls. The cached portion is processed at a reduced cost (typically 90% discount on input tokens) and lower latency.

For RAG: your system prompt + instructions are the same for every query — this is the cacheable prefix. The retrieved context and user query change per request. With caching, a 500-token system prompt that previously cost ~$0.0013 per query (at GPT-4o rates) now costs ~$0.00013 — a 90% saving on that portion.

Implementation: structure your prompt so the static content comes first (system prompt, instructions, format examples), followed by the dynamic content (retrieved chunks, user query). The API automatically caches the static prefix. For Anthropic, you explicitly mark cache breakpoints. This is nearly free improvement and should be implemented from day one.

Standard metadata filtering requires the application code to explicitly construct filter conditions — the developer decides which fields to filter and how. Self-query retrieval uses an LLM to automatically extract structured filters from natural language queries.

Example: User asks "Show me HR policies from 2024 about remote work." Standard approach: the developer must parse "HR", "2024", and "remote work" from the query and construct {department: "HR", year: 2024, topic: "remote work"}. Self-query approach: an LLM is given the metadata schema (available fields and their types) and the query, and outputs: {search_query: "remote work policies", filters: {department: "HR", updated_after: "2024-01-01"}}.

Self-query is more flexible (handles ad-hoc filter expressions from users without developer intervention) but adds latency (an LLM call before retrieval) and can make errors (the LLM might extract wrong filters). Best used when the metadata schema is well-defined and users frequently ask filterable queries.

A knowledge base gap is when the user asks a legitimate question that the knowledge base doesn't cover. Detecting and handling these well is critical for user trust.

Detection: (1) Retrieval score thresholding — if the best chunk's similarity score is below a threshold (e.g., 0.3), the knowledge base likely doesn't cover the topic. (2) LLM confidence assessment — ask the LLM to rate its confidence that the context answers the question. (3) Claim coverage analysis — check if the generated answer makes claims not present in any retrieved chunk.

Handling: (1) Graceful decline — "I don't have information about that in the available documents." (2) Suggest alternatives — "I couldn't find information about X, but I found related information about Y. Would that help?" (3) Escalation — route to human support for unanswered questions. (4) Gap logging — track unanswered queries to identify content gaps for the customer. (5) Feedback to tenant — "Your knowledge base doesn't cover [topic]. Consider adding documentation about it."

This last point is a valuable SaaS feature: proactively telling tenants what topics their users ask about that aren't covered.

A feedback loop turns user interactions into retrieval improvements. Steps: (1) Collect signals: explicit feedback (thumbs up/down, ratings), implicit signals (user reformulates question = bad first answer, user copies answer = good answer, user follows up with "that's wrong" = bad answer). (2) Analyze failures: cluster thumbs-down queries by topic and failure mode — was it a retrieval failure (wrong chunks), a generation failure (right chunks, wrong answer), or a knowledge gap? (3) Improve retrieval: use positive query-chunk pairs as training data for fine-tuning a custom embedding model or reranker. Negative pairs identify chunks that should not be retrieved for certain queries. (4) Expand knowledge base: knowledge gap queries → notify tenant to add missing content. (5) Update evaluation set: add failed queries to your golden test set. (6) A/B test improvements: deploy retrieval changes to a subset of traffic, measure impact on feedback scores.

This is a flywheel: more queries → more feedback → better retrieval → better answers → more trust → more queries.

Requirements: Answer customer questions using help articles, FAQ, and past ticket resolutions. Low latency (<3s). Handle 1000+ queries/hour. Multi-language. Integration with ticketing system for escalation.

Architecture: Ingestion pipeline: help articles and FAQ synced from CMS via webhook → parsed (Markdown/HTML) → chunked (document-structure-aware, 400 tokens, 50 token overlap) → embedded (text-embedding-3-small) → stored in Qdrant with metadata (category, language, last_updated, article_id). Past ticket resolutions: nightly batch job extracts resolved tickets → filters for quality (CSAT > 4) → chunks and embeds.

Query pipeline: (1) Query rewrite (resolve pronouns from chat history, translate if non-English). (2) Hybrid search: vector + BM25 (Elasticsearch) with RRF. Filter by language and product category if detectable. (3) Rerank with Cohere Rerank v3 (multilingual). (4) Generate with GPT-4o-mini (low cost, fast), temperature 0.1, system prompt enforcing citations and "I don't know" behavior. (5) Stream response with source links.

Escalation: If retrieval score < 0.3 or user explicitly says "talk to a human" → create ticket in the ticketing system with the conversation context.

Feedback loop: Thumbs up/down on answers → weekly analysis of failures → content gap reports to the content team.

Architecture tiers:

Free tier: Up to 100 documents, 500 queries/month. Shared Qdrant collection with tenant_id metadata filter. text-embedding-3-small. GPT-4o-mini for generation. No reranking.

Pro tier: Up to 10,000 documents, 10,000 queries/month. Dedicated Qdrant namespace per tenant. Hybrid search (semantic + BM25). Cohere Rerank v3. GPT-4o for complex queries (auto-routing based on query complexity). Semantic caching. Custom branding on responses.

Enterprise tier: Unlimited documents. Dedicated Qdrant instance. Custom embedding model fine-tuning. Document-level ACL. SSO/SAML. Dedicated support. SLA guarantees. VPC deployment option.

Tech stack: FastAPI backend, Qdrant (managed for starter, self-hosted for enterprise), PostgreSQL (tenants, documents, billing), Redis (caching, rate limiting, job queues), S3 (document storage), BullMQ (ingestion jobs), Langfuse (observability), Next.js frontend.

Ingestion flow: Upload API → validate file type and size → store raw file in S3 → queue ingestion job → parse (unstructured library) → chunk (recursive + document structure) → embed (batched) → upsert to Qdrant → update document status in PostgreSQL → notify user via WebSocket.

Query flow: Query API → auth middleware (extract tenant_id from JWT) → rate limit check → semantic cache check → query rewrite (if chat mode) → hybrid search (vector + BM25) → rerank → generate (stream) → log trace → return response.

Security: Mandatory tenant_id filter on every vector query (enforced at the retriever wrapper level, not per-endpoint). Input sanitization for prompt injection. Output filtering for PII. Per-tenant encryption keys for enterprise.

Unique challenges: Extreme accuracy requirements (wrong legal advice is dangerous). Long documents (100+ page contracts). Complex cross-references ("as defined in Section 3.2(a)"). Specialized terminology. Strict confidentiality.

Chunking strategy: Document-structure-aware chunking using section/clause boundaries. Preserve section numbers and headers as metadata. Use parent-child chunking: embed small clause-level chunks for precise retrieval, return full section as context. Create a separate "definitions" index from the definitions section for cross-reference resolution.

Retrieval: Hybrid search heavily weighted toward BM25 (legal queries often reference specific terms, section numbers, clause language). Custom legal NER to extract entities (parties, dates, amounts) as metadata for self-query retrieval. Multiple retrieval passes: first for the specific clause, then for definitions and cross-references.

Generation: Conservative system prompt — cite specific sections, express uncertainty, never extrapolate beyond the text. Temperature 0. Include a disclaimer that this is not legal advice. Use a more capable model (GPT-4o or Claude Opus) given the high stakes.

Evaluation: Collaborate with lawyers to build a gold-standard evaluation set. Faithfulness is the #1 metric — every claim must be traceable to a specific clause. Human review for any response that includes a recommendation.

At 100M documents with an average of 10 chunks each, you're looking at 1 billion vectors. This requires careful architecture.

Vector DB: Milvus (designed for billion-scale). Use IVF-PQ indexing — IVF for coarse-grained partitioning into ~10,000 clusters, PQ for compression within clusters. This reduces RAM from ~6TB (full float32 HNSW) to ~200GB (PQ compressed). Distribute across multiple nodes with Milvus's built-in sharding.

Embedding: Self-hosted embedding model (BGE or E5) on GPU cluster. Batch processing for ingestion (not real-time API calls). Embedding cost would be $2M+ if using OpenAI API for 1B vectors — self-hosting is mandatory at this scale.

Ingestion: Distributed pipeline with Kafka for queuing. Multiple worker nodes for parsing, chunking, and embedding. Checkpointing for fault tolerance (if a worker crashes mid-batch, resume from last checkpoint).

Query optimization: Two-stage routing: first determine which partition(s) to search (using metadata or a lightweight classifier), then search only those partitions. Aggressive caching — at this scale, even a 10% cache hit rate saves enormous compute. Consider a global BM25 index (Elasticsearch) as first-pass filter before vector search.

Monitoring: Track recall at the partition level to detect index degradation. Monitor cluster health, replication lag, and query latency per partition.

Evaluation pipeline: (1) Golden dataset: 200+ (question, answer, relevant_chunks) triples curated by domain experts. Updated quarterly. (2) Automated evaluation: run RAGAS metrics (faithfulness, answer relevancy, context precision, context recall) on every deployment — block deployment if any metric drops more than 5%. (3) Shadow evaluation: run production queries through both the current and candidate pipeline, compare metrics offline before promoting. (4) A/B testing: route 5-10% of traffic to the candidate pipeline, measure user feedback (thumbs up/down rate) and automated metrics.

Continuous improvement: (1) Daily: automated alert if thumbs-down rate exceeds baseline. (2) Weekly: cluster failed queries by topic, identify top 5 failure modes, create improvement tickets. (3) Monthly: human evaluation of 50 random production queries by domain expert. Calibrate automated metrics against human judgments. (4) Quarterly: retune chunking, retrieval, and reranking parameters. Update golden dataset with new failure cases. Evaluate new embedding models and rerankers.

Infrastructure: Langfuse for tracing and cost tracking. RAGAS for metric computation. PostgreSQL for evaluation results and trend tracking. Grafana dashboard for quality metrics over time.

Challenge: Unlike static documents, CRM and database data changes in real-time. You can't re-embed on every change.

Hybrid approach: (1) Static knowledge (help docs, policies) → standard RAG pipeline with vector DB. (2) Dynamic data (CRM records, recent orders, account status) → SQL/API queries at runtime, no embedding.

Architecture: Use Agentic RAG with tool use. The agent has access to: a knowledge base search tool (for static docs), a CRM query tool (for customer data), a database query tool (for transactional data). The LLM agent decides which tools to invoke based on the query.

Example: "What's the return policy for my recent order?" → Agent calls CRM tool to get recent order details, then knowledge base tool for return policy, then synthesizes: "Your order #12345 (placed Dec 1) is a Premium item. Per our policy, Premium items have a 60-day return window, so you have until Jan 30."

For semi-dynamic data (changes hourly/daily), use CDC (Change Data Capture) to trigger re-embedding of changed records. For rapidly changing data, always query live.

PostgreSQL (relational metadata):

-- Core tables
tenants (id, name, plan, settings_json, created_at)
users (id, tenant_id, email, role, permissions)
documents (id, tenant_id, filename, file_type, s3_key, status, chunk_count, created_at, updated_at)
conversations (id, tenant_id, user_id, created_at)
messages (id, conversation_id, role, content, trace_id, created_at)
feedback (id, message_id, type, comment, created_at)

-- Analytics
query_traces (id, tenant_id, query, rewritten_query, chunks_json, scores_json,
              answer, model, prompt_tokens, completion_tokens, latency_ms, cost, created_at)

Vector DB (Qdrant): One collection per tenant (or shared with tenant_id filter). Each point: vector (1536-dim), payload: {tenant_id, document_id, chunk_index, text, metadata (headers, page_num, section), created_at}.

Redis: Cache keys: rate_limit:{tenant_id}, semantic_cache:{tenant_id}:{query_hash}, embedding_cache:{text_hash}. Session data for conversations.

S3: Raw documents: s3://bucket/{tenant_id}/documents/{doc_id}/{filename}. Parsed/chunked data: s3://bucket/{tenant_id}/processed/{doc_id}/chunks.json.

Two modes, different retrieval strategies:

Q&A mode: Standard RAG — embed question, retrieve top-k relevant chunks, generate grounded answer. Works well because questions are specific and retrieval can find targeted chunks.

Summarization mode: RAG is actually poor at summarization because vector search finds the most similar chunks, not the most important ones. A summary needs representative coverage of the entire document, not just the parts most similar to a query. Solutions: (1) Map-reduce: split document into chunks, summarize each chunk independently, then summarize the summaries. (2) Hierarchical: use Graph RAG's community summaries for corpus-level summarization. (3) Stuff method (small docs): if the document fits in the LLM's context window, pass the whole thing — no retrieval needed. (4) Iterative refinement: process the document in chunks sequentially, refining a running summary with each chunk.

Mode detection: Use a classifier or LLM to detect whether the user wants Q&A or summarization, then route to the appropriate pipeline. "Summarize the Q3 report" → summarization mode. "What were Q3 revenue numbers?" → Q&A mode.

Step 1: Isolate the layer. Is it a retrieval problem or a generation problem? Manually inspect the retrieved chunks — are they relevant to the query? If the chunks are relevant but the answer is wrong → generation problem (prompt issue, temperature too high, context too long). If the chunks are irrelevant → retrieval problem.

Step 2: Debug retrieval. Embed the query and manually inspect the top-20 results and their similarity scores. Are the scores generally low (<0.5)? → The knowledge base might not cover the topic. Are scores high but chunks are wrong? → Embedding model might be poor for your domain, or chunks are too large (diluted embeddings).

Step 3: Check chunking. Inspect the chunks that should have been retrieved. Are they in the vector DB? Is the relevant information in a single, coherent chunk, or spread across multiple chunks? Bad chunking often manifests as "the answer is in the DB but retrieval can't find it."

Step 4: Test BM25. Does keyword search find the right chunks? If BM25 finds them but vector search doesn't → the query and documents use very different vocabulary. Implement hybrid search.

Step 5: Check filters. Is a metadata filter inadvertently excluding relevant chunks? (Wrong tenant_id, date filter, category filter.)

Step 6: Compare embedding. Compute cosine similarity between the query embedding and the expected chunk's embedding. If it's low, the embedding model doesn't capture their semantic relationship well — consider a different model or contextual retrieval to enrich the chunks.

Investigation: Pull the traces for reported hallucinations. For each: (1) Were the retrieved chunks relevant? (2) Does the answer contain claims not present in any chunk? (3) Is the LLM extrapolating from partial information? (4) Are multiple chunks providing conflicting information that the LLM resolves incorrectly?

Common root causes and fixes:

(1) Low-relevance chunks in context: The retrieval returns marginally relevant chunks, and the LLM fills gaps with its parametric knowledge. Fix: implement a relevance score threshold — don't include chunks below 0.3 in context. Add a reranker to improve precision.

(2) System prompt too permissive: The prompt doesn't strongly enough constrain the LLM to the context. Fix: strengthen the "only answer from context" instruction. Add explicit examples of when to say "I don't know."

(3) Temperature too high: The LLM adds creative embellishments. Fix: set temperature to 0.

(4) Context too long: "Lost in the middle" effect — the LLM ignores relevant context buried in the middle of 10+ chunks. Fix: reduce to top 3-5 chunks after reranking.

(5) Faithfulness not being measured: Fix: add RAGAS faithfulness scoring to your monitoring. Alert when faithfulness drops below 0.85.

(6) Post-generation guardrails: Implement claim verification — decompose the answer into claims and check each against the context using NLI.

Systematic diagnosis by pipeline stage:

(1) Embedding latency: Has the embedding API slowed down? Check P95 embedding latency in traces. If increased, check the provider's status page or consider local embedding.

(2) Vector DB performance: Has the index grown? More vectors = slower search, especially with complex metadata filters. Check: total vector count, index segment count (too many unmerged segments = slow), RAM usage (if the index doesn't fit in RAM, search falls back to disk). Solutions: compact/merge segments, increase RAM, shard the collection.

(3) Reranker latency: Are you reranking more documents than before? If the initial retrieval top_k increased from 20 to 50, reranking takes 2.5x longer. Check if the reranker API itself is slower.

(4) LLM generation: Is the context getting larger? If chunks are bigger or you're sending more chunks, input tokens increase, and generation takes longer. Check average prompt_tokens per query over time. Also check: are you hitting rate limits (causing queuing), has the LLM provider degraded?

(5) Cache hit rate: Has it dropped? If new documents were added without cache warming, cache misses increase, and every query runs the full pipeline.

(6) Infrastructure: Check CPU, memory, network latency between services. A noisy neighbor on shared infrastructure can cause latency spikes.

Diagnosis steps:

(1) Check ingestion status: Did the document actually process successfully? Check the job queue for errors. Common failures: PDF parsing errors (scanned images without OCR, password-protected, corrupt file), embedding API timeout on large batches, vector DB insert failures.

(2) Inspect parsed content: Pull the parsed text from S3/storage. Is it readable? Scanned PDFs produce garbled text without OCR. Complex layouts (multi-column, headers/footers) confuse basic parsers. Check if tables and figures were extracted.

(3) Check chunk count and quality: How many chunks were created? For a 500-page PDF, you'd expect 1000-5000 chunks. If it's 10, something went wrong. Inspect a sample of chunks — do they contain coherent, complete thoughts?

(4) Test retrieval directly: Pick a sentence from the PDF and use it as a query. Does it retrieve the correct chunk? If not, the embedding or indexing has an issue.

(5) Chunk size mismatch: If chunks are too large (5000 tokens), the embeddings are diluted and can't match specific questions. Re-chunk at a smaller size.

Fixes: Upgrade PDF parser (try LlamaParse or Docling for complex PDFs), add OCR for scanned content, reduce chunk size, implement contextual retrieval to enrich chunks, add parent-child chunking for long documents.

This is a high-risk operation because you cannot mix vectors from different embedding models in the same index — they exist in incompatible vector spaces.

Migration plan:

(1) Evaluate first: Before committing, run your evaluation dataset with the new model. Compare retrieval metrics (precision, recall) against the current model. Only proceed if the new model is measurably better or required (e.g., model deprecation).

(2) Blue-green deployment: Create a new collection/index with the new model's vectors. Re-embed the entire corpus with the new model (batch process, may take hours/days depending on corpus size). Validate: run the golden test set against the new index.

(3) Shadow testing: Route production queries to both old and new indices in parallel. Compare results. Alert on significant differences.

(4) Cutover: Once validated, switch the query pipeline to the new index. Keep the old index available for 48-72 hours as a rollback option.

(5) Cleanup: After the rollback window, delete the old index and update the embedding model reference in your configuration.

Key risks: The new model might perform worse on edge cases not covered by your test set. Some embedding models handle certain domains differently. Always have a rollback plan.

Cost breakdown analysis first: Identify where the money goes. Typically: LLM generation (60-70%), embedding (5-10%), vector DB (10-15%), reranking (5-10%).

LLM cost reduction: (1) Model tiering — route simple queries to GPT-4o-mini ($0.15/$0.60 per 1M tokens) instead of GPT-4o ($2.50/$10). Use a classifier or simple heuristics (query length, complexity) for routing. Potential savings: 50-80%. (2) Reduce context size — better retrieval (reranker!) means fewer chunks needed. Going from 8 chunks to 4 halves input token cost. (3) Semantic caching — cache similar query results. Even a 20% hit rate saves 20% of LLM costs. (4) Prompt caching — cache the static system prompt prefix (supported by OpenAI and Anthropic). (5) Batching — for non-interactive workloads, use batch APIs at 50% discount.

Embedding cost reduction: Switch from text-embedding-3-large to text-embedding-3-small (6x cheaper). Use smaller dimensions (MRL — e.g., 256 instead of 1536). Cache embeddings for repeated queries.

Infrastructure: Right-size your vector DB instance. Use spot instances for embedding batch jobs. Implement per-tenant cost tracking and rate limiting to prevent outlier tenants from driving costs.

Data residency: Data must stay in specific regions (EU for GDPR, specific US regions for HIPAA). This affects: vector DB deployment region, LLM API endpoint (use regional endpoints or self-hosted models), object storage region, backup locations.

Data isolation: Dedicated infrastructure per tenant — separate vector DB instances, separate processing queues, separate storage buckets. No shared resources that could leak data.

Encryption: At-rest encryption with customer-managed keys (CMK). In-transit encryption (TLS everywhere). Consider field-level encryption for particularly sensitive fields in metadata.

PII handling: Implement PII detection at ingestion (using Presidio or similar). Redact PII from chunks before embedding (or offer configurable policies). Support "right to be forgotten" (GDPR Article 17) — ability to delete all data for a specific user, including cached embeddings, traces, and responses.

Audit trail: Log every data access with who, what, when, why. Immutable audit logs. Retention policies compliant with regulations (HIPAA: 6 years, GDPR: as long as necessary for stated purpose).

Model considerations: Some regulated industries may not allow sending data to external LLM APIs. Options: self-hosted open-source models (Llama, Mistral), Azure OpenAI (data doesn't leave Azure tenant), AWS Bedrock (data stays in your VPC). Document all third-party data processors for compliance.

SOC 2 / HIPAA BAA: Ensure all vendors (vector DB, LLM provider, cloud hosting) have appropriate certifications and are willing to sign a Business Associate Agreement (for HIPAA).