Interview: Design RAG for Millions of Documents

What you will be able to do

• ✓Design an ingestion pipeline for millions of documents
• ✓Estimate embedding time and cost
• ✓Design a low-latency query path
• ✓Plan sharding, caching and failure handling
• ✓Choose the metrics to monitor

Intermediate: What do you clarify before designing RAG at scale?

How many documents and how fast they change, document types, query volume and latency target, accuracy needs, permissions, languages, budget, and where the system must run.

• •Write the numbers down - they drive every later choice.

Intermediate: How do you design ingestion for millions of documents?

A queue of document jobs and a pool of workers that parse, clean, chunk, embed in batches and upsert into the index - with retries, a dead-letter queue for failures, and progress tracking.

• •Parsing and embedding are the slow parts - scale workers for them.

Intermediate: How do you avoid re-embedding everything when documents change?

Store a content hash per document (or per chunk) and re-process only what changed; delete chunks of removed documents; keep stable chunk IDs.

• •Incremental ingestion keeps cost and time low.

Advanced: How would you estimate the time and cost of embedding the whole collection?

Count tokens (documents x average tokens), divide by measured throughput for batches, and multiply by price per token for a hosted model - or by GPU hours for a local one. Add re-embedding when the model changes.

• •Example: 10 million chunks x 300 tokens = 3 billion tokens.
• •Measure throughput with batches first - batching nearly halved the time in Lesson 2.4.

Advanced: How do you keep query latency low?

Set a latency budget per step: query embedding, retrieval with a tuned ANN index, re-ranking only the top candidates, and a fast or streaming LLM. Cache embeddings and frequent answers, and run independent steps in parallel.

• •Generation usually dominates - Lesson 1.6 measured 1.3 s for the model and under 1 ms for the rest.
• •Stream tokens so the user sees progress.

Advanced: How do you shard and replicate the index?

Shard by document range, tenant or hash so each shard fits in memory; query shards in parallel and merge the top results; add replicas for read throughput and failover.

• •Sharding by tenant also helps isolation and deletion.

Advanced: What caching would you add?

Cache query embeddings, retrieval results for repeated questions, and full answers for very common questions - with expiry tied to document updates so cached answers do not go stale.

• •Semantic caching (similar questions) needs care: similar is not identical.

Advanced: How do you handle failures in the pipeline?

Retries with backoff for temporary errors, a dead-letter queue for documents that keep failing, idempotent upserts so retries do not duplicate chunks, and alerts on ingestion lag.

• •A stuck queue means a stale index - monitor freshness.

Advanced: Which metrics would you monitor in production?

Ingestion lag and failures, index size, retrieval latency and recall on a test set, end-to-end p95 latency, tokens and cost per query, answer quality from evaluations and user feedback, and "no answer" rates.

• •Freshness is a quality metric too.

Professional: Walk me through your full design for RAG over 10 million documents.

Ingestion: connectors -> queue -> parse/clean/chunk workers -> batched embedding -> upsert into a sharded vector index plus a keyword index, with hashes for incremental updates. Query: API -> auth and filters -> query cleaning -> hybrid retrieval -> re-rank top 50 -> prompt with top 5 -> streaming LLM -> citations. Around it: caching, tracing, evaluation set, cost budgets and alerts.

• •Say the numbers: chunks, QPS, latency budget, monthly cost.
• •End with risks and what you would build first.

Professional: How would you cut the monthly cost of this system by half?

Find the biggest cost first from traces. Typical levers: a smaller or cheaper LLM for easy questions (routing), fewer and shorter chunks in the prompt, answer caching, cheaper or local embeddings, quantised or shortened vectors, and turning off re-ranking where it does not help.

• •Measure quality after every cut.

Professional: How would you roll out a new version of the pipeline safely?

Build the new index in parallel, evaluate it offline, shadow-test it on live traffic, then shift traffic gradually with metrics and a fast rollback to the old index.

• •Never re-index in place without a way back.

Interview questions