LLM 3–4 - Decoding, Prompt Engineering, and RAG

Welcome back. In LLM 2 we opened the Transformer and followed information through embeddings, attention, feed-forward blocks, BERT, and GPT. This combined LLM 3–4 article begins where that architecture produces next-token scores and follows the entire application path: decoding turns scores into text, prompts specify behavior, tools perform checkable actions, retrieval supplies external evidence, and validation decides whether a claim is allowed to leave the system.

This is the bridge from language model to language-model system. We first study greedy search, beam search, sampling, temperature, and contrastive decoding. We then treat prompts as testable specifications, examine in-context learning and reasoning patterns, connect models to tools and structured validation, and finally build RAG from inverted indexes and BM25 through dense retrieval, ANN, HNSW, hybrid ranking, context construction, and citation checks. The former Lecture 3 and Lecture 4 articles repeated parts of prompt engineering; they are consolidated here so each idea is introduced once and then used by the RAG system.

From Representations to Generated Text

Reusing a Pretrained Model

A pretrained Transformer is useful because its internal representations can be reused. An encoder such as BERT can supply a contextual vector for every token or a sequence-level representation. A downstream system may freeze those vectors and train a small classifier on top, or fine-tune part or all of the pretrained network. The same encoder can therefore support sentence classification, question answering, named-entity recognition, or token tagging with different output heads.

A pretrained encoder can feed sentence-pair, single-sentence, span-prediction, and token-level heads. Course slide 2.

A decoder-only model exposes a different interface. Given a prefix , it defines a distribution over an output sequence by the chain rule:

The model never emits a whole paragraph in one operation. At step , it produces logits over the vocabulary, converts them to probabilities, chooses or samples one token, appends that token to the prefix, and repeats. Generation stops when it produces an end token or reaches a limit. This apparently small choice—*how do we turn a probability distribution into one token?*—is the decoding problem.

Probability Is Not Truth

Suppose the vocabulary at one step contains only three continuations with probabilities , , and . The probabilities express the model’s learned distribution under this context. They do not certify that the first continuation is factually correct. A token can be linguistically plausible yet false; an incorrect token with nonzero probability can be sampled; and a confident model can still be wrong because its training data or internal representation is incomplete.

This gives us a clean definition of hallucination at the system level: the model generates content that is unsupported, incorrect, or inconsistent with the available evidence. Decoding can make low-probability continuations more or less likely, but decoding cannot turn a probability model into a truth oracle. Factual applications need grounding, retrieval, tools, constraints, or verification in addition to a good decoder.

Deterministic Decoding

Greedy Search

The simplest decoder chooses the most likely token at every step:

This is greedy decoding. It is fast, deterministic when ties are resolved consistently, and often appropriate when variability is undesirable. But it optimizes each local decision, not the probability of the complete sequence. A slightly less probable first token may open a path whose later tokens are collectively much more probable. Once greedy search chooses the first branch, it never returns to compare the alternative.

The ideal maximum-a-posteriori sequence would be

but enumerating every possible sequence is impossible: with vocabulary size and output length , there are roughly candidates. We therefore use approximate search.

Beam Search

Beam search keeps the best partial hypotheses rather than only one. At each step it expands every surviving hypothesis with candidate next tokens, adds the new log probability to the accumulated score, and retains the best new prefixes. With , beam search becomes greedy search; a larger beam explores more branches but consumes more time and memory.

Beam search delays commitment by preserving several high-scoring partial sequences. Course slide 15.

We score in log space because products of many probabilities become numerically tiny:

Every log probability is non-positive, so a raw sum often favors short outputs: extending a sequence adds another negative term. Systems therefore use an end-token policy and often a length-adjusted score such as

where controls the length preference. This expression is representative rather than universal; libraries implement several penalties and normalizations. The important point is that the scoring rule is part of the model’s observed behavior. An apparently “bad model” may actually be a beam or length-setting problem.

Why the Highest-Probability Answer Is Not Always Best

Mode-seeking is useful for tasks with a narrow expected answer, but natural language often has many valid continuations. The global mode can be bland, repetitive, or generic because safe phrases collect probability mass across many contexts. Increasing the beam can even worsen open-ended text by searching more thoroughly for that generic mode. Diverse beam search deliberately discourages beams from collapsing onto nearly identical candidates; stochastic beam variants introduce randomness while retaining search structure.

For translation or constrained generation, beam search remains useful because fidelity matters and the output space is comparatively focused. For storytelling or brainstorming, sampling is usually a better match. “Best decoder” is therefore not a model-wide property. It depends on whether the task rewards reproducibility, coverage, creativity, factuality, or exact constraint satisfaction.

Sampling and Distribution Control

Ancestral Sampling

Instead of taking an argmax, ancestral sampling draws

at every step. Sampling respects the model’s distribution and can produce diverse outputs from the same prompt. It also exposes the long tail: a modern vocabulary may contain tens of thousands of tokens, and many receive tiny but nonzero probability. Over a long sequence, repeatedly allowing the entire tail creates opportunities for incoherent or unsuitable choices.

Truncated sampling removes part of that tail and renormalizes the remaining probabilities:

• Top- sampling keeps exactly the highest-probability tokens.
• Top- or nucleus sampling keeps the smallest set whose cumulative probability is at least .
• Epsilon sampling keeps tokens whose individual probability is at least a threshold .

Top- has a fixed candidate count even when the distribution changes shape. Top- adapts: a sharp distribution may need only a few tokens, while a flat distribution may retain many. Epsilon sampling uses an absolute probability floor. None is automatically safest; each changes what can be generated, and combinations can be used.

Temperature

Temperature transforms logits before softmax:

For , logit differences are magnified and the distribution becomes sharper. For , differences shrink and the distribution becomes flatter. As , sampling approaches greedy selection. Temperature changes relative probabilities; top- and top- decide which tokens remain eligible. They are related controls, not synonyms.

Lower temperature concentrates mass on leading tokens; higher temperature spreads it across more alternatives. Course slide 23.

A practical policy follows from the task. For extraction, classification, or code that must satisfy a schema, use low randomness and validate the output. For ideation, use moderate sampling and ask for several candidates. If factuality matters, do not rely on temperature alone: retrieve evidence and verify claims.

Contrastive Decoding

Contrastive decoding compares two models: an expert that represents the desired distribution and a weaker amateur that helps expose generic or locally tempting continuations. A candidate can be rewarded for high expert likelihood and penalized when it is also too likely under the amateur. A schematic score is

subject to a plausibility filter so that the method does not choose bizarre tokens merely because the smaller model dislikes them. Exact formulations vary. The central idea is comparative: prefer text that the stronger model supports beyond what a weaker model already explains.

Contrastive decoding favors plausible continuations that receive stronger support from the expert than from the amateur. Course slide 25.

Prompts as Model Interfaces

What a Prompt Actually Contains

A prompt is the information presented to a model to specify the current task. Good prompts often contain several distinct components:

1. Instruction: what operation the model should perform.
2. Input data: the document, question, table, or record to process.
3. Context: background facts and definitions needed for the task.
4. Examples: demonstrations of desired input–output behavior.
5. Role or persona: the perspective or expertise relevant to the response.
6. Output contract: format, tone, length, fields, and constraints.

Instruction, context, input data, and an output indicator form a useful minimum prompt structure. Course slide 51.

Not every prompt needs all six. A simple rewrite may need only text, audience, and tone. A legal-document extractor may need definitions, the document, a JSON schema, and a rule for missing values. The principle is to make hidden assumptions visible. “Summarize this” leaves audience, purpose, length, and treatment of uncertainty unspecified; the model must guess.

Prompting is also distinct from fine-tuning. A prompt changes the context for this inference call. Fine-tuning updates parameters across calls. Retrieval adds external information. Decoding chooses from the resulting token distribution. Keeping these mechanisms separate makes debugging much easier.

System, User, and Assistant Messages

Chat APIs typically organize a conversation into roles. The system message establishes application-level behavior and constraints. The user message contains the current request and data. Previous assistant messages can demonstrate style or preserve conversational history. Internally, the interface serializes these messages into a token sequence using special boundary markers; the model still receives tokens, not abstract chat bubbles.

A chat interface ultimately serializes role-tagged messages into the token sequence consumed by the model. Course slide 37.

Role separation is useful for application design, but it is not a perfect security boundary. Untrusted text inside a document may contain instructions that compete with the intended task. A robust system labels data as data, limits available tools, validates arguments, and enforces authorization outside the model.

Templates and Answer Prediction

A prompt template separates stable instructions from variables:

Task: Classify the support ticket.
Allowed labels: billing, access, bug, other.
Ticket: {{ticket_text}}
Return JSON: {"label": "...", "reason": "..."}

Templates improve consistency and make prompts testable. They also reveal which values require escaping or delimiting. Answer prediction begins the output with a structure the model should continue—for example, {"label":—which can reduce formatting drift, though it does not replace parsing or schema validation.

Building and Evaluating Prompts

An Iterative Engineering Loop

Prompt engineering is an experimental loop, not a one-time wording contest. First define the goal and acceptance criteria. Write a baseline prompt. Test it on representative cases, including difficult and adversarial ones. Inspect failures. Change one relevant element, then test again on the same evaluation set. Record the prompt version, model, decoding settings, and results.

Prompt work becomes engineering when revisions are driven by repeatable tests and failure analysis. Course slide 54.

“Good” is task-specific. For extraction, measure field accuracy and invalid JSON. For summarization, check coverage, attribution, length, and unsupported claims. For code, run tests. For customer communication, review tone and policy compliance. A polished answer to one hand-selected example is not evidence of reliability.

Context and Constraint Design

Use direct language and place essential information where it is easy to associate with the task. Delimit long input documents. Specify what to do when evidence is absent. Ask for a format that the program can validate. Replace vague constraints such as “be concise” with operational ones such as “use at most five bullets, one sentence each.” If the task has a rubric, include it.

A useful pattern is:

Goal
Context and definitions
Input data between explicit delimiters
Required steps or decision rules
Output schema
Failure behavior and constraints

The order is not sacred. What matters is that the model can distinguish instructions from data and that the application can judge the result. Long prompts are not automatically better: irrelevant detail competes for attention, consumes context, and can introduce conflicting cues.

The Interview Pattern

Sometimes the user has not supplied enough information for a responsible answer. The interview pattern explicitly asks the model to collect missing requirements before solving:

The model gathers missing constraints before committing to a solution. Course slide 77.

For example: “Ask one question at a time about the audience, budget, deadline, and required output. When enough information is available, summarize the requirements and propose a plan.” This is valuable when missing constraints would change the answer. It is unnecessary when the task is already precise or when delay is costly. The deeper lesson is that uncertainty about the problem definition should be resolved before optimizing the response.

In-Context Learning

Zero-, One-, Few-, and Many-Shot Prompts

In-context learning (ICL) means eliciting task behavior through examples placed in the prompt, without updating the model’s weights. A zero-shot prompt provides instructions but no demonstration. One-shot gives one example. Few-shot gives a small set. Many-shot uses a much larger set, made possible by long context windows.

Demonstrations define the local task through examples; they occupy context but do not update parameters. Course slide 57.

Examples can teach label names, output format, edge-case policy, and the kind of reasoning expected. If you want positive, neutral, and negative rather than free prose, demonstrations make that contract concrete. But a demonstration is also data. A wrong label, inconsistent format, or misleading example can steer the model in the wrong direction.

More Examples Are Not Automatically Better

Many-shot studies show that longer contexts can continue to improve performance on some difficult tasks, especially when the examples are diverse and informative. Yet repeating the same example mostly adds tokens, not new evidence. Distinct demonstrations tend to be more valuable because they cover more of the task’s variation.

Performance can continue to improve in the many-shot regime, but the useful number of examples varies by task. Course slide 66.

Too many examples can also hurt. Irrelevant or contradictory demonstrations add noise; a long prompt may obscure the current query; and compute cost grows with context length. The correct question is not “How many examples can fit?” but “Which examples most clearly define the behavior needed for this input?” Retrieval can help select demonstrations similar to the current case.

Ordering, Balance, and Coverage

ICL can be surprisingly sensitive to presentation. The order of examples may change the answer. An imbalanced prompt may bias the model toward the majority label. Omitting a label entirely can make it harder for the model to emit that label, even if the instruction lists it. Abstract or deliberately flipped labels reveal another influence: the model enters the prompt with prior associations from pretraining, and the demonstrations must overcome them.

Example order, label balance, and label coverage can materially change in-context performance. Course slides 70–72.

Therefore build an ICL set as carefully as a small training set: verify labels, cover important classes and edge cases, avoid accidental shortcuts, and test multiple orders. Keep an untouched evaluation set. If performance changes drastically under harmless reordering, the prompt is not yet robust.

Reasoning Patterns

Chain-of-Thought Prompting

Some tasks benefit when the model produces intermediate steps before an answer. Few-shot chain-of-thought (CoT) prompting includes demonstrations with worked reasoning; zero-shot CoT uses a cue such as “work through the problem step by step.” The additional tokens can help the model decompose arithmetic, symbolic, or multi-hop tasks that are difficult to solve in one jump.

Intermediate reasoning can expose the quantities and operations that connect a word problem to its answer. Course slide 81.

CoT is not a proof of correctness. A model can write a fluent but invalid derivation, or reach the right answer through a mistaken argument. In production, prefer checkable intermediate artifacts: equations that a symbolic tool can verify, citations that can be opened, code that can be tested, or a structured plan whose constraints can be evaluated.

Automatic Chain-of-Thought

Hand-authoring diverse reasoning demonstrations is expensive. Auto-CoT groups questions into clusters, selects representative questions, and uses a model to generate candidate rationales for those representatives. The resulting demonstrations are then used for other questions. Clustering aims to cover different reasoning patterns rather than selecting near-duplicates.

Auto-CoT seeks diverse demonstrations by clustering questions before generating rationales. Course slide 86.

Automatically generated rationales must still be filtered. A confident error becomes a reusable bad example. Correct final answers, valid intermediate steps, and coverage should all be checked before the demonstrations enter a prompt library.

Tree of Thoughts

A single chain commits to one continuation at every reasoning step. Tree of Thoughts (ToT) keeps multiple candidate states, evaluates them, and explores promising branches. A problem is represented by states, possible next “thoughts,” a search policy, and an evaluator. Breadth-first search, depth-first search, or a beam-like policy can traverse the tree.

Tree search preserves alternative plans instead of committing to one reasoning path immediately. Course slide 90.

ToT can help with planning, puzzles, or tasks where early choices are reversible and can be scored. It costs more model calls and depends heavily on the evaluator. If the evaluator cannot recognize progress, additional branches merely multiply plausible mistakes. “Multiple experts deliberate” is a related prompting pattern: generate independent proposals, compare them, revise, and aggregate. Independence and a clear selection rule matter more than theatrically assigning expert names.

ReAct, Tools, and Reliable Workflows

Interleaving Reasoning and Action

ReAct interleaves reasoning with actions in an external environment. The model may decide that it needs evidence, issue a search or database action, receive an observation, and continue. This closes a loop that ordinary prompting leaves open:

A ReAct trace uses observations from tools to revise the next action rather than relying only on model memory. Course slide 95.

Tools are valuable because some operations should not be approximated in language. A calculator should multiply and return . A database should supply the current account status. A search or retrieval system should provide evidence that can be cited. The model’s role is to decide when and how to call the tool, then explain the result—not to imitate the tool from memory.

Function calling gives the model a schema for available actions and asks it to return a function name plus arguments. The application must validate the arguments, enforce authorization, execute the function, and decide what result is safe to return. Never let generated arguments bypass normal security rules. Tool availability expands capability and risk at the same time.

Post-Processing Is Part of the System

An answer may be used as-is, formatted for display, reduced to selected fields, or mapped to an application action. Each path needs post-processing. Markdown and code require safe rendering. Classification needs an allowed-label check. Regression needs numeric parsing and range validation. Entity extraction needs a schema and a policy for missing values. A function call needs type checks, permissions, retries, and error handling.

Suppose the model should emit

{"priority":"high","team":"billing"}

The program should parse JSON, reject extra or missing fields, verify that both values belong to allowed sets, and ask for a repair or route to review if validation fails. A prompt that requests JSON improves the probability of valid output; a parser determines whether the output is actually valid.

This is also where factual checks belong. Retrieval can attach source passages; a verifier can require every claim to be supported; business rules can refuse impossible values. Reliability emerges from the composition of model, decoder, tools, validators, and fallback behavior.

Common Application Patterns

Writing, Translation, and Revision

Drafting an email is a useful small example because the user usually knows the facts but wants help with expression. A strong request supplies the recipient, purpose, required details, relationship, tone, and desired length. The model can then produce a draft that the user reviews. It should not invent a meeting time, promise, or policy. The boundary is clear: the model transforms supplied intent into language; the user remains responsible for factual and interpersonal commitments.

Translation has a similar structure but a different evaluation problem. The prompt should identify source and target languages, audience, domain, terminology rules, and whether the goal is literal fidelity or natural localization. Names, code, citations, and product terms may need to remain unchanged. For important text, back-translation is not enough to establish correctness; use a fluent reviewer or a domain glossary, and compare meaning at the sentence and document levels.

Proofreading should be separated from rewriting. “Correct grammar and punctuation without changing meaning or voice” asks for conservative edits. “Rewrite for a nontechnical executive audience” authorizes structural change. Asking the model to return both the revision and a compact edit log makes changes reviewable. That is especially important when a stylistic improvement could silently alter a numerical qualification or legal claim.

Code and Document Work

Code generation works best when the prompt behaves like a compact specification: state the language and version, function signature, data types, edge cases, performance constraints, forbidden dependencies, and tests. Generated code must be executed in an appropriate sandbox and checked with tests and static analysis. A plausible code block is a hypothesis about an implementation, not evidence that the implementation works.

For summarization, first decide what the summary is for. An executive summary, study guide, meeting recap, and evidence table are different outputs. Specify the audience, target length, required topics, and whether claims must cite locations in the source. Long documents may need chunking followed by synthesis, but naive chunk summaries can lose relationships across sections. A hierarchical pipeline should preserve document metadata and let the final synthesizer access the evidence behind each intermediate claim.

Meeting support illustrates the full lifecycle. Before a meeting, an LLM can turn source documents into an agenda and unresolved questions. During or after it, a transcript can be transformed into decisions, action items, owners, and deadlines. Each action item should be traceable to transcript evidence and confirmed by a person; speech recognition errors or casual suggestions should not become official commitments automatically.

Extraction and Structured Actions

Entity extraction turns prose into records: people, organizations, dates, amounts, relationships, or domain-specific fields. Define the schema, give examples of ambiguous cases, preserve source spans when possible, and distinguish missing from not applicable. A useful output includes both the normalized value and the exact text supporting it. That makes later review much easier than accepting an unexplained field.

Function calling goes one step further: the structured output selects an operation. A travel assistant might call search_flights; a support system might call lookup_order; a research tool might call search_documents. The model proposes a call, but the host program decides whether it is valid and authorized. Read-only search may run automatically, while purchases, messages, deletions, or account changes should require stronger checks and often explicit confirmation.

Across all these applications, the recurring pattern is the same: place the model where semantic flexibility is valuable, and surround it with deterministic components where correctness is testable. Let the model draft, classify, map language to a schema, or choose among safe tools. Let ordinary software enforce types, permissions, arithmetic, persistence, and irreversible actions.

Application Frameworks

LangChain as Orchestration

Frameworks such as LangChain package recurring application patterns. A prompt template fills variables into a reusable instruction. An output parser converts model text into a typed structure. Memory stores conversation state outside the model and inserts relevant history into later prompts. The model itself remains stateless between independent API calls unless the application resends state.

Apparent conversational memory is usually external state selected and passed into a new model call. Course slide 111.

A chain connects components. A simple sequential chain feeds one model output into the next step. A more general sequential chain can combine several named inputs and outputs. A router chooses among specialized chains according to the request. These abstractions are convenient, but they do not remove the need to inspect every boundary: What exact text enters the next prompt? Can a malformed output propagate? Where are errors handled? How is latency measured?

Chains compose model calls and program steps; routers select a path based on the request. Course slide 112.

LlamaIndex and Retrieval-Centered Systems

LlamaIndex focuses on connecting private or domain-specific data to LLM applications. Data connectors ingest documents or records. Parsers split and annotate them. Index structures—often vector indexes, but also lists, trees, or graphs—organize retrievable units. A query engine retrieves relevant context, optionally performs multi-step querying, and synthesizes a response. A chat engine adds conversational state around that retrieval process.

This is the basic retrieval-augmented generation (RAG) pipeline:

Retrieval changes the evidence available to the model; it does not guarantee that the answer uses that evidence correctly. Evaluate retrieval recall separately from answer faithfulness. A missing document is a retrieval problem; a claim contradicting a retrieved passage is a generation or validation problem. Separating those stages prevents vague “the RAG failed” diagnoses.

Putting the Whole System Together

We can now describe an LLM application without hiding any essential step:

1. Define the task and acceptance criteria. Decide what counts as a correct, safe, and useful answer.
2. Construct context. Gather user input, instructions, demonstrations, conversation state, and retrieved evidence.
3. Call the model. Choose an appropriate model and serialize role-tagged messages or a completion prompt.
4. Decode. Select greedy, beam, or sampled generation and set temperature and truncation controls for the task.
5. Use tools when needed. Delegate search, calculation, databases, and actions to systems that can perform them directly.
6. Parse and validate. Enforce schemas, permissions, factual support, and business rules outside the model.
7. Evaluate and iterate. Record failures, revise the responsible component, and retest on a stable evaluation set.

The key is causal clarity. If outputs are repetitive, inspect decoding and prompt diversity. If labels drift, inspect demonstrations, balance, and schema validation. If facts are stale, add retrieval or tools. If arithmetic is inconsistent, use a calculator. If a multi-step plan fails too early, consider branching search or explicit checkpoints. If the answer format breaks downstream code, strengthen the output contract and parser rather than merely asking the model to “be more careful.”

This completes the model-facing half of the system. Decoding determines how we move through the model’s distribution; prompts and demonstrations define the local task; reasoning patterns and tools organize difficult, checkable work; parsers and validators constrain the result. We can now address the missing half: how current or private evidence is found, ranked, fitted into context, and verified.

What RAG Changes

External Evidence at Request Time

Retrieval-augmented generation (RAG) retrieves external evidence relevant to a request, selects evidence for the context, and generates an answer conditioned on both the question and that evidence. It can provide information that is current, private, or too large to encode reliably in model parameters. It can also be cheaper and easier to update than retraining a model whenever a policy changes.

Model parameters and request context play different roles. Parameters contain general language capabilities and patterns learned during training; a retrieval request does not rewrite them. The application inserts current policy text into the context for this answer. If the policy changes tomorrow, update the document store and retrieval path—not the model’s memory of the world.

RAG is more than semantic search. Search returns relevant records. A RAG system also selects context, asks a generative model to use it, and validates the resulting claims. Lexical, dense, or hybrid retrieval can all supply candidates.

Offline Indexing and Online Querying

The pipeline has two broad phases. During indexing, the application reads documents, splits them into usable passages, creates lexical or dense representations, stores the passages and metadata, and builds search indexes. During an online query, it:

1. interprets the question;
2. retrieves candidates;
3. optionally reranks them;
4. selects permitted, nonredundant context;
5. generates an answer;
6. validates citations and claims.

Indexing prepares passages; the online path retrieves candidates, optionally reranks, selects context, and generates. Course page 70.

Every stage has a distinct failure mode. If the correct policy never becomes a candidate, generation cannot repair retrieval. If it is retrieved but removed during truncation, inspect context construction. If it reaches the model but the answer contradicts it, inspect generation and validation. This separation is the foundation of practical RAG debugging.

Lexical Retrieval and BM25

Inverted Indexes Preserve Exact Terms

Lexical retrieval is strongest when exact identifiers matter: product codes, error messages, names, dates, and quoted phrases. An inverted index maps each term to a postings list of documents containing it. For three documents,

D1: BB43300 warranty, US
D2: BB43300 setup guide
D3: CC100 warranty, US

the posting list for BB43300 is and the list for warranty is . An AND query intersects them and returns .

An inverted index retrieves exact lexical candidates efficiently; a ranker scores those candidates next. Course page 72.

Tokenization, case normalization, stemming, stop-word handling, field boosts, and query operators all affect the result. Serial numbers should often remain intact. A tokenizer that splits BB43300 unpredictably can destroy the very exact match lexical search is meant to preserve.

BM25 Scoring

BM25 combines term rarity, term frequency with diminishing returns, and document-length normalization. A common form is

Here is the frequency of query term in document , is document length, and is average document length. controls how quickly frequency saturates; controls the strength of length normalization. A typical inverse-document-frequency term is

where is the number of documents and is the number containing the term. Rare terms receive more weight than common ones.

If , document length equals the average, and , the frequency contribution becomes

At , the contribution is ; at , it is approximately . Repeating a term helps, but each additional occurrence helps less.

BM25 limits the reward for repeated terms instead of increasing the score linearly forever. Course page 76.

Length normalization prevents long documents from winning simply because they repeat more terms. It can also penalize legitimately long records, so and must be evaluated on the actual corpus. BM25 reduces keyword stuffing incentives; it does not prevent every form of search manipulation.

Dense Retrieval and Chunking

Closing the Vocabulary Gap

Lexical search fails when a query and a relevant passage express the same idea with different words. “Strong pain in the side of the head” and “sharp temple headache” may share no useful keyword. A dense embedding model maps both into vectors trained so that semantically related queries and passages score similarly.

Dense representations can bridge meaning when exact vocabulary does not overlap. Course page 78.

Dense retrieval does not discover a universal geometry of meaning. Similarity depends on the encoder, training objective, domain, input formatting, and scoring metric. A model trained for general sentence similarity may not rank product policies correctly. Query and document encoders must be compatible, and embedding model versions must be tracked when the index is updated.

Bi-Encoders and Similarity

A bi-encoder computes a query vector and each document vector separately:

Document vectors can be computed once and reused for many queries, which makes large-scale retrieval practical.

A bi-encoder makes stored document vectors reusable across queries. Course page 81.

Dot product and cosine similarity can rank the same vectors differently. Cosine removes vector magnitude:

For , has dot product and cosine , while has dot product and cosine . Dot product ranks first; cosine ranks first. If vectors are normalized before storage, dot product and cosine produce the same ordering. The index and evaluator must use the metric for which the embeddings were trained.

Chunking Is a Retrieval Decision

Documents are usually split into chunks before embedding. A tiny chunk may match the query but omit the sentence needed to answer it. A huge chunk preserves context but consumes tokens, mixes topics, and can dilute the retrieval signal. Useful chunk boundaries often follow headings, paragraphs, code units, policy clauses, or tables rather than arbitrary character counts.

Chunks should contain enough local context to support an answer and remain traceable to their source. Course page 83.

Every chunk should retain metadata such as source ID, document version, heading, product, region, effective date, access policy, and character or page offsets. The vector finds a candidate; metadata determines whether the candidate is applicable, authorized, current, and citable.

Approximate Nearest Neighbors

Why Exact Search Becomes Expensive

Exact nearest-neighbor search compares a query with every stored vector. For vectors of dimension , this costs about distance work per query. Top- selection does not require fully sorting all distances, but it still requires computing them.

Memory matters as well. One million 768-dimensional float32 vectors contain

or about GB of raw vector values before IDs, metadata, graph edges, and index overhead. At larger scales, an approximate nearest-neighbor (ANN) index reduces work by searching selected regions or candidate paths. The cost is that it may miss the exact nearest result.

Navigable Small-World Graphs

A navigable small-world (NSW) index represents vectors as graph nodes. New nodes connect to selected nearby nodes, while some longer links allow movement between distant regions. Search begins at an entry node, evaluates its neighbors, moves toward candidates closer to the query, and explores the neighborhood until its stopping rule is satisfied.

Local links support refinement while longer links make distant regions reachable. Course page 96.

A purely greedy walk can stop at a local choice. Let and nodes be , , , and , with edges , , and . Starting at and moving only to a strictly closer neighbor reaches at distance . From there is no closer connected node, so the walk stops—even though is the exact nearest node at distance . Reaching would require first moving through the temporarily worse node .

An entry point and local graph structure can lead approximate search away from the exact nearest neighbor. Course page 105.

Practical graph search keeps a candidate set instead of following only one edge. More exploration generally improves recall at a latency cost.

HNSW Adds a Hierarchy

Hierarchical Navigable Small World (HNSW) assigns nodes to random maximum levels. Every node appears in the dense base layer; progressively fewer nodes appear in higher layers. Search starts in a sparse upper layer, makes long-distance progress, descends through denser layers, and finishes with local candidate exploration at layer .

Random maximum levels create sparse navigation layers above a dense base graph. Course page 109.

Upper layers support coarse navigation; the base layer supplies the final candidate neighborhood. Course page 108.

HNSW exposes a recall-latency-memory trade-off. Greater graph degree and more construction effort usually improve index quality but consume memory and indexing time. Exploring more candidates at query time can improve recall but increases latency. The often-quoted logarithmic intuition is not a universal worst-case guarantee; measure the target workload against an exact reference.

Measuring ANN Recall

For exact top- IDs and approximate top- IDs , define

If and , recall@3 is . The comparison must use the same corpus, vectors, metric, filters, and . High ANN recall only means the approximate index resembles exact vector search. If both return an outdated policy, the freshness problem remains.

A vector database connects vector IDs to source text and metadata, supports similarity queries, and coordinates updates and deletes. Filtering may be required before or during ANN search so that users see only authorized records. Post-filtering can leave fewer than eligible results. Stored text, metadata, and vector indexes must remain synchronized.

Hybrid Retrieval and Reranking

Complementary Candidate Signals

Lexical and dense retrieval fail differently. Lexical search preserves exact identifiers such as BB43300; dense search can match paraphrases such as “How long is coverage?” to a passage containing “warranty duration.” A hybrid retriever runs both, unions their candidate IDs, removes duplicates, and combines their rankings.

Raw BM25 and cosine scores live on different scales, so adding them without calibration is unreliable. Reciprocal rank fusion (RRF) combines ranks instead:

where is result list , ranks start at , and is a rank constant. With , candidate at lexical rank and dense rank scores

while candidate at ranks and scores

wins because it ranks strongly across the two lists. RRF is robust to incompatible raw-score scales, but it cannot recover a document absent from every candidate list.

Reranking a Smaller Set

A bi-encoder retrieves cheaply from a large corpus because query and document vectors are computed separately. A cross-encoder jointly processes each query-document pair, allowing richer interaction but requiring a model call or forward pass per pair. This makes cross-encoders well suited to reranking dozens of candidates rather than scanning millions of documents.

Reranking spends more compute on a candidate subset after the high-recall retrieval stage. Course page 123.

Reranking improves ordering, not truth. “The capital of Canada is Sydney” may be highly related to the query while still false. A reranker also cannot select the correct policy if retrieval never returned it. When the answer is missing, inspect candidate recall before tuning the reranker.

Context Construction, Verification, and Debugging

Build Context Under a Token Budget

The final model input should contain relevant, permitted, and nonredundant passages while reserving space for instructions and output. Context construction may deduplicate overlapping chunks, group adjacent sections, prefer authoritative versions, and remove passages outside the product, region, date, or user authorization scope.

Preserve metadata in the final context. A passage saying “12 months” is not enough if the model cannot tell which product, country, date, or source it belongs to. For the running example, the selected context might be:

Context element	Content
Application rule	Answer only when supplied evidence supports the claim; otherwise report insufficient evidence.
Selected evidence P1	BB43300; US; effective 2026-08-01; warranty 12 months.
Excluded candidate P2	A policy for another product; not evidence for this question.
Generation budget	Reserve space for a concise answer and a P1 citation.

The supported answer is narrowly scoped: “For BB43300 in the US, the supplied policy states a 12-month warranty [P1].” If no applicable policy exists, answer “Insufficient evidence to determine the warranty.” Do not silently generalize a US policy to Canada or a single product to an entire product family.

A Citation Must Support the Claim

A citation is useful only when it supports the exact claim next to it. If P1 states that BB43300 has a 12-month US warranty, it does not support “All BB devices have a two-year worldwide warranty [P1].” Claim verification should decompose the response into checkable statements and compare each with the permitted passages.

Missing, conflicting, and hostile evidence require explicit behavior:

• Missing: ask for required context or abstain.
• Conflicting: compare authority, scope, and effective date; do not average policies.
• Hostile: treat instructions inside retrieved text as evidence content, not application authority.
• Unauthorized: exclude the record before generation, even if it is highly relevant.

Diagnose the Stage That Failed

A good trace records query transformations, filters, lexical and dense candidates, scores, ANN settings, reranker results, selected chunks, final model input, output, and claim-to-source checks. Then an observed failure points to a first inspection target:

Observed failure	First stage to inspect	Useful evidence
Correct policy absent from candidates	Retrieval	Corpus, filters, query, scores, exact-vs-ANN comparison
Policy retrieved but absent from model input	Context construction	Selected chunks, deduplication, authorization, truncation
Policy in context but answer unsupported	Generation and validation	Exact input, output, and claim-to-passage support

Frameworks may name components differently—loader, reader, splitter, parser, embedding model, vector store, retriever, postprocessor, prompt builder, response synthesizer, tracing hook—but they organize the same responsibilities. The names do not replace measurement.

The whole lesson can now be summarized in one division of labor:

• Instructions specify the task.
• Demonstrations show expected behavior.
• Retrieval supplies candidate evidence.
• ANN trades exact vector neighbors for speed and memory efficiency.
• Hybrid search and reranking decide which candidates deserve attention.
• Context construction determines what actually reaches the model.
• Generation turns selected evidence into language.
• Validation checks whether each claim is supported.

If a Canadian customer asks about BB43300 but the selected context contains only a US policy, the correct response is not to guess from the US duration. The assistant should state that the supplied evidence does not establish Canadian coverage, and the application should retrieve a Canada-specific policy or request the missing authority. That final restraint is what turns a fluent answer into an evidence-grounded system.