3.1 Embedding: How Machines "Understand" the Meaning of Language

First, why it's needed: at bottom, computers can only do math with numbers — they don't understand the relationship between the words "cat" and "dog." But if we can turn every word and every sentence into a string of numbers (say, a "coordinate" made up of 1,024 numbers), and arrange things so that "phrases with similar meaning" end up "close together" in the space these numbers represent while "phrases with unrelated meaning" end up far apart, then the computer can indirectly judge whether "these two sentences are about the same thing" simply by calculating the distance between two coordinates. This string of numbers is called an "embedding vector," and the conversion process is called "embedding."

An analogy: imagine a vast library where the librarian doesn't shelve books alphabetically, but instead places books on similar topics on neighboring shelves — a book about cats and a book about dogs both belong to the "pets" category, so they'd be placed close together; a book about cats and a book about auto repair would be placed far apart. To find "books similar to this one," you just look at what's sitting next to it. Embedding does exactly this, except instead of placing books on three-dimensional shelves, it places them in an abstract space with hundreds or thousands of dimensions (the human brain can't directly visualize what "1,024-dimensional space" looks like, but mathematically it's perfectly computable).

How this string of numbers is "learned": nobody manually decrees that "cat = [0.2, 0.8, ...]." Instead, it's done through a training method called contrastive learning — the model is shown vast quantities of "these two sentences are related" and "these two sentences are unrelated" examples, and every time it gets one wrong, its parameters are nudged slightly, pulling "related" pairs closer in the space and pushing "unrelated" pairs further apart. After repeating this millions of times, the model organically "figures out" a coordinate system in which "close together = similar in meaning" holds true. No single dimension is deliberately designed to "represent a specific meaning" — meaning is distributed across the combined "position" formed by all dimensions together, much like a person's character isn't determined by any single trait but by the overall impression composed of many traits.

The technical process in concrete terms (two steps):

1. Tokenization: first, the text is cut into small units the model can process, called "tokens." Note that a token is neither a "word" nor a "character" — it's a segment divided by frequency of occurrence: common algorithms (BPE, SentencePiece) start by splitting text into its smallest units, then repeatedly merge the pair that "appears together most frequently," until a vocabulary of the desired size is reached (typically 50,000 to 150,000 entries). So high-frequency words may be a single token, while rare words get split into several subword pieces.

This design has a practical benefit: no matter how rare or newly coined a word is, it can always be decomposed into known subwords or even individual bytes — there's never a situation where "this word is unrecognized and can't be processed." That was a common failure point with earlier tokenization methods.

One more point that's practically relevant to cost: the same passage of text takes up very different numbers of tokens in different languages. English runs roughly 4 characters per token; Chinese typically uses one to two characters per token. Since both model pricing and "how much it can process at once" are measured in tokens rather than characters, the same volume of Chinese content often consumes more tokens than its English equivalent.

1. Encoding (Transformer neural network): the sequence of tokens is fed into a neural network called a "transformer," which computes a vector for each token, then combines them via "pooling" (merging many token vectors into a single vector representing the whole sentence) and "normalization" (scaling the vector to unit length, so similarity can be compared using a standardized method). The result is a single vector that represents the meaning of the entire sentence. How many numbers this vector contains (i.e., its "dimensionality") — in 2026 the common range is 384 to 4,096, with general-purpose scenarios typically starting at 768 or 1,024, then adjusting based on retrieval quality and storage cost.

A note: earlier embedding models mostly used "encoder" architectures like BERT, while the current top-performing batch (e.g., NV-Embed, E5-mistral, GritLM) instead start from a large language model and add a pooling layer plus contrastive fine-tuning on top. Both approaches produce sentence vectors, and the usage is exactly the same — readers don't need to dig into the difference.

One limitation you must know about: embedding models have a maximum input length per call. Every embedding model specifies a maximum number of tokens for a single input; anything beyond that limit is silently truncated — with no error or warning. A chunk that's too large may have its second half simply not represented in the vector, making it permanently invisible to retrieval. This is one of the hard reasons why the "offline indexing" step in Chapter One must chunk long documents first: chunk sizes must fall within the length limit of the embedding model being used.

A rule that's easy to trip on but critically important: when you convert documents from your knowledge base into vectors ahead of time (this is called "indexing"), and when you convert the user's question into a vector at query time (this is called "querying"), you must use the same embedding model for both. The reason is simple: different models "learn" different coordinate systems, just like two maps that use different scales and different origin points — you can't take coordinates from Map A and look them up on Map B. So if you want to switch to a different embedding model, you can't just swap out the query side — you have to recompute every vector in the entire database (called re-embedding), and the dimensionality of the stored vectors and the query vectors must match.

The "dimensionality must match" above means the vectors stored in the database and the vectors computed at query time must be the same length — otherwise they simply can't be compared. But this doesn't mean the dimensionality itself is permanently fixed. Current mainstream models widely support a technique called Matryoshka (nesting-doll) embeddings: a long vector computed by the same model can simply have its trailing dimensions chopped off without retraining — for example, keeping only the first 256 out of 1,024 dimensions typically degrades retrieval quality by only a few percentage points, while dramatically reducing storage and search costs. If you want to save costs this way, that's perfectly fine — just make sure both the stored vectors and the query vectors are truncated to the same length.

3.2 Retrieval: How to Find Everything That Should Be Found

Chapter One mentioned that retrieval in a real system is a "wide-to-narrow funnel." This section explains why the funnel is designed this way — what deficiency each layer compensates for in the layer above it.

Start with the base capability: what vector search can do

Section 3.1 explained that text with similar meaning produces vectors that are close together. So the most basic form of "retrieval" is: convert the question into a vector too, then find the entries in the database whose positions are closest. As for how "closest" is concretely calculated, in practice it usually yields a score between 0 and 1, where closer to 1 means more closely matched in meaning — the "score 0.82" you see in the system is this number. Under the hood it's computing the angle between two vectors: the more aligned the directions, the higher the score. No need to dig into the details.

The greatest strength of this approach is that it doesn't depend on literal wording. A user asks "how do I move old data to the new version"; the documentation says "historical record migration procedure." The two sentences share not a single word in common, yet vector search connects them anyway. This is its fundamental advantage over traditional keyword search.

First-layer deficiency: at scale, comparing every single entry is unacceptably slow

If the database holds ten million vectors, comparing against every one of them and sorting the results for each question is something real-time Q&A simply can't sustain. The solution is ANN (Approximate Nearest Neighbor): rather than insisting on finding the absolute most similar entries with 100% accuracy, ANN uses pre-built index structures to compare against only a small fraction, trading a tiny loss in precision for a speedup of several hundred times.

This "precision" has a standard measure, called recall@k (top-k recall): out of the truly most similar top k entries, how many were actually found and returned. It's an adjustable knob, not a fixed setting — and the cost curve gets steep the higher you push it: going from 0.8 to 0.95 increases latency by roughly 30%, which is tolerable; pushing from 0.95 toward 0.99 can increase latency three- to fivefold. Most production systems settle around 0.95.

The specific index structures come in three main types, and which one to use depends on data scale:

• HNSW (Hierarchical Navigable Small World graph): pre-connects all vectors into a "neighbor relationship graph"; at query time, it works like playing "six degrees of separation" — starting from one point, at each hop jumping to a neighbor closer to the target, reaching the target region in just a few steps. The default choice for most production systems in 2026 — high recall, supports adding new data at any time. Its one hard constraint is that the entire graph must fit in memory, so the practical single-machine limit is roughly 100 to 200 million vectors.
• IVF (Inverted File Index): uses k-means to roughly cluster all vectors into groups, then at query time only searches the most relevant few clusters in detail. Lower memory usage than HNSW, but also lower recall under equivalent conditions — suited for large datasets with mostly static content and tight memory budgets.
• DiskANN: keeps the full index on SSD, with only compressed vectors and navigation structures in memory. The standard answer at the billion-vector scale — a single node can handle one billion vectors, 95% recall, 5ms latency, with memory usage roughly 90% lower than pure in-memory approaches.

Implementations of these methods can be found in FAISS (an open-source vector search library) and various commercial vector databases.

Second-layer deficiency: vector search misses things that require exact literal matches

Vectors' strength is understanding semantics, and that's also their weakness — they're insensitive to literal text. A user asks "does model X-200 support this feature?"; vector search is likely to return a bunch of passages about product specifications in general, yet miss the specific line that actually mentions X-200. Product model numbers, error codes, personal names, legal article numbers, version numbers — for content where a single character difference means something entirely different, semantic similarity is virtually powerless.

So production systems typically run two retrieval tracks in parallel: vector search for "meaning matches" and keyword search (the industry-standard algorithm is called BM25) for "literal matches," then merge the two result sets. This is hybrid search, and what Chapter One called "searching through two channels at once." When merging, because the two scoring systems aren't on the same scale, a conversion rule is needed. The common approach is called RRF (Reciprocal Rank Fusion) — rather than comparing scores directly, it only looks at what rank each result occupies in its respective list, and entries that rank highly in both get a higher combined score.

Third-layer deficiency: some of what's retrieved, the user shouldn't be allowed to see

Feeding retrieval results directly into the prompt is dangerous: different users have access to different documents; a customer shouldn't see internal-version documentation, and an external partner shouldn't see internal pricing details. So at the time of chunking and indexing, every chunk must carry metadata about which document it came from and who is allowed to see that document (i.e., ACL — Access Control List), and retrieval must filter in real time based on the querying user's identity. The same applies to timeliness filtering — superseded old-version documents shouldn't be dug up and used as evidence.

This step is more engineering-intensive than it sounds: filter first, then search breaks the ANN index structure (the neighbor-relationship graph was built on the full dataset — remove some of the points and the paths break); search first, then filter risks retrieving 50 entries but being left with only 2, or even zero, after filtering. Mature vector databases provide specialized filtered-search mechanisms for this, but it remains one of the most frequent sources of performance issues in real deployments.

Fourth layer: the dozens of rough-screened results still need further selection

The goal of the preceding layers is "don't miss anything" — better to bring back too many than too few. But the material stuffed into the prompt can't be too much: there are length limits, and irrelevant content distracts the model's judgment. So a final round of reranking is needed: a more precise but slower model takes the few dozen rough-screened results, compares each one against the question individually, rescores and reorders them, and keeps only the most on-point handful.

A reranking model works differently from an embedding model: the embedding model converts the question and the document separately into vectors and then compares distances — fast but coarse; the reranking model reads the question and the document together in one pass and then scores — slow but much more accurate. Precisely because it's slow, it can only be used after the candidate set has been narrowed to a few dozen — which also explains why the entire funnel has to be "wide first, then narrow": use fast-and-coarse methods to shrink the scope from millions to dozens, then use slow-and-precise methods to pick a handful out of those dozens.

3.3 LLM Serving: How the Model Handles Many Users at Once

The previous two sections covered "how the material gets found." This section covers what happens after the material is handed to the model. Note the keyword here is serving — the focus isn't only on how the model computes internally, but on how a server organizes things when it needs to serve hundreds or thousands of users simultaneously.

A single inference happens in two phases

The way a model generates text is called autoregressive generation: it produces one token at a time, appends it to the existing text, then predicts the next token based on "everything so far," repeating until it produces a stop token.

This creates two phases with opposite computational characteristics:

• Prefill: reads the entire input (system instructions + retrieved material + user question) all at once in parallel, computing two things (K and V, explained below) for every token, and storing them in a GPU memory buffer called the KV cache. This phase is compute-bound.
• Decode: starting from the first output token, generates one at a time, sequentially. Each new token written requires looking back at all preceding content. This phase is memory-bandwidth-bound — the bottleneck isn't how fast you can compute, but how fast you can read the KV cache into the processor.

Why the KV cache must exist: decode requires looking back at all preceding text for every new token. If every token required recomputing all the preceding tokens from scratch, then by the 100th token you'd be recomputing the first 99 — this redundant computation grows quadratically with text length, making long outputs completely impractical. Storing and reusing previously computed results brings the computation down to near-linear. Trading GPU memory for eliminated redundant computation — that's the entire point of the KV cache.

So what are K and V? When the model processes each token, it computes three vectors: Q (the question this token carries: "which preceding tokens are relevant to me?"), K (the sign each token holds up, reading "here's what I'm about"), and V (the actual information this token carries). The current token takes its Q and compares it against the K of every preceding token; whichever matches most strongly gets a higher weight; then the V values from all tokens are combined via weighted average — this is "attention." The reason only K and V are cached, not Q, is that K and V get reused by every subsequent new token — worth storing; Q is only used once in the current step, then discarded.

Two metrics for measuring speed

These two phases each correspond to a standard industry metric; understanding them lets you follow most discussions about inference performance:

• TTFT (Time To First Token): the time from sending the request to seeing the first output token, determined by prefill.
• TPOT / ITL (Time Per Output Token / Inter-Token Latency): after output begins, the rhythm between successive tokens, determined by decode.

It's worth noting that total wait time is usually dominated by decode. A response of roughly 500 tokens, at 80 milliseconds per token, takes about 40 seconds for decode alone; by comparison, the time to first token might be only 200 milliseconds. So "the longer the answer, the slower it feels" is a linear accumulation — the user experience is very direct.

How a server handles many users at once: continuous batching

If a server processes one request at a time, finishing it before accepting the next, the GPU sits idle most of the time — because during decode, compute capacity goes underutilized, resulting in severe waste.

Modern inference frameworks (vLLM, SGLang, etc.) use an approach called continuous batching: in each iteration, all currently active requests each advance by one token, then the next iteration begins; when a request finishes it exits the batch, and when a new request arrives it joins immediately — no need to wait for the entire batch to complete. This is the key to keeping the GPU fully utilized, and the reason it can serve dozens to hundreds of users simultaneously.

This mechanism also explains a phenomenon everyone has encountered: the response stutters for several hundred milliseconds mid-stream. When a new request's prefill squeezes into the batch, it demands a large burst of compute, forcing the requests currently outputting token-by-token to wait — what the user sees is the text stream suddenly freezing.

A few practical cost rules

• Longer inputs make prefill costs rise faster than you'd expect. The computational complexity of attention is quadratic in input length, so doubling the reference material from 2,000 to 4,000 tokens doesn't just double this component of the cost — it roughly quadruples it. This is the hidden price of "stuff more material into the prompt."
• Longer answers increase decode time linearly, and decode usually dominates the user's total wait time.
• The KV cache is a significant consumer of GPU memory, and it grows as the context gets longer — it's the main constraint on how many users a single machine can serve concurrently. To address this, the industry developed optimizations like PagedAttention¹: managing KV cache the way an operating system manages memory pages, preventing large blocks of GPU memory from sitting idle. The open-source inference framework vLLM uses exactly this approach.

An important optimization for prefill: prompt caching. In a RAG system, the beginning of every request is often identical — the same system instructions, sometimes the same long document. Since the content is the same, the computed KV cache is also the same, so there's no need to recompute it every time. Cloud providers therefore widely offer cross-request prefix caching: the KV cache for this shared prefix is retained for a period (typically minutes to hours), and subsequent requests that hit the same prefix reuse it directly, saving precisely that quadratic prefill cost.

This mechanism has an easily overlooked implication: the KV cache is not, as commonly assumed, "computed and discarded, existing only within the current request." With caching enabled, a portion of the content resides briefly in the provider's infrastructure — Chapter Five will return to this point when discussing compliance.

3.4 Who Calls Whom: How the System's Components Divide the Work in a Single Query

The core rule: across the entire system, only the "backend / orchestration layer" actively initiates actions. The vector database, the model services — these components are all passive: they respond only when asked, and they never communicate directly with each other.

Using the secretariat analogy from Chapter One, the backend is the secretary who owns this task. Which archive room to pull files from, what to do if they can't be found, whether the material is sufficient, whether to go back for another round, whether the draft is fit for submission — every one of these decisions is made by the secretary, and if something goes wrong, the secretary bears responsibility. The archive room's only job is "when someone comes to pull files, hand them over"; the writer's only job is "when material arrives, draft a response." The two never deal with each other directly — they may not even know the other exists. So the backend is not a messenger; it's the sole owner of this task.

The complete communication sequence:

1. Frontend → Backend (corresponds to Chapter One ❶ Receive the question): the user's question is sent to the backend.
2. Backend → Embedding service (corresponds to ❷ Embed & understand): sends the question text over, requesting it be converted into a query vector.
3. Backend → Vector database (corresponds to the first half of ❸ Retrieve): performs a similarity search using the query vector, simultaneously applying permission filtering based on the querying user's identity (ACL — Access Control List), and retrieves the most relevant entries (called top-k). The keyword search described in 3.2 is typically completed in this same step — most vector databases have this capability built in, returning both retrieval tracks' results merged in a single call. Only when the keyword index is a separate standalone system (e.g., a separately deployed Elasticsearch) does this step become two calls, with the backend merging the two result sets.
4. Backend → Reranking service (corresponds to the second half of ❸ Retrieve): sends the few dozen rough-screened results along with the question, requesting the service to score and reorder each one, keeping only the most relevant handful. The reranking model is the same type of component as the embedding model — both are standalone model services running on GPUs, which can be self-hosted or called via a ready-made API.
5. Backend (completed internally, no external call) (corresponds to ❹ Augment): assembles the system instructions, retrieved original text, conversation history, and user question into the complete prompt.
6. Backend → Generation model service (corresponds to ❺ Generate): sends the complete prompt over, requesting it to generate an answer.
7. Backend → Frontend (corresponds to ❻ Guardrails and ❼ Respond with citations): receives the model's output stream while checking it; the portions that pass are pushed to the interface along with their source citations.
8. Backend → Logging system (corresponds to ❽ Log): writes a record of this entire Q&A exchange for the permanent record.

Why the two "eight-step" breakdowns don't align neatly: Chapter One divided by "what action is performed"; this section divides by "who sends a message to whom." The two framings were never meant to map one-to-one. ❸ Retrieve is a single action but requires two to three separate communications; ❻ Guardrails and ❼ Respond with citations are two actions but share a single return transmission. Same system, different angle, naturally different slicing.

About step 7: the answer is "streamed" over, not "sent" after it's fully written

Section 3.3 explained that the model generates token by token. The backend doesn't wait for the entire answer to be complete before sending it back — it pushes tokens to the frontend as they arrive, which is why you see text appearing one character at a time (technically this usually uses SSE or WebSocket).

This creates a real engineering conflict: streaming output and guardrails are at odds. If tokens have already been pushed to the user's screen, how can you still "check before releasing"? The practical compromise is to detect problems in-flight while streaming, and the moment something is flagged, immediately interrupt and retract or replace what's already been displayed. This is why you sometimes see an AI's response vanish mid-sentence, replaced by "I'm sorry, I'm unable to answer that question."

Four points worth remembering:

1. The downstream components are mutually unaware of each other's existence. The vector database doesn't know there's a model service; the reranking service doesn't know there's a logging system — it's the backend that takes the output of one component, translates it into the format the next component expects, and passes it along. The more components there are, the more pronounced this becomes: in the eight steps above, the backend communicates with six different parties, while the number of direct connections among those parties is zero.

2. All judgment and gatekeeping can only happen at the backend. It's the only component with full visibility, and it sits within the company's own sphere of control (the model service may well belong to another company entirely). This is why compliance and security review logic must live in this layer — Chapter Five will cover this in detail.

3. The backend is also responsible for all "what if something goes wrong" handling. If the model service is rate-limited, should it queue and retry? If retrieval times out, should it degrade gracefully or return an error outright? If a step fails, where should it fall back to? Only the backend can make these calls. In real systems, the code for handling exceptions is often more extensive than the code for the happy path — this is precisely the difference between an "owner" and a "messenger."

4. Agentic RAG doesn't change this diagram (covered in detail in Chapter Four). It simply has the backend run steps 2 through 4 for additional rounds, with the number of rounds decided by the model on the fly. The division of labor, who can talk to whom — none of that changes.