When You Need an Agent, and When You Don't

The RAG described earlier is a fixed pipeline: retrieve material, assemble, generate — one pass through. For many questions, that's enough.

But some tasks can't be completed in a single pass. Take the AI customer support example again: a customer reports "after upgrading to the new version, a certain feature stopped working." To answer this, the system needs to check whether there are known issues with that version, then look up which configuration the customer is using, and possibly also check whether there are relevant change notes for that specific configuration — and what to look up in the next step depends on what was found in the previous one. If the first step already turns up the answer in the known issues list, the remaining steps don't need to happen at all.

This mode of operation — "break the task down on its own, decide the next step on its own, loop until done" — is what an agent is. The core distinction from a fixed pipeline is: how many steps to take, and what to do in each step, is decided by the model on the fly — not hard-coded in advance.

The flip side also needs to be stated clearly: simple tasks should not use agents. For questions that can be answered in a single step, calling the model directly or running a fixed RAG pipeline is better — faster, cheaper, and easier to debug when something goes wrong. The prevailing advice is to start with the simplest fixed pipeline and add complexity only when genuinely necessary. Autonomy has costs, which we'll cover at the end of this chapter.

ReAct: The Most Common Operating Pattern

The most fundamental and widely used agent pattern is called ReAct¹, named after the paper's title combining Reasoning and Acting. In the original paper, the model's execution trace alternates between three parts:

• Thought: based on the information it has so far, the model judges what to do next
• Action: the model requests a tool call — querying a database, running a calculation, hitting an API endpoint
• Observation: the content returned after the tool executes, fed back to the model by the harness

Note that the first two are produced by the model; the third is fed in from outside. Later references often write these three steps as Reason / Act / Observe to align with the ReAct name — they mean the same thing.

The three steps cycle repeatedly until the model judges the task complete.

ReAct is the default starting point, but not the only pattern. When a task is complex enough that a single loop can't keep up, the model can be made to first decompose the entire task into a plan and then execute each item; when the same type of error recurs, a layer can be added for the model to review its failure reasons. The industry's experience is: start with ReAct, establish baselines for success rate, tool-call accuracy, latency, and cost; only upgrade once you've confirmed that a single agent genuinely can't solve the problem — premature complexity is a common and expensive mistake.

Four Supporting Concepts

Harness

The model itself is simply an inference engine that generates output given input context. It doesn't maintain state on its own, doesn't loop on its own, and can't directly execute external operations. What actually organizes the model into an agent is the surrounding runtime layer, typically called the harness. The harness drives the model through repeated cycles of thinking, calling tools, receiving results, and continuing to reason, enabling the system to carry tasks through to completion.

Beyond driving this loop, the harness is also responsible for: maintaining the roster of available tools, parsing the model's tool-call requests and actually executing them, deciding what goes into the context window each round (whether old content gets compressed or dropped), retrying on errors, intercepting high-risk actions for human approval, and logging the entire process.

In the language of Section 3.4, the harness is that "sole active initiator" — the backend. An agent doesn't change the communication diagram; it just has the backend run some of those steps repeatedly.

The term is borrowed from software testing (a "test harness" is scaffolding that runs code under controlled conditions). It deserves its own name because the quality of the harness matters as much as the quality of the model: the same model with a different harness can see task success rates differ by more than double. A significant proportion of enterprise agent projects that fail trace the root cause to harness design, not model capability. Anthropic has a nice way of putting it — every component in the harness encodes an assumption about "what the model can't do on its own"; once the model improves at that thing, the corresponding component should be removed.

Function Calling (Tool Use)

The model does not actually execute anything. All it can do is output a structured piece of text meaning "I'd like to use the 'search product documentation' tool, with the argument 'legacy data migration'" — and then the harness carries out the execution and relays the result back.

In other words, the model can only express intent; it can't even initiate an action on its own — this is the flip side of the rule from Section 3.4: the initiative always stays with the backend. (This capability is now more commonly called "tool use" or "tool calling"; "function calling" is the earlier name.)

MCP (Model Context Protocol)

An open standard that specifies the interface tools and data sources should use to connect with an agent. Before it existed, each new tool integration required writing bespoke adapter code; with a unified standard, it's like going from "every appliance with its own proprietary plug" to a USB port.

Two costs are worth knowing about. First, context cost: MCP tool descriptions are resident in the context; connect too many servers and a significant chunk of the context window is consumed before the conversation even starts (the industry currently mitigates this with "load tools on demand"). Second, security: the MCP specification itself contains no authentication or authorization mechanisms; as of the first half of 2026, dozens of related vulnerability reports have been filed, and any third-party MCP server should be audited before connecting.

Skill

A pre-packaged workflow description: a document spelling out how a certain type of task should be done — what phrasing to use, which steps to follow, what counts as done — which the agent follows when needed. In practice it's just a folder containing an instruction file, optionally with templates and reference materials.

The key design is called progressive disclosure: normally only the skill's name and a one-sentence description are loaded; only when a task matches does the full content get read into context. So you can install dozens of skills without blowing up the context window.

Skills and MCP are often confused, but they're actually complementary: MCP solves "what it can reach" — connecting the agent to external systems; skills solve "how it should work" — teaching it a set of procedures. A skill doesn't connect to anything or execute any calls; it's simply a written set of instructions. Most production-grade agents need both.

Memory

The model has no memory — Chapter One said as much when explaining Loop C: multi-turn conversation works by stuffing the history back into the context. Agents face the same problem, and it's worse: a task that runs for dozens of rounds produces thinking, tool calls, and return results that quickly exceed what the context window can hold.

So agent memory comes in two tiers:

• Short-term memory: what's currently in the context window — the entire process of this task up to the present moment. It has a hard capacity limit, and the harness must continually decide what to keep, what to compress, and what to drop.
• Long-term memory: an external store outside the context that persists past experiences, conclusions, and user preferences, retrieving them when needed. Its implementation is essentially RAG — except the retrieval target isn't company documents, but the agent's own history.

Memory went from being essentially a synonym for "context window" around 2024 to being recognized as a core architectural layer on par with reasoning, orchestration, and tools — no longer optional.

Agentic RAG: Handing Retrieval Decisions to the Model

Agentic RAG is the third generation of RAG mentioned in Chapter One.

Its approach applies this chapter's mechanisms to the retrieval step: retrieval is no longer a fixed step three in a pipeline, but becomes a tool the agent can invoke on demand. The model judges for itself whether the material gathered so far is sufficient to support an answer; if not, it adjusts its angle and retrieves another round, repeating until satisfied — this is exactly Loop A from Chapter One.

What must be emphasized: what changes is not the components, but who's in command. The vector database, embedding service, reranking service, and generation model are all unchanged; the communication patterns between the backend and these components also remain the same. The sole difference is that "how many retrieval rounds to run, and how to retrieve" shifts from preset, hard-coded rules to the model's real-time judgment during execution.

The cost of this is reduced predictability. Under fixed rules, how many steps a Q&A exchange will take, how many model calls it requires, and how much it will cost are all known in advance. Once the model decides on the fly, these depend on its judgment in the moment, and the model's output is inherently stochastic — the same question might take one retrieval round this time and four the next. Cost, latency, and even the answer itself go from being fixed values to being a range.

Costs and Risks

Context snowballs. With each loop iteration, the harness must resend all preceding thoughts, tool calls, and return results to the model — the input keeps growing; Section 3.3 explained that prefill cost grows quadratically with input length, so this part of the cost rises faster than the number of rounds.

However, this is exactly the scenario where prompt caching shines — each round of an agent's context is the previous round with content appended at the end; the long prefix at the beginning is byte-for-byte identical and can directly reuse previously computed results. Major providers charge roughly one-tenth of the full price for cache-hit portions. This produces a design rule that runs counter to intuition: place stable content (system instructions, tool descriptions, reference material) verbatim at the beginning, and append all changing content to the end. Caching requires the prefix to be byte-for-byte identical; change the tool ordering or insert a timestamp in the middle, and the cache misses. It's also not a silver bullet: what gets discounted is only the input portion; the thoughts and tool calls generated each round are output, still billed at full price; the cache itself has an expiration window; and if the harness compresses context mid-task, the prefix changes and the cache is invalidated.

It may not stop. The model's ability to judge "the task is complete" is unreliable — it might bounce back and forth between two tools, or fall into an infinite loop. So the harness must enforce hard limits: a maximum number of rounds, a maximum spend, and what to do on timeout.

Tool failures cascade. If a tool returns an error or anomalous data, the model may continue reasoning on that basis and veer further and further off course. Production systems need to validate inputs before execution, retry on transient failures, and explicitly tell the model about failures rather than silently skipping them.

High-stakes actions need human approval. The most fundamental difference between an agent and a Q&A system is that an agent actually takes action — sending emails, modifying data, placing orders. For actions with irreversible consequences, the standard practice is to set up human-approval checkpoints and maintain complete operational audit trails. This should be treated as a first-class component at the architecture design stage, not patched in after an incident. Chapter Five revisits this from a compliance perspective.

Agents amplify the damage of prompt injection. If a Q&A system is injected, the worst outcome is a wrong answer; if an agent is injected, it may actually carry out the attacker's desired operation. Chapter Six covers this risk in detail.