Every request to a language model is assembled. Something decides that this system prompt, these tool definitions, those four retrieved passages and the last eleven turns of conversation are what the model sees, and that everything else in the world is not. Context engineering is the name for making that decision deliberately instead of letting it fall out of whatever your framework does by default.
Anthropic frames it as the successor to prompt engineering: prompt engineering asks how to word an instruction, while context engineering asks “what configuration of context is most likely to generate our model’s desired behavior” across system instructions, tools, external data and message history. That question is answered by code running on every request, not by a string someone edited once.
Assembly is a technique in its own right. The cost of what you assemble is separate arithmetic. The two pull the same way often enough that people conflate them, but a prompt can be cheap and still be wrong, and the fix for a wrong prompt is sometimes to spend more.
The window is a budget with competing claimants
A fixed number of tokens is available per request, and several independent parts of your system want them. None of those parts knows about the others.
| Claimant | Who controls it | How it grows |
|---|---|---|
| System prompt | You, at design time | By accretion, as instructions are added to fix bugs |
| Tool and function definitions | The tool registry | With every tool anyone adds, whether relevant or not |
| Retrieved documents | The retriever, at query time | With top-k and chunk size, both usually constants |
| Conversation or agent history | The loop | Monotonically, one turn or one tool call at a time |
| Output space | Shared with the input | Inversely: input crowds out room to answer |
The last row is the one most often missed. Input and output share the window on most APIs, so a prompt consuming nearly all of it leaves a model that cannot finish a long answer. Truncated JSON from a structured-output call is frequently this, not a formatting failure.
The claimants also differ in a property that matters more than size: whether anything bounds them. The system prompt is bounded because a human wrote it, and retrieved documents by top-k. Tool results and history are bounded by nothing unless you write the code that bounds them, and uncontrolled inputs are where context problems come from.
For each block in the prompt you should be able to state how many tokens it is, on what fraction of requests it appears, and what decision it is supposed to change. A block that cannot answer the third question is not context, it is residue.
More context is not better context
The intuition that a larger window is strictly better rests on the assumption that irrelevant tokens are inert. They are not.
Position effects are measured, not folklore. Liu et al. tested multi-document question answering and key-value retrieval while varying only where the relevant document sat in the input. Accuracy was highest when the relevant information was at the beginning or the end of the context and “significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models”. The content was identical across conditions. Only the position changed.
Irrelevant text degrades answers directly. Shi et al. built GSM-IC, grade-school arithmetic problems with irrelevant sentences added to the problem statement, and found model accuracy dramatically decreased when irrelevant information was included. They also found partial mitigations, including explicitly instructing the model to ignore irrelevant content and decoding with self-consistency, which tells you the effect is a distraction failure rather than a capacity failure.
Plausible-but-wrong text is worse than unrelated text. The retrieval result that should change how you configure top-k comes from Cuconasu et al., who found that a retriever’s highest-scoring documents which are not actually relevant to the query negatively impact effectiveness, while adding clearly random documents did not hurt and in their setting helped. Near-miss passages are the expensive kind of noise, and near-miss passages are exactly what a semantic retriever produces when the corpus does not contain the answer.
Anthropic describes the same phenomenon as an “attention budget” that every additional token draws down, and reports it as a performance gradient with no cliff at a particular length. So there is no threshold to stay under, just a cost per token of attention you are paying whether the token earns it or not. That reverses the default instinct. When answers get worse, try removing input before adding it.
Compaction, and what it costs you
An agent loop or a long conversation will exceed any window eventually, because the history grows with every step and nothing removes anything. Compaction is the practice of summarising the conversation as it approaches the limit and restarting the window from the summary.
Anthropic describes its Claude Code implementation as passing the message history back to the model to compress, preserving architectural decisions, unresolved bugs and implementation details while discarding redundant tool output, then continuing with that summary plus the most recently accessed files (Anthropic Engineering, 2025). The useful part of that is that it names what is kept, not just that it summarises.
Three strategies exist and they are not interchangeable.
| Strategy | Keeps | Loses | Suits |
|---|---|---|---|
| Compaction (summarise and restart) | Narrative continuity | Specifics whose relevance appears later | Long back-and-forth work |
| Structured notes outside the window | Whatever the agent chose to write down | Anything it did not think to record | Iterative work with milestones |
| Sub-agents with clean windows | A distilled result per task | The reasoning that produced it | Parallel search and analysis |
Anthropic’s advice on tuning a compaction prompt runs counter to how prompts are usually tuned: maximise recall so the summary captures everything relevant from the trace, then iterate on precision to remove the superfluous. Tuning for brevity produces a summary that reads well and has silently dropped the constraint that mattered.
The failure mode is specific. After compaction the model cannot tell you what was lost, because from its perspective nothing was: the summary is the history. You observe it as an agent repeating work it already did, or reopening a decision settled fifty steps ago. Log every compaction event with the pre-compaction token count and the summary itself. Without the summaries this class of failure is not debuggable at all.
Tool results are the uncontrolled input
In an agentic system, tool output is typically the bulkiest thing in the window and the only thing nobody sized. A query returns however many rows it returns. An HTTP fetch returns however large the page is. A file read returns the file. And because the loop appends results to the history, each one is re-sent on every subsequent step until something removes it.
Anthropic treats tools as a context design surface, not an integration detail: definitions should be self-contained and unambiguous, and the common failure is a bloated tool set where even a human engineer could not say which tool applies. If a person cannot definitively choose, an agent cannot be expected to do better, which is a workable test for your own registry.
What to do about the results themselves, in rough order of how much they help:
- Truncate at the tool boundary, and say so. Return the first n rows with an explicit marker that the result was cut and how to get the rest. A silent truncation produces an agent that reasons confidently over a partial answer.
- Return references instead of payloads. Anthropic calls this just-in-time context: the agent holds file paths, stored queries or links, and pulls the content in only when it needs it. Claude Code is described as combining this with a small amount of up-front context, using shell primitives to inspect large data without loading it. The trade is latency and the need for the agent to navigate well, which is not free.
- Clear stale results. Once a tool result is deep in the history and has been acted on, the raw output rarely needs to stay. Anthropic describes tool result clearing as one of the lightest forms of compaction and has shipped it as a platform feature.
- Shape the result for a reader that pays per token. Drop null fields, drop identifiers nothing will dereference, and do not return a formatted table when a list of values will do.
There is a security dimension the engineering framing tends to skip. Tool results and retrieved documents are untrusted input arriving in the same token stream as your instructions, and the model draws no architectural line between the two. See prompt injection; the point here is that context assembly decides what untrusted text reaches the model, which makes it a security boundary whether you designed it as one or not.
Retrieval: precision over recall, once the budget binds
Classic information retrieval optimises recall because a human will scan the result list and pick. Retrieval into a context window has no such reader. Every passage returned is consumed at full weight, so a marginal irrelevant passage does not cost a glance, it costs answer quality per the papers above. That inverts the usual tuning target:
| Knob | Recall instinct | What a bounded context wants |
|---|---|---|
| top-k | Raise it, cheap insurance | Lower it, and make it dynamic rather than constant |
| Score threshold | Rarely used | Used, so a query with no good match returns nothing |
| Reranking | Optional polish | Load-bearing, because it is what makes a small k viable |
| Chunk size | Large, for context | Sized to the answer unit, so a chunk is mostly signal |
| Empty result | A bug | A valid answer, surfaced as “not in the corpus” |
The threshold row is the one that most often changes behaviour. A retriever configured to always return five chunks will return five chunks for a question the corpus cannot answer, and those five will be the highest-scoring irrelevant passages, which is the case Cuconasu et al. identified as harmful. Returning nothing and letting the model say it does not know is usually the better outcome. That needs a threshold you calibrated, not a constant k.
Ordering is the second lever, and it follows directly from the position effect: put the material you most need attended to at the start or the end of the retrieved block, not buried in the middle of a long concatenation. Note that this can conflict with prefix caching, which wants stable content first and volatile content last for cost reasons. When the two conflict, they conflict. Decide which one you are optimising instead of assuming a layout satisfies both.
Where retrieval runs matters less than whether these knobs are reachable. A vector index inside the transactional database, such as pgvector in PostgreSQL, lets a similarity search be filtered by ordinary SQL predicates in the same query, so tenant scoping, permissions and recency filters apply before ranking rather than after. That is a precision mechanism: the cheapest way to keep irrelevant passages out of the window is to make them unretrievable for that user. Broader treatment of the pipeline is on the retrieval-augmented generation page.
How you would know this is going wrong
Context problems are hard to see because the artefact that caused the failure is not stored anywhere by default. The framework built the prompt, sent it, and discarded it. Minimum instrumentation:
- Log a structured summary of the assembled context per request. Token count per block: system, tools, retrieved, history, response. Without the breakdown you cannot tell a retrieval problem from a history problem.
- Record which chunks were retrieved and their scores. When an answer is wrong, the question is whether the evidence was present, and that is unanswerable after the fact otherwise.
- Alert on distributions, not means. The interesting requests are the long tail: sessions that compacted three times, the tool call that returned a large payload.
- Evaluate at realistic context lengths. Every effect on this page is length-dependent, so a prompt validated on short inputs tells you nothing about production.
Three symptoms map to causes reliably enough to memorise. Answers that degrade as a session gets longer point at history growth or compaction loss. Answers that cite something adjacent to the right thing point at retrieval precision. Answers truncated mid structure point at output space crowded out by input.
Where the discipline stops
Context engineering decides what the model sees, not what the model can do. If the information is not in the corpus, no assembly will retrieve it. If the model cannot perform the task on a clean, minimal prompt, better assembly will not fix that either, and OpenAI’s guidance is to pin production to specific model snapshots and build evaluation suites that measure behaviour when anything changes (OpenAI). Whether retrieval is the right lever at all is a separate decision, on retrieval or fine-tuning.
This is a young discipline with a few well-supported findings and a lot of untested practice. Position effects, distraction by irrelevant text and the harm from near-miss retrieved passages are measured in published work. That compaction suits your workload better than note-taking, or that your prompt ordering is optimal, is established by nothing except your own evaluation.
Related
- Context window economics - what the assembled context costs per request
- Retrieval-augmented generation - the retrieval side in full
- pgvector - filtered similarity search in PostgreSQL
- Prompt engineering - wording the instruction, once the context is decided
- Retrieval or fine-tuning - when assembly is the wrong lever
- Prompt injection - the untrusted text you just assembled
What to watch
Specific things that would change what this page concludes.
- Whether vendor-side tool-result clearing and memory tools remove the need for application-level compaction, or just move it
- Whether newer long-context models flatten the position effect that Liu et al. measured, on published evaluations rather than vendor claims