A 200,000-token context window is not 200,000 tokens of usable attention, and treating it as free storage is the most expensive habit in production LLM work. Context engineering is the practice of deciding what enters the context window, in what order, and what gets left out. The window is a budget, not a bucket. A token spent on a retrieved document is one unavailable to conversation history.
Context engineering treats the context window as a fixed allocation with four competing line items: system instructions, tool definitions, retrieved evidence, and conversation history. Each is billed per token on every call. Spend where the model needs grounding, cut where it does not, and put static content first so the provider’s prompt cache pays for it once.
Why does adding more context make answers worse?
Attention is not uniform across the window. Liu and colleagues measured this in Lost in the Middle (TACL, 2024): accuracy on multi-document question answering peaked when the relevant passage sat at the beginning or the end of the input and degraded sharply in the middle. The curve is U-shaped, and it holds even for long-context models.
An irrelevant document does not cost nothing: it pushes the relevant one further from an edge. Raising retrieval depth from ten chunks to twenty buries the good chunk deeper. Start with tokens and the context window if that mechanic is unfamiliar.
What each line item actually buys
| Line item | What it buys | What it costs |
|---|---|---|
| System and tools static |
Consistent behaviour and tool arguments. | Fixed overhead on every call, trivial ones included. |
| Retrieved chunks variable |
Grounding in facts absent from training. | The largest expense, and the most often wasted. |
| Conversation history growing |
Continuity and pronoun resolution. | Grows without limit. Turn 3 rarely earns turn 40. |
| Output reservation reserved |
Room to finish the answer. | Comes out of the input allowance. Truncation is silent. |
Ordering matters more than most teams realise. OpenAI’s prompt caching guide is explicit that cache hits require an exact prefix match, so static instructions belong at the front and user-specific content at the end. Reverse them and the cache misses every request: same tokens, same answer, larger bill. That is where prompt design ends and context engineering begins.
What does context engineering actually cost?
The objection deserves stating. Summarisers and relevance filters are failure surfaces themselves, and a summariser that drops the clause the user asked about is worse than a bloated prompt that at least contained the answer. Pruning trades a diffuse problem for a sharp one that hides better: text you never sent never appears in a log.
So the discipline is not “send less” but “know what you sent, and why”. Count tokens per line item before tuning anything. tiktoken does it locally in three lines, and the usage field on every response reports what came from cache. Agents raise the stakes, because short-term and long-term memory retention decides what history survives.
Where the budget should land
Start from the output, not the input. Reserve what the answer needs, subtract the static prefix you cannot remove, and treat the remainder as the pot retrieval and history compete over. A defensible default: half of it to retrieved evidence, a quarter to a rolling conversation summary, a quarter left empty. Empty headroom is not waste. It is why a long tool result does not quietly push your system prompt out, a failure examined in avoiding context overload.
Rank retrieved chunks by relevance, then place the strongest at both ends of the retrieved block rather than in descending order. The U-shaped curve is a constraint you can exploit.
Reordering costs nothing, and very few pipelines bother.
- Context engineering allocates a fixed token budget across instructions, tool schemas, retrieved evidence, and history.
- Accuracy on long inputs follows a U-shaped curve, so a passage buried mid-context is used less reliably than one at an edge.
- Prompt caching pays out only on an exact prefix match, making static-first ordering a billing decision, not a style preference.
- Reserve output tokens before input tokens, because truncation fails quietly and still returns a fluent answer.
- Measure spend per line item first: what feels expensive rarely is.
Conclusion
Set a ceiling per line item, log the real spend, and review it like a query plan. Most of that pressure originates upstream in retrieval, so the next step worth taking is how a RAG pipeline decides what to send: chunk size and reranking settle your context bill first.
Frequently Asked Questions
What is context engineering?
Context engineering is the practice of deciding what information enters a model’s context window, in what order, and what gets excluded. It covers system instructions, tool definitions, retrieved documents, and conversation history. The goal is allocating a fixed token budget so the model receives what it needs without diluting its attention.
What is the difference between prompt engineering and context engineering?
Prompt engineering shapes the instruction you write. Context engineering shapes everything else the model reads alongside it: retrieved passages, tool schemas, prior turns, and their ordering. Prompt engineering is a single-message concern. Context engineering is a systems concern spanning retrieval, memory, caching, and token accounting across an entire application.
Does a larger context window remove the need to manage context?
No. Research on long-context models shows accuracy varies with where relevant information sits, peaking at the beginning and end of the input and dropping in the middle. Larger windows also cost more per call and add latency. Capacity increases what you can send, not what the model reliably uses.
How do I count tokens before sending a request?
Use the tokenizer matching your model. The tiktoken library encodes text locally and returns exact counts without an API call, so each part of a prompt can be measured separately. Most provider APIs also return input, output, and cached token counts in the usage field of every response.
Where should static instructions go in a prompt?
At the very beginning. Prompt caching systems match on exact prefixes, so content that changes between requests must sit after content that stays identical. Putting a user question ahead of a long system prompt breaks the prefix match and forces the provider to reprocess and rebill the whole prefix.
How much of the context window should be left empty?
Reserve enough for the full output plus a margin for unexpectedly large tool results, commonly around a quarter of the usable window. Input truncation fails silently, dropping whichever tokens sit at the boundary, and the model still returns a fluent answer built from an incomplete prompt.