Why Your 128K Context Window Isn't: The Lost-in-Middle Problem and How to Measure What You Actually Have
The gap between advertised and usable context is wider than most teams realize
Your language model vendor claims 128,000 tokens. But what you're actually getting—the information the model reliably uses—is often a fraction of that number. The difference isn't a rounding error. It's the difference between a system that works and a system that confidently produces wrong answers while insisting it has all the facts.
This isn't a new problem, but it's one that organizations still struggle to account for when budgeting, evaluating models, or building retrieval-augmented generation (RAG) systems. The architecture works against you, not because the models are broken, but because of how attention mechanisms behave under load.
The lost-in-the-middle phenomenon: what the research actually shows
LLMs exhibit a tendency to use information at the beginning and end of a context window more reliably than information placed in the middle . This isn't subtle degradation. It's measurable, reproducible, and present across the frontier models you're probably evaluating right now.
The effect mirrors a pattern from human memory research: people preferentially recall items from the beginning (primacy) and end (recency) of a study list with higher accuracy, producing a characteristic U-shaped curve . The parallel is not metaphorical— evidence points to the role of LLMs' intrinsic attention biases and architectural biases as the mechanism driving the behavior.
What does this mean in practice? Bigger context windows do not ensure perfect recall—they give room for more context, but they do not give equal attention to every part of that context . A model with 128K tokens can still miss a critical fact positioned at token 64,000 while correctly processing the same fact if it appeared at token 500 or token 127,500.
Where the performance cliff actually appears
Context rot is the measurable performance degradation LLMs experience as input length increases, with Chroma testing 18 frontier models and finding every one gets worse . But the deterioration isn't uniform across all tasks or all models.
Recent systematic experiments across 5 open- and closed-source LLMs on math, question answering, and coding tasks reveal that, even when models can perfectly retrieve all relevant information, their performance still degrades substantially (13.9%–85%) as input length increases but remains well within the models' claimed lengths .
This is the critical distinction: the model has not run out of tokens. It has not truncated your input. It is still processing everything you sent. And yet its reasoning quality declines measurably as the context grows longer. Teams routinely assume their context window is "big enough." They pick a model with 128K or 1M tokens, load it up, and wonder why output quality degrades. Context rot explains why: capacity is the wrong metric .
The mechanism: why this happens
Models often rely more on the beginning and end of a long context window to understand the task and shape the answer . The architectural culprit is positional biases like RoPE decay —the mathematical encoding that tells the model where information sits in the sequence. As sequence length grows, the model's attention patterns become increasingly concentrated at the boundaries.
The information in the middle of the context window competes with irrelevant text, repeated phrases, and distractors . Even in carefully constructed scenarios with minimal noise, the sheer weight of processing a long sequence pulls the model's focus away from the middle section.
There's a secondary layer: context rot is performance degradation that happens gradually as input context grows longer—more tokens in, worse output out, even when the model's context window isn't close to full . This is not about hitting a hard limit. It's about signal-to-noise ratio degrading as you add tokens, regardless of whether they're relevant.
How to measure actual usable context: a framework
Stop treating your model's advertised context window as a guarantee. Start measuring. Here's what a practical evaluation looks like:
1. Test at different positions, not just at capacity
Embed a unique fact in documents of varying lengths, from 750 words to 96 thousand words, and place that fact at different positions to measure how placement affected retrieval accuracy . This replicates the real-world problem: you don't care whether the model can handle 128K tokens in theory. You care whether it can reliably extract information from position 40K when the full context is 100K.
2. Measure retrieval separately from reasoning
Extending the input length alone substantially degrades LLM reasoning capability, even if the model is still able to retrieve the relevant evidence—inserting white spaces with minimal distraction does not prevent the model from extracting all conditions correctly, but nevertheless causes it to reach the wrong answer . Your evaluation needs to distinguish between "the model found the right information" and "the model used it correctly in reasoning." These are not the same.
3. Test with realistic noise, not clean scenarios
Production fixes include two-stage retrieval (broad recall + cross-encoder reranking), hybrid search (semantic + BM25), and strategic ordering (top evidence at start and end) . But before you implement those fixes, measure how your model performs with typical distractors—redundant chunks, outdated information, competing claims.
4. Benchmark ordering strategies
The most effective approach positions the highest-ranked documents at the beginning and end of the context window, with lower-ranked documents in the middle, leveraging the model's inherent attention biases rather than fighting against them . This is not complex. It's a simple reordering of your retrieval results. But it works.
Practical implications for architecture decisions
The lost-in-the-middle problem gets worse when teams send too much unfiltered context into the model and hope the LLM will sort it out. The better fix starts before prompt assembly: decide which context is trusted, current, relevant, and specific enough to enter the context window, removing duplicate chunks, stale definitions, weak evidence, and loosely related policies .
This reframes the decision. You're not choosing between a 32K and 128K model. You're choosing between different trade-offs in context management overhead. A 32K model with disciplined context engineering and strategic reordering may outperform a 128K model loaded with noisy context.
The operational cost difference is significant. Overloading the context increases costs. LLMs are stateless, which means they do not have memory of past conversations—for every message you send, the entire conversation history must be sent back to the model . Every extra token you shove into the context window, whether relevant or not, is a token you're paying for and latency you're incurring.
What this means for your team
Capacity is the wrong metric. Signal-to-noise ratio is what determines output quality . When evaluating a model or building a RAG pipeline, treat the advertised context window as a ceiling, not a target. Measure the effective context—the portion of tokens where accuracy remains acceptable for your use case—and design your system around that number.
Test position sensitivity. Rank your retrieval results strategically. Filter ruthlessly. And most importantly: don't assume that a model with a larger context window solves the problem. It often just makes it bigger.
| Metric | What It Measures | Why It Matters |
|---|---|---|
| Advertised context window | Maximum token capacity (e.g., 128K) | Upper bound only; tells you nothing about usable performance |
| Effective context length | Tokens where accuracy remains above threshold (e.g., 85%) | Real operating window for your use case |
| Position sensitivity | Accuracy variance based on information placement | Reveals U-shaped degradation; informs retrieval ordering strategy |
| Retrieval accuracy with context bloat | Whether model extracts correct information amid noise | Isolates retrieval from reasoning; shows if problem is architecture or preprocessing |
| Signal-to-noise ratio | Ratio of relevant to irrelevant tokens in context | More predictive of output quality than raw context size |