Do Long Context Windows Actually Get Used?

What did the researchers set out to measure?

Sellers report the longest input a model accepts, which is not the same as the length over which it reasons well. The authors of a preprint called LongSpan wanted to measure the second thing, which they call the effective context length.

They built synthetic and semi-synthetic documents from 8,000 up to 512,000 tokens. The passage holding the answer was placed at the start, the middle or the end. The semi-synthetic versions came from public-domain text, and the misleading passages were chosen to look like the evidence, so a model could not simply spot the sentence that did not fit. Answers had to cite the supporting passage as well as state the answer.

Six task types covered simple retrieval, multi-hop reasoning and aggregation across several passages. Each condition had 3,600 test items per context length and was run with three prompt phrasings. Fourteen publicly described models took part, and the answers were scored automatically against reference answers.

How much of the window do models actually use?

Across the 14 models, average accuracy fell from 91 percent at 8,000 tokens to 64 percent at 128,000. The decline was gradual, not a cliff, which makes the loss easy to miss in ordinary use.

Position mattered as much as length. When the evidence sat in the middle third of the document, accuracy was on average 14 points lower than when it sat at the start or the end.

The gap between models was wide. The best model lost only 9 points between 8,000 and 128,000 tokens, while the weakest lost 41. But the authors caution that the ranking changed with the task: no single model led everywhere.

They then defined the effective window as the length at which accuracy stays within 10 points of the short-context score. By that measure it was roughly 30 to 40 percent of the advertised window. Task type mattered too. Simple retrieval held up much better than jobs that required combining several facts.

Why does the effective context length matter?

If the usable window is smaller than the advertised one, then loading everything into a prompt can quietly lower quality, and the model still answers fluently, which makes the errors hard to notice.

A fair reading is that long documents deserve a placement decision. Put the material that matters most at the start or the end rather than burying it. Trim what you can. The paper also suggests that retrieval tools, which fetch just the relevant passages, are not made obsolete by long windows; the two approaches look complementary.

The result fits what many people who work with long inputs report anecdotally. What it changes is the default assumption that a larger advertised window means more usable material. What it does not change is that some tasks stayed accurate at long lengths, so the right question is not whether long context works in general but whether it works for the job at hand.

What does the paper not show?

Takeaways

• saysAverage accuracy fell from 91 percent at 8,000 tokens to 64 percent at 128,000 tokens across the 14 models.
• saysEvidence in the middle third of a document was on average 14 points harder for these models to use than evidence at the start or end.
• explainsA context window is the length of text a model can take in at once; accepting a long input is not the same as reasoning well over it.
• interpretsA practical reading is to place the most important material at the start or end, trim what can be trimmed, and test your own task at your own length.
• interpretsRetrieval pipelines and long windows look complementary rather than substitutes, so fetching the relevant passages still has a role.

Ask the paper

More on this