A context window is a ceiling, not a workspace.
The number tells you what fits. It does not tell you what the model still reasons well over.
Capacity and usable attention are different quantities, and only one is advertised
The advertised window is a hard architectural limit — exceed it and the request is rejected. What it does not describe is how well the model attends across that span, and those two things diverge well before the ceiling. Retrieval accuracy in the middle of a very long context is measurably worse than at either end, which is why a fact buried at 60% depth in a 900k-token prompt can be missed by a model that would find it instantly in a 20k prompt. The practical consequence is that a large window is best used as headroom for not having to chunk, not as an invitation to put everything in. Cost compounds this: you pay input rate on every token every turn, so a conversation carrying 500k tokens of context pays for all of it on each exchange.
- What the number is
- A hard cap on input + output combined. Exceeding it is a request error, not degradation.
- What it is not
- A promise of uniform attention. Mid-context recall degrades before the ceiling.
- Cost behaviour
- Every token is re-billed at input rate on every turn unless it is cached.
- Output shares the budget
- A large max_tokens reduces what is left for input on the same request.
- Practical use
- Headroom to avoid chunking — not a reason to skip retrieval.
- The real limit
- Usually cost or latency, both of which bind long before the token ceiling does.
FAQ
It will fit, and that is not the same as it working well. Attention across very long spans is uneven, so a specific detail in the middle is genuinely easier to miss than the same detail in a short prompt. Retrieval — selecting the relevant few thousand tokens — usually produces better answers than pasting everything, and costs a fraction as much.
Because the API is stateless: every turn re-sends the entire history, and you pay input rate on all of it again. A conversation that has accumulated 100k tokens costs 100k input tokens per message, not per session. Prompt caching or compaction is the answer.
No. It changes how much it can see at once, not how well it reasons. A larger window helps when your task genuinely requires holding a lot at once and does nothing at all for tasks that fit comfortably in a small one.
The request fails with an explicit error rather than silently truncating — which is the desired behaviour, since silent truncation would produce confidently wrong answers with a missing middle. Handle it by compacting history or retrieving less.