What is a context window?
The context window is the amount of text a language model can take into account at once, measured in tokens.
Everything the model will use for an answer has to fit in the context window. That includes the question, earlier messages in the conversation and the documents it is given to read.
A large context window makes it possible to read long texts. It does not mean your whole document collection fits. An organisation usually has far more documents than there is room for.
That is why systems that answer from your own documents pick out the passages that match the question. It keeps answers focused and makes it clear which source they build on.
How it works
The context window is filled from several sources. The background instruction, the conversation so far, the retrieved documents and the question itself. The answer the model writes counts as well. When the window is full something has to be dropped, and the service decides what.
Why it matters
Its size decides what can be done in a single step. Comparing two long contracts needs room for both. Answering a question from a whole document library instead needs the right passages to be picked out. For an organisation, the selection is usually what decides quality.
What to keep in mind
Models do not always weigh all the text equally well. Something in the middle of a very long input can get less attention than what sits at the start or the end.
A larger window also makes each question slower and more expensive. Fill it with what belongs to the question.
What it means for you
Ask how an AI service chooses what to read when you have many documents. The answer says a lot about how accurate it will be.
Common questions
What happens when the text does not fit?
Whatever does not fit, the model cannot take into account. In long conversations, early parts can drop out, and a long document may need to be split.
Do we need a large context window to search all our documents?
No. With RAG, only the passages needed for the question are retrieved. The context window just has to hold those.
Do answers get better with more text?
Not always. Text that is unrelated to the question can make the answer worse. A few relevant passages usually work better.
Related terms
Read more
All terms Updated