What is a token?
A token is the smallest piece of text a language model reads and writes, often a short word or part of a longer word.
Language models split text into smaller pieces called tokens. A common word can be one token, while a long or unusual word is split into several.
Tokens are used to measure how much text a model handles. How much it can read at once is given in tokens, and many public AI services charge per token.
Swedish words are often long compounds. They can therefore use more tokens than the equivalent English text.
How it works
The split follows a fixed vocabulary that was settled when the model was built. Common words and word pieces have their own tokens, while rare words are assembled from several smaller parts. Numbers, punctuation and spaces count too. The model reads and writes in tokens and turns them back into ordinary text on the way out.
Why it matters
In practice tokens affect three things. How long a document the model can read in one go, how long an answer takes, and what it costs when a service charges by use. The price often covers both the text you send and the text that comes back.
What to keep in mind
The same text comes to a different number of tokens in different models, because they split text in different ways. A price per token therefore does not tell the whole story. Try typical documents from your own work and see what it actually costs.
A long conversation gets slower and more expensive with every question, because earlier questions and answers are often sent again. Starting a new conversation when the topic changes can give faster and better answers.
What it means for you
Ask how the price is calculated if a service charges for usage. Long documents can make a big difference.
Common questions
Why is text counted in tokens?
A language model splits text into smaller pieces that it computes with. How much it can read at once and what usage costs are both often measured in tokens.
How much text is a token?
It varies. Short common words are often a single token, while long or rare words are split into several. Languages other than English often need more tokens for the same content.
Do tokens affect what we pay?
Often, for services that charge by usage. With your own installation, cost is driven by running the environment instead.
Related terms
Read more
All terms Updated