What is a transformer in AI?
A transformer is a type of neural network that weighs how all the words in a text relate to each other, and it underlies most modern language models.
The technique was first described by researchers in 2017. It builds on a mechanism called attention. For each word the model judges which other words in the text matter most to understand it.
Earlier models read text word by word in order and easily lost the thread in long sentences. A transformer sees the whole text at once.
That made it possible to train much larger models on much more text. It is a large part of why language models became so much better.
How it works
The text is first split into tokens, and each token is given a long row of numbers that describes its meaning. In every layer the model works out how much each token should be influenced by the others, in parallel across the whole text. After many layers, every word carries its context with it.
Why it matters
The transformer explains why language models understand a question phrased in a new way and can hold a long line of reasoning together. The same design is used for images and sound, which has made multimodal AI possible.
What to keep in mind
Attention costs more the longer the text is. That is one reason the context window has a limit and why long inputs make answers slower.
A transformer does not know what is true either. It is very good at patterns in text, and fact-checking has to come from the source material.
What it means for you
You do not need to know the technique to use it. What matters is what the model gets to read and how the answers can be checked.
Common questions
Why does the transformer matter so much?
It made it possible to train models efficiently on very large amounts of text. Most modern language models are built on it.
Where does the technology come from?
It was first described in a research paper from 2017. Since then it has become the foundation of generative AI.
Do we need to understand the technology to use it?
No. For an organisation, what matters more is where the model runs, what material it answers from and how answers can be checked.
Sources: Vaswani et al. 2017 on arXiv
Related terms
Read more
All terms Updated