What is inference in AI?
Inference is when a trained AI model is used to produce an answer, as opposed to when it is being trained.
Training happens once, or at long intervals, and needs enormous resources. Inference happens every time someone asks a question.
Every answer needs computing power. Where it sits decides where your questions are processed. With a public AI service it happens on the supplier's servers, with your own installation it happens at your site.
How fast the answer comes and what it costs depend mainly on inference. That applies to cloud services and your own operations alike.
How it works
During inference the model reads the question and the material and works out the answer one token at a time. Each new token requires a calculation through the whole model. That is why long answers take longer, and why larger models need more memory and faster hardware.
Take a case officer asking about the rules for a particular grant. The system fetches the right section of the guidelines, the model reads it and writes the answer. All of that is inference, and it happens while the officer waits.
Why it matters
Inference is what decides how AI behaves day to day. Response time, cost and where the data is processed are all settled here. For an organisation with strict confidentiality needs, it is often decisive that inference can happen in its own environment, with local AI that can run fully offline.
What to keep in mind
Cost grows with use. A pilot with a handful of users says little about the picture once everyone is on the service. Plan for a realistic volume of questions and documents, and ask how response times hold up when many people ask at once.
What it means for you
Ask where inference happens. That is where your questions and documents are actually processed.
Common questions
Why does inference matter to us?
Inference happens every time someone asks a question. Where it runs decides where the question and the material are processed.
Does inference need powerful hardware?
Less than training. Large models often need GPUs to answer quickly, while smaller models can run on simpler servers.
Related terms
Read more
All terms Updated