What is multimodal AI?
Multimodal AI can handle several kinds of content, such as text, images and speech, and connect them in the same task.
Early language models worked only with text. A multimodal model can also interpret an image, read a scanned document or understand speech.
That opens up tasks that used to need several tools. A technician photographs a rating plate and asks which manual applies. A doctor dictates and gets a structured note to review.
Images and audio can contain information as sensitive as text. The same questions about where it is processed apply here.
How it works
A multimodal model turns different kinds of content into the same sort of internal representation. An image is split into small patches, sound into short segments, and both are handled much like text. That is why the model can answer a question about an image or summarise a conversation with the same technology it uses for text.
Why it matters
A lot of knowledge lives outside text. Drawings, photographs, scanned forms and conversations. Multimodal AI makes it possible to search and use that material, and to work by voice when your hands are busy.
A property manager photographs a water leak and asks what to do. The assistant interprets the image and answers from the company's own instructions.
What to keep in mind
Interpreting images and sound can be less reliable than text, especially with poor quality, handwriting or a lot of background noise. Review the output more closely at first.
Check, too, that the same permissions and storage rules apply to images and audio as to documents.
What it means for you
See where speech and images could save time in your work, and check that they are handled as securely as text.
Common questions
What can multimodal AI be used for at work?
For example, reading a scanned document, interpreting an image in a manual or turning speech into text and summarising it.
Does the same data protection apply to images and audio?
Yes, the same rules apply. Personal data is harder to spot in advance, though, such as faces in an image or voices in a recording.
Related terms
Read more
All terms Updated