What multimodal means, and the plan
How models handle images, audio, video and documents by turning them into tokens or vectors, what that costs, the route through vision-language models, speech, embeddings, diffusion and real-time voice, and what you will be able to do.
#The idea
A language model works on sequences of vectors. Multimodal models extend that: pictures, sound and video are turned into vectors too, so one model can read an image, answer a question about it, transcribe speech or write a reply that is spoken aloud.
- 1Image, audio or videoraw numbers
- 2Encoderpatches or frames become vectors
- 3Language modelattends over text and media together
- 4Outputtext, or a decoder produces audio or images
Two broad families:
| Family | What it does | Examples |
|---|---|---|
| Understanding models | Take media in, produce text | Vision-language models, speech recognition |
| Generation models | Produce media | Text to speech, image and video diffusion |
#Why it needs its own course
Media changes the cost model. A page of text is a few hundred tokens; an image is hundreds to thousands, audio is thousands of frames per minute, and video multiplies that. Latency, memory and pricing all follow. Serving also differs: encoders, diffusion samplers and streaming audio each have their own bottlenecks.
#The route
| Part | Question | Time |
|---|---|---|
| Start here (3 lessons) | How do images and audio become numbers? | about 1.5 hours |
| How vision-language models work | What do images cost, and how are they read? | 1 hour |
| Audio: ASR and TTS | How do speech in and speech out work? | 1 hour |
| Embeddings and rerankers at scale | How do I search images and text together? | 1 hour |
| Image and video diffusion, few-step generation | How are images made, and how fast can they be? | 2 hours |
| Real-time voice pipelines | How do I build a low-latency voice assistant? | 1.5 hours |
| Capstone | A multimodal assistant of your own | about 5 hours |
#What you will be able to do
- Estimate the token and memory cost of images, audio and video.
- Choose and serve a vision-language model, and read its limits honestly.
- Build a speech-in, speech-out pipeline and find where its latency goes.
- Explain diffusion, and what few-step methods change.
- Design a document or image pipeline with OCR, embeddings and reranking.
#Prerequisites
- Local LLM Inference basics: tokens, context, VRAM, quantisation (the course).
- Nothing about images or audio is assumed: the next two lessons cover what you need, with runnable examples.
#Exercise
A vision-language model turns each 14 by 14 pixel patch into one token. How many tokens is a 224 by 224 image, and what happens to a 1,024 by 1,024 image?
Answer: 224 / 14 = 16, so 16 × 16 = 256 tokens. A 1,024 by 1,024 image gives about 73 × 73 = 5,329 patches, over twenty times as many, which is why large images are resized or tiled and why they use so much context.
Next: images as numbers.
Quick quiz
+10 XP per correct answer, +25 XP for a perfect score
1. How does a vision-language model read an image?
2. Why do images cost more than they look?
3. What is a common reason real-time voice assistants feel slow?
Found this useful?
Send it to someone who is learning the same thing.