🎯 About this course
Multimodal AI is easy to demo and hard to trust. This course teaches the engineering behind reliable systems: how contrastive and generative vision-language models work, how to build and evaluate image–text search, how to prompt VLMs and probe for hallucination, and how to build document pipelines that combine OCR, extraction and arithmetic validation.
You will implement the contrastive loss, a multimodal index with filters, VQA and probe metrics, OCR error rates, an extractor with cross-field validation, a slice-and-interval evaluation harness and a cost-aware cascade. Labs are lightweight NumPy and pure-Python exercises that run offline; the applied studio uses synthetic receipts and openly licensed photos.
👤Who it is for
Engineers and data scientists adding image and document understanding to products.
✨What makes it different
Every technique comes with its measurement, its failure modes and its cost.
By the end you will be able to
- Distinguish contrastive dual encoders from generative VLMs, explain how images become tokens and estimate image token cost.
- Build and evaluate a multimodal retrieval index with metadata filtering, duplicate handling and recall/MRR metrics.
- Prompt and evaluate VLMs for captioning and VQA, measuring and mitigating hallucination.
- Design document-understanding pipelines with OCR, extraction, cross-field validation and human review, and measure CER and STP rate.
- Evaluate multimodal systems on representative data with slices, confidence intervals, robustness tests and bias checks.
- Deploy multimodal systems with cascades, caching, cost models and privacy and prompt-injection safeguards.
- Deliver a multimodal assistant with a measured quality, cost and governance case documented in a model card.