LLMOps & Inference at Scale

Serving frameworks, batching, caching, quantisation, and the cost / latency trade-offs that decide whether an LLM project is viable.

⏰ 6 weeks🎯 Standard🧩 Certificate🖥 Any device · our GPUs

Apply for this course Where this leads

Generated from this course

🎯 About this course

Running LLMs in production is a systems problem. This course explains why: prefill versus decode, the memory cost of the KV cache, and how continuous batching and paged attention changed serving economics. You will quantify memory and capacity, simulate batching and autoscaling, implement quantisation and caches, and reason about cost with utilisation, not just list prices.

Graded labs are pure Python simulations and arithmetic, so no GPU is required; instructor-led demos show vLLM-style serving on real hardware. The capstone is a recommendation memo backed by benchmarks, a capacity model and a Pareto analysis.

Who it is for

ML, platform and backend engineers responsible for LLM cost, latency and reliability.

What makes it different

It teaches the numbers — memory, throughput, utilisation and unit cost — so decisions are engineering, not guesswork.

By the end you will be able to

Explain prefill, decode and continuous batching and reason about TTFT, TPOT, throughput and goodput.
Calculate KV-cache and weight memory, plan concurrent capacity and explain paged allocation and GQA.
Apply quantisation and distillation and evaluate the quality cost of each.
Design prefix, exact and semantic caches and routing cascades with measured hit precision.
Plan GPU autoscaling with warm pools, queue-based signals, admission control and tenant rate limits.
Model cost, latency and quality trade-offs, find the Pareto frontier and choose an operating point against constraints.
Deliver a defensible inference recommendation combining benchmarks, capacity model, cost analysis and risk plan.

Curriculum

Topics covered

Core topics 6 topics

Serving frameworks and continuous batching
How LLM serving works: prefill and decode, throughput vs latency, and why continuous batching changed everything.
KV cache and memory management
The KV cache: what it stores, how big it gets, and how paging and prefix sharing manage GPU memory.
Quantisation and distillation
Quantisation and distillation: shrink models to cut memory, cost and latency — and measure what you lose.
Caching and request routing
Caching and routing: exact and semantic caches, prompt-prefix reuse and model routing to cut cost and latency.
Autoscaling GPU fleets
Autoscaling GPU fleets: queue-based scaling, cold starts, warm pools, spot capacity and multi-tenant fairness.
Cost / latency / quality trade-offs
Cost, latency and quality trade-offs: build an inference budget, find the Pareto frontier and choose an operating point.

Course at a glance

See how this course is built

Where the time goes

Longer spokes mean more hours on that module.

Course roadmap click a step

Try a lesson0 / 3

Sample questions from this course. Your progress is kept only in your browser.

Week by week

Drag the slider, or press play, to walk through the course and see what you will have built by each week.

Week 1 of 6

Wk 1–2

Serve efficiently

You produceAn LLM endpoint with batching and caching, benchmarked

Wk 3–4

Shrink and scale

You produceA quantised deployment with an autoscaling policy

Wk 5–6

Cost model and project

You produceA cost / latency report and a production-ready config

What you'll build

Serving simulators and capacity planner

Batching and autoscaling simulators with a KV-cache capacity planner.

Cost and quality analysis

Utilisation-aware unit-cost analysis and a Pareto table of configurations.

Inference recommendation memo

A benchmark-backed recommendation with capacity model, cost comparison and risk plan.

Why take this course

Understand the real bottlenecks: prefill, decode and memory
Quantify capacity and unit cost instead of guessing
Labs need no GPU — simulations and arithmetic
Learn cost levers: caching, routing, quantisation, autoscaling

You will build

A static vs continuous batching simulator
A KV-cache capacity planner and paged block allocator
An int8/int4 quantiser with per-row scales
An LRU cache, semantic cache and cascade router
An autoscaling simulator with cold starts and a token-bucket limiter
A Pareto chooser and fleet cost model

What you can do afterwards

Size and cost an LLM deployment with confidence
Choose an operating point using cost, latency and quality constraints
Diagnose serving bottlenecks and scaling failures

Career fit

🧭 Where this fits

Roles whose screened skills map to this course, and the skills it lets you show.

Skills you will build

LLM serving & inference at scale
Model serving, batching, drift & retraining
Quantisation, distillation & inference optimisation

How to take it

💳 Delivery & fees

One course, four ways to attend. Fees are per course, in rupees excluding 18% GST (the GST-inclusive amount is shown too) with Canadian-dollar fees alongside.

Your fee includes our GPU lab infrastructure — no cloud credits to buy. See how we compare, with sources →