Fine-tuning & Model Evaluation

LoRA / QLoRA, instruction tuning, preference data, and building evaluation suites you can trust before you ship.

⏰ 7 weeks🎯 Standard🧩 Certificate🖥 Any device · our GPUs

Generated from this course

🎯 About this course

Fine-tuning is easy to start and hard to justify. This course focuses on the judgement and the evidence: when fine-tuning is the right tool, how LoRA/QLoRA make adaptation cheap, how to prepare leak-free instruction and preference data, and how to know whether a change actually helped.

You will build evaluation suites with programmatic graders, quantify uncertainty with Wilson intervals and paired bootstraps, calibrate LLM judges against human labels, and write release gates that protect critical slices. The applied studio has you fine-tune a small model for structured extraction and decide — honestly — whether to ship it.

Who it is for

Engineers and data scientists shipping LLM features who need to change models safely.

What makes it different

The graded labs need no GPU; the emphasis is on evaluation, statistics and decision-making, which is where most teams struggle.

By the end you will be able to

Decide between prompting, retrieval and fine-tuning using a cost-, data- and evaluation-aware framework.
Explain and implement LoRA/QLoRA, including parameter, memory and merge calculations.
Prepare clean, leak-free instruction and preference datasets and compute preference-tuning objectives.
Design task-specific evaluation suites with tagged cases, programmatic graders, slices and confidence intervals.
Use LLM judges responsibly: write rubrics, detect position bias and measure agreement with humans.
Build regression tests and release gates using paired statistical comparison and slice floors.
Deliver an evidence-based ship / no-ship decision for a model change with cost, latency and rollback considerations.

Curriculum

Topics covered

Core topics 6 topics

When to fine-tune, and when not to
Fine-tuning is expensive to set up and maintain — data collection, training runs, evaluation, re-training as your needs change; prompting and RAG solve most "I need the…
LoRA and QLoRA
LoRA (Low-Rank Adaptation) freezes the base model's weights and trains a small number of additional low-rank matrices, which is far cheaper in memory and compute than…
Instruction and preference tuning
Instruction tuning trains a model on (instruction, good response) pairs to make it better at following directions generally; preference tuning — e.g.
Building evaluation suites
The same evaluation discipline from the LLM Application Engineering course applies here, with one addition: for a fine-tuned model, you must always compare against the…
LLM-as-judge and its pitfalls
When evaluating a fine-tuned model, be especially careful that the judge model itself wasn't trained in a way that's correlated with your fine-tuning data — that can…
Regression testing and release gates
Before promoting a fine-tuned checkpoint to production, require it to clear a fixed evaluation bar on a held-out set that was never used in training or hyperparameter…

Course at a glance

See how this course is built

Where the time goes

Longer spokes mean more hours on that module.

Course roadmap click a step

Try a lesson0 / 3

Sample questions from this course. Your progress is kept only in your browser.

Week by week

Drag the slider, or press play, to walk through the course and see what you will have built by each week.

Week 1 of 7

Wk 1–2

Data and method

You produceA curated tuning dataset and a baseline evaluation

Wk 3–5

Fine-tune and compare

You produceA LoRA-tuned model that beats the baseline on your evaluation

Wk 6–7

Evaluation harness and capstone

You produceA reusable evaluation suite and a written analysis of the trade-offs

What you'll build

Evaluation suite

A tagged suite of 60+ cases with held-out split, programmatic graders and a calibrated judge where needed.

Statistical comparison report

A baseline-versus-candidate comparison with paired intervals and slice analysis.

Ship / no-ship release memo

An evidence-based release decision for a fine-tuned model with cost, latency and rollback plan.

Why take this course

Know when fine-tuning is (and is not) the right tool
Evaluation suites, judges and statistics you can defend
Auto-graded labs that need no GPU
Release gates that protect critical slices

You will build

A fine-tune decision helper and cost break-even model
LoRA maths: parameter counts, forward pass and merge
A dataset cleaner and DPO loss
An eval harness with slices and Wilson intervals
A paired-bootstrap release gate

What you can do afterwards

Decide ship / no-ship for a model change with evidence
Explain LoRA and evaluation trade-offs to stakeholders
Build regression protection for LLM products

Career fit

🧭 Where this fits

Roles whose screened skills map to this course, and the skills it lets you show.

Skills you will build

Fine-tuning (LoRA) & model evaluation
Fine-tuning (LoRA), guardrails, cost / latency
Tool use, agents, evaluation suites

How to take it

💳 Delivery & fees

One course, four ways to attend. Fees are per course, in rupees excluding 18% GST (the GST-inclusive amount is shown too) with Canadian-dollar fees alongside.

Your fee includes our GPU lab infrastructure — no cloud credits to buy. See how we compare, with sources →