LLM Application Engineering

Prompting & model I/O 6 topics

• System vs user prompts
  The system prompt sets the model's role, constraints and tone for the whole conversation; the user prompt carries the specific request.
• Few-shot & pattern prompting
  Showing the model 2-4 worked examples ("few-shot") is often more reliable than describing the desired behaviour in prose, especially for formatting or style you can't…
• Reliable structured / JSON output
  Ask for a specific schema, give one example of valid output, and validate every response before you trust it — models occasionally emit near-valid JSON (trailing commas…
• Streaming responses
  Streaming tokens back to the user as they're generated cuts perceived latency even though total generation time is unchanged — for anything longer than a short answer…
• Handling refusals & truncation
  A response can be truncated because it hit the model's max-token limit, not because the model finished — always check the stop reason your provider returns, not just…
• Token & context budgeting
  Every model has a fixed context window that covers the system prompt, conversation history, any retrieved documents and the response — if you don't budget it, long…

Retrieval (RAG) 6 topics

• Chunking strategies
  How you split documents into chunks determines what the retriever can find.
• Embeddings & vector stores
  An embedding model turns text into a vector such that semantically similar text ends up nearby in vector space; a vector store indexes those vectors for fast…
• Hybrid (keyword + vector) search
  Vector search finds semantic matches but can miss exact terms (product codes, names, acronyms) that keyword search catches trivially, and vice versa — keyword search…
• Re-ranking & metadata filtering
  Initial retrieval (vector or hybrid) is optimised for recall — get plausible candidates fast; a re-ranker (often a cross-encoder) then scores that smaller candidate set…
• Citations & “I don't know” handling
  Attach a citation — source, chunk id — to every claim the answer makes so a user, or your own eval process, can verify it against the retrieved text; an answer with no…
• Retrieval evaluation sets
  Build a small, hand-checked set of realistic queries paired with the chunk(s) that should be retrieved for each — this lets you measure retrieval quality (e.g.

Tool use & agents 6 topics

• Function calling & schemas
  Function/tool calling lets the model choose to invoke one of your predefined functions and returns structured arguments, instead of you parsing free text to guess intent.
• Input validation & safe execution
  Never execute an argument the model produced without validating it the same way you'd validate untrusted user input — a "run this SQL" or "call this shell command" tool…
• Planning loops & memory
  An agent loop repeatedly asks the model "given the goal and what's happened so far, what's the next action" and executes that action, feeding the result back in — this…
• Multi-step task decomposition
  Breaking a large task into smaller sub-tasks — plan, execute, check — generally produces more reliable results than asking for the whole thing in one shot, because each…
• Agent failure modes & recovery
  Common agent failures: looping on the same failed action, hallucinating a tool result instead of calling the tool, silently giving up and returning a plausible-sounding…
• When not to use an agent
  An agent — multi-step, tool-using, autonomous loop — is the right tool when the task genuinely requires dynamic, unpredictable steps.

Evaluation 6 topics

• Building labelled test sets
  A labelled test set — real or realistic inputs paired with a known-good (or known-acceptable-range) output — is what turns "it seems to work" into a number you can track.
• Offline evaluation suites
  Run your test set against the pipeline automatically — on every prompt or model change — and score it with a mix of exact-match, rule-based checks and, where those don't…
• LLM-as-judge and its pitfalls
  Using a second LLM call to grade an answer's quality scales further than human review, but judges have known biases — they tend to prefer longer answers, answers similar…
• Regression testing on prompts
  Treat prompt and pipeline changes like code changes: run the full eval suite before merging, and require a clear reason — a demonstrated quality or cost improvement —…
• Tracking quality over time
  Log evaluation scores per release/version, and slice them by input type, user segment or difficulty — an aggregate score can hold steady while one important slice…
• Reading eval results critically
  A high aggregate score can hide a systematic gap — the eval set doesn't cover a case that's common in production — or gaming, where the model learned to satisfy the…

Guardrails, safety & cost 6 topics

• Input / output filtering
  Filter both what goes into the model — prompt injection attempts, PII you shouldn't be sending, disallowed request types — and what comes out — unsafe content, leaked…
• Prompt-injection basics
  Prompt injection is when untrusted content — a user message, a retrieved document, a tool result — contains instructions that hijack the model's behaviour away from your…
• Rate & spend controls
  Put hard limits on requests-per-user, tokens-per-request and total spend per time window before you ship anything that calls a paid API on user input.
• Response caching
  Caching identical or near-identical requests — exact-match, or semantic-similarity match on the query — cuts both latency and cost for common questions, but only where a…
• Model selection for cost / latency
  Bigger, more expensive models are not always more accurate for a given task.
• Graceful degradation
  Design every LLM-dependent feature with a fallback for when the model call fails, times out, or is rate-limited — a cached answer, a simpler rule-based response, or a…

Production & iteration 6 topics

• Request tracing & observability
  Log the full prompt, the model/version used, the response, latency, token counts and cost for every call, redacting sensitive data as needed.
• Collecting & using user feedback
  A simple thumbs up/down, or an implicit signal — did the user retry, did they abandon the session — is a cheap and continuous source of real-world quality data.
• Prompt & config versioning
  Store prompts, model names, and pipeline configuration in version control, tagged with the eval score they were measured against, so you can always answer "what changed"…
• Deployment of an LLM service
  Beyond normal service deployment concerns — health checks, rollout, rollback — an LLM service needs to handle provider outages and rate limits gracefully, support…
• Finding the weak stage in a pipeline
  When a RAG or agent pipeline gives a bad answer, isolate which stage failed — retrieval didn't find the right chunk, the chunk was found but the model ignored it, the…
• Shipping an improvement with confidence
  Before shipping a change, know what problem it's meant to fix, measure it against your eval suite (including the slices most likely to be affected), and have a rollback…

A RAG assistant

Retrieval-augmented assistant over a real document corpus, with its own retrieval evaluation set.

A tool-using agent

An agent that calls your code safely, with input validation, guardrails and spend limits.

A shipped capstone

A complete, documented LLM application on a real brief — built, measured and presented.

Why take this course

• Mentor-reviewed code, not just video lectures — every lab gets real feedback.
• Built around one real, evolving LLM application from week 1 to the capstone.
• Covers the parts most courses skip: evaluation, cost control, and shipping safely.

You will build

• A retrieval-augmented assistant with citations and a measured "I don't know" rate.
• A tool-using agent with input validation and a bounded retry/failure policy.
• An offline evaluation harness you can run on every prompt or model change.

What you can do afterwards

• A portfolio-ready capstone with a written evaluation report, not just a demo.
• A working mental model for when to prompt, when to retrieve, and when to fine-tune.