CareerOS
Sign inGet started
Learning Paths

AI Engineering

Build products on top of large language models: prompts that hold up, retrieval over your own data, agents with tools, evaluations you trust, and a system you can run in production.

Finish this and you can design, evaluate and ship an LLM feature end to end, and explain every trade-off you made to a senior engineer.

Starts at
Intermediate
Topics
5
Projects
5
Of material
53 h
  • LLMs
  • Prompt engineering
  • RAG
  • Agents
  • Evaluation
  • Python
01Beginner

LLM APIs and prompting that holds up

Call a model from code, structure its output, and write prompts that behave the same way on the hundredth input as on the first.

Foundation

  • Prompt Engineering Guide

    DAIR.AI · docs · 4 h

    A map of every prompting technique with examples. Read the basics and the techniques sections before anything else.

  • Prompt engineering interactive tutorial

    GitHub · interactive · 5 h

    Nine chapters of notebooks with exercises. Do the exercises; reading the answers teaches far less.

Practitioner

  • OpenAI Cookbook

    OpenAI · docs · 6 h

    Worked recipes for structured outputs, function calling and embeddings. Most patterns carry over to any provider.

Project

A job-posting extractor that never returns broken JSON

A small CLI that turns any job posting into a validated record: title, seniority, must-have skills, nice-to-haves and location.

  1. Define the schema first. A Pydantic or Zod model with every field required and nullable where unknown.
  2. Collect twenty postings from different industries and save them as text fixtures.
  3. Prompt for structured output and validate every response against the schema; on failure, retry once with the validation error in the prompt.
  4. Report the failure rate before and after the retry, and write down which postings broke it and why.
02Intermediate

Retrieval and RAG over your own data

Embeddings, chunking and vector search, then retrieval-augmented generation that cites its sources and says so when it does not know.

Foundation

Practitioner

  • Build a RAG app

    LangChain · docs · 2 h

    A complete pipeline in one tutorial. Build it once with the framework, then once without, to see what it hides.

  • Sentence Transformers documentation

    Sentence Transformers · docs · 2 h

    How embedding models are trained and evaluated, and how to pick one for your language and domain.

Project

A documentation assistant that cites every answer

Answer questions over one real documentation site, with a link to the passage behind every claim.

  1. Crawl and chunk one docs site; record chunk size and overlap as parameters.
  2. Embed and index the chunks in a local vector store.
  3. Write thirty questions with known answers before you build the prompt.
  4. Answer with citations and refuse when the top passages do not support an answer.
  5. Tune one parameter at a time and keep a table of accuracy against chunk size and number of passages retrieved.
03Intermediate

Agents, tools and protocols

When an agent is the right design and when a fixed workflow is better, tool calling, the Model Context Protocol, and agents that recover from their own mistakes.

Practitioner

  • Building effective agents

    Anthropic · article · 30 min

    The patterns that work in practice, and why the simplest one that solves the problem is usually the right one.

  • Hugging Face Agents Course

    Hugging Face · course · 15 h

    From what an agent is to building one with several frameworks. Free certificate if you finish the assignments.

  • Model Context Protocol

    Model Context Protocol · docs · 1 h 30 min

    The open standard for connecting models to tools and data. Build one small server to understand it.

Project

A research agent with a budget and a stop condition

An agent that answers a question by searching, reading and summarising, within a hard limit on steps and cost.

  1. Give it three tools: search, fetch a page, and write a note.
  2. Enforce a budget of steps and tokens in code, not in the prompt.
  3. Log every step so a run can be replayed and inspected.
  4. Compare against a fixed workflow (search, read the top three, summarise) on ten questions, and say honestly which one won.
04Intermediate

Evaluating LLM systems

Turn "it seems better" into numbers: error analysis, test sets, model-graded checks and regression tests that run on every change.

Practitioner

  • Your AI Product Needs Evals

    hamel.dev · article · 45 min

    The most practical account of why evaluation is the core of an AI product, with a worked case study.

  • promptfoo: getting started

    promptfoo · docs · 1 h 30 min

    An open-source tool for running prompt and model test suites in CI.

Builder

  • OpenAI Evals

    GitHub · repo · 2 h

    A framework and registry of evaluations. Read a few to see how good test cases are written.

Project

An evaluation harness for your RAG assistant

Make every change to the assistant from the RAG topic prove itself.

  1. Do error analysis first: read fifty real outputs and label what went wrong.
  2. Turn each failure type into a test, with code checks where possible and a model-graded check only where you must.
  3. Check the grader against your own labels and report how often it agrees.
  4. Run the suite in CI so a pull request shows the score change.
05Advanced

LLMs in production

Latency, cost, caching, guardrails, security and monitoring: what changes when real users arrive.

Practitioner

  • LLM Bootcamp

    Full Stack Deep Learning · video · 10 h

    Lectures on the whole lifecycle of an LLM product, from prompting to deployment and user experience.

  • OWASP Top 10 for LLM Applications

    OWASP · docs · 1 h 30 min

    Prompt injection, data leakage and the other risks every LLM feature has to be designed against.

Builder

Project

Harden one LLM feature for real traffic

Take any feature you built earlier and make it safe and affordable to run.

  1. Measure the baseline: p50 and p95 latency and cost per request.
  2. Add a cache and a cheaper fallback model, and measure again.
  3. Red-team it with ten prompt-injection attempts from the OWASP list and fix what gets through.
  4. Add tracing so any bad answer can be followed back to its inputs.

Not sure this is your direction?

A three-minute counselling sample is free.

Try counselling