CareerOS
Sign inGet started
Learning Paths

Data Engineering

Move data reliably from where it is made to where it is used: SQL and modelling, pipelines and orchestration, warehouses, batch and streaming.

Finish this and you can build a tested, scheduled pipeline from raw files to a modelled warehouse that an analyst trusts.

Starts at
Beginner
Topics
4
Projects
4
Of material
79 h
  • SQL
  • Data modelling
  • Python
  • Airflow
  • dbt
  • Spark
01Beginner

SQL and relational databases

Queries, joins, window functions, indexes and transactions: the language every data job is built on.

Foundation

  • SQLBolt

    SQLBolt · interactive · 3 h

    Short interactive lessons in the browser. Finish all of them in a weekend.

  • PostgreSQL tutorial

    PostgreSQL · docs · 4 h

    The official tutorial: install, query, and the advanced features you will meet at work.

Practitioner

  • Use The Index, Luke

    use-the-index-luke.com · book · 8 h

    Why queries are slow and how indexes fix them, for developers rather than DBAs.

Project

Answer ten business questions in SQL

Load a real public dataset into Postgres and answer questions a manager would ask.

  1. Load and constrain the data with keys and types, and note what does not fit.
  2. Write ten questions first, then the queries, using at least three window functions.
  3. Make two slow queries fast with indexes and show the plans before and after.
NYC TLC trip records
02Intermediate

Data modelling for analytics

Facts, dimensions and slowly changing data: models that make the right question easy to ask.

Foundation

  • dbt Fundamentals

    dbt Labs · course · 5 h

    Models, tests, documentation and sources, in the tool most analytics teams now use.

Practitioner

  • Dimensional modelling techniques

    Kimball Group · article · 3 h

    The vocabulary every analytics engineer uses: facts, dimensions, grain and slowly changing dimensions.

Project

A tested star schema in dbt

Model the dataset from the SQL project for analysts.

  1. Decide the grain of each fact table and write it down.
  2. Build staging, dimension and fact models in dbt.
  3. Add tests for keys, nulls and accepted values, and fix what they find.
  4. Publish the docs site with the lineage graph.
03Intermediate

Pipelines and orchestration

Ingest, transform and load on a schedule, with retries, backfills and alerts when something breaks.

Foundation

  • Data Engineer Handbook

    GitHub · repo · 2 h

    A curated map of books, communities, newsletters and projects for the field.

Practitioner

Project

A pipeline that survives a bad day

Schedule the whole flow from raw files to your dbt models.

  1. Ingest monthly files idempotently, so a rerun never duplicates rows.
  2. Orchestrate ingest, dbt run and dbt test as one DAG with retries.
  3. Backfill a year and time it.
  4. Break it on purpose with a malformed file and show the alert and recovery.
04Advanced

Batch at scale, and streaming

When data no longer fits on one machine, and when it never stops arriving.

Practitioner

  • Apache Spark quick start

    Apache Spark · docs · 1 h 30 min

    DataFrames and the execution model. Run it locally before reaching for a cluster.

  • Introduction to Apache Kafka

    Apache Kafka · docs · 45 min

    Topics, partitions, producers and consumers, from the project itself.

Project

Live aggregates from a stream

Compute a rolling metric from an event stream.

  1. Produce events by replaying the trip data into Kafka at speed.
  2. Consume and aggregate in five-minute windows.
  3. Handle late events and explain your choice of watermark.
  4. Compare with the batch answer and explain any difference.

Not sure this is your direction?

A three-minute counselling sample is free.

Try counselling