Template

Machine Learning Engineer — Competency Roadmap

Work towards being a machine learning engineer: the mathematics, the modelling and the engineering needed to take models from an experiment to something that runs reliably in production.

This comprehensive roadmap guides you from zero programming experience through foundational mathematics, classical machine learning, deep learning, production software engineering, and end-to-end MLOps architectures. At 12 hours per week, this extensive curriculum provides the complete territory needed to transition into the field across 12 distinct competency phases. By the end, you will have engineered and deployed full-stack, monitored ML systems with automated pipelines and reproducible infrastructure.

By the end: You will be able to design, train, evaluate, deploy, and monitor scalable machine learning systems in production using containerized microservices, automated CI/CD pipelines, and cloud-agnostic MLOps patterns.

Starting levelBeginnerStyleBuilding things
12h / week12 phases27 tasks~222h total

This is the map — make this roadmap yours

It shows what this journey generally looks like. Tell Kaidoro your version of the goal and it builds the plan around where you are actually starting, what to do first, the hours you really have, and what you have already finished.

1

Programming & Software Engineering Foundations

Establish core software craftsmanship in Python, command-line environments, version control, and modular testing practices before touching machine learning algorithms.

  • Set up a modern Python development environment
    ~4hLearn1 resource

    Reliable environment management prevents dependency conflicts and establishes professional development hygiene from day one.

    You'll learn

    • venv — standard library module for creating isolated Python environments
    • Ruff — extremely fast Python linter and code formatter written in Rust
    • pyproject.toml — standard configuration file for Python packaging and tools

    Install Python 3.11+, configure VS Code or PyCharm, set up virtual environments using venv or uv, and establish code formatting with Ruff and Black.

    Done when: You can create an isolated virtual environment, install packages, and automatically format and lint code on save.

    How to work through it

    1. Install Python 3.11+ using the official installer or pyenv
    2. Configure VS Code with Python and Ruff extensions
    3. Create and activate a virtual environment using venv
    4. Configure pyproject.toml with Ruff formatting rules
  • Build a modular CLI data processing tool
    ~8hBuild1 resource

    Machine learning engineering relies on modular software architecture rather than unstructured script files.

    You'll learn

    • typer — modern Python library for building CLI applications based on type hints
    • argparse — standard Python library for command-line parsing
    • Modular packaging — organizing Python files into importable packages

    Write a structured command-line interface tool in Python using argparse or typer that reads CSV files, performs string manipulation and aggregation, and writes clean JSON outputs. Structure the project into distinct modules with __init__.py files.

    Done when: The CLI tool executes from the terminal, parses arguments cleanly, and produces expected structured outputs across three sample datasets.

    How to work through it

    1. Define modular directory structure separating CLI interface from core business logic
    2. Implement CLI flags and input validation using typer or argparse
    3. Write file parsers and error handlers for malformed CSV rows
    4. Execute the CLI tool against test CSV files
  • Implement automated testing with pytest and Git version control
    ~6hBuild1 resource

    Automated testing is the fundamental safety net for reliable engineering systems.

    You'll learn

    • pytest — industry standard Python test runner and framework
    • pytest fixtures — explicit, modular test setup and teardown helpers
    • Conventional Commits — standardized specification for commit messages

    Initialize a Git repository, establish branch management workflows, and write comprehensive unit tests with pytest including fixtures and edge-case testing for the CLI tool.

    Done when: The test suite passes with 100% test success across at least 8 distinct unit tests and changes are cleanly committed with conventional commit messages.

    How to work through it

    1. Initialize Git repo and write a robust .gitignore
    2. Install pytest and write unit test files under a tests directory
    3. Implement pytest fixtures for reusable test data generators
    4. Run pytest and verify all assertions pass
2

Applied Mathematics & Vectorized Computing

Acquire practical proficiency in linear algebra, multivariable calculus, probability, and statistics implemented computationally with NumPy.

  • Build matrix operations from scratch with pure Python and NumPy
    ~10hBuild1 resource

    Vectorization is essential for fast tensor operations and understanding model computational bottlenecks.

    You'll learn

    • Broadcasting — NumPy mechanism for arithmetic operations across arrays of different shapes
    • SIMD Vectorization — single instruction multiple data CPU parallelism leveraged by NumPy
    • Cosine Similarity — geometric measure of orientation between multi-dimensional vectors

    Implement core linear algebra operations (dot products, matrix multiplication, vector norms, and matrix inversion) first using native Python loops, then vectorizing them with NumPy to measure performance differences.

    Done when: Benchmarks demonstrate your vectorized NumPy implementation running at least 20x faster than pure Python loops on 1000x1000 matrices.

    How to work through it

    1. Write pure Python nested loops for 2D matrix multiplication
    2. Re-implement using NumPy array broadcasting and np.dot
    3. Profile execution time using timeit across varying matrix sizes
    4. Calculate Euclidean, Manhattan, and cosine distances between vectors
  • Implement numerical gradient descent for multivariable functions
    ~8hBuild1 resource

    Gradients form the mathematical backbone of model training across classical ML and deep neural networks.

    You'll learn

    • Partial Derivative — rate of change of a multivariable function along a single axis
    • Gradient Vector — direction of steepest ascent in multi-dimensional parameter space
    • Learning Rate — hyperparameter governing optimization step magnitude

    Code multivariable derivatives, partial differentiation, and gradient descent optimization algorithms from scratch without external ML libraries to minimize loss functions.

    Done when: The optimization script successfully converges on global minima for quadratic and Rosenbrock test functions within 1e-4 tolerance.

    How to work through it

    1. Define analytical functions and their exact partial derivatives in NumPy
    2. Implement standard gradient descent with configurable learning rates
    3. Add momentum tracking to prevent oscillations in steep valleys
    4. Plot loss trajectories across optimization steps
  • Build a statistical hypothesis testing and probability engine
    ~8hBuild1 resource

    Statistical rigor is required to evaluate model lift and detect real performance drift in production.

    You'll learn

    • Central Limit Theorem — distribution of sample means approaches normality as size increases
    • P-value — probability of observing extreme test results under the null hypothesis
    • Statistical Power — probability of correctly rejecting a false null hypothesis

    Implement probability distribution sampling (Gaussian, Binomial, Poisson), calculate confidence intervals, compute p-values, and conduct A/B hypothesis tests via Monte Carlo simulations.

    Done when: The statistical engine correctly identifies whether sample differences are statistically significant at alpha=0.05 across simulated A/B test experiments.

    How to work through it

    1. Simulate normal and non-normal random variables with numpy.random
    2. Calculate empirical cumulative distribution functions and summary metrics
    3. Implement two-sample t-test and Mann-Whitney U test calculations
    4. Construct a Monte Carlo simulation for power analysis
3

Data Engineering & Tabular Wrangling

Master structured data manipulation, relational modeling, exploratory data analysis, and scalable processing using Pandas, Polars, and SQL.

  • Write complex analytical SQL queries against SQLite/PostgreSQL
    ~8hPractice1 resource

    Feature extraction and dataset curation in engineering pipelines almost always originate from SQL databases.

    You'll learn

    • Window Functions — SQL calculation over an explicit partition of table rows
    • B-Tree Index — database index structure speeding up query search time
    • Query Execution Plan — database engine roadmap detailing query cost

    Design relational schemas, perform multi-table joins, compute aggregation metrics using window functions (e.g., ROW_NUMBER(), LAG(), LEAD()), and optimize indices.

    Done when: You can write a single SQL query computing 7-day rolling averages and customer cohort retention tables on a million-row dataset.

    How to work through it

    1. Set up a local SQLite or PostgreSQL database and load sample transaction data
    2. Write multi-table inner, left, and outer join queries
    3. Implement window functions with PARTITION BY and ORDER BY clauses
    4. Profile query performance using EXPLAIN ANALYZE and add appropriate indices
  • Engineer reproducible tabular transformation pipelines with Polars
    ~8hBuild1 resource

    High-throughput feature engineering requires modern, memory-efficient dataframe engines like Polars.

    You'll learn

    • Lazy Evaluation — deferring execution until the full query graph is optimized
    • Apache Parquet — columnar storage file format optimized for fast analytical read queries
    • Cyclical Encoding — trigonometric mapping for periodic continuous variables

    Clean, impute, transform, and aggregate dirty tabular datasets using Polars lazy execution and Pandas, implementing custom feature engineering pipelines.

    Done when: The automated pipeline parses raw transaction logs, extracts 25+ engineered numerical and categorical features, and outputs clean parquet files without memory leakage.

    How to work through it

    1. Load raw messy data into Polars LazyFrame
    2. Handle null values with median/mode strategies and categorical encoding
    3. Extract datetime features (hour, day of week, cyclical sine/cosine encodings)
    4. Persist output in Apache Parquet format with snappy compression
4

Classical Machine Learning & Validation

Learn the theory, mathematics, and implementation of supervised and unsupervised classical algorithms with strict validation protocols.

  • Implement Linear and Logistic Regression from mathematical first principles
    ~8hBuild1 resource

    Writing algorithms from scratch demystifies loss functions, decision boundaries, and regularization mechanics.

    You'll learn

    • Sigmoid Function — S-shaped curve mapping real values to probabilities between 0 and 1
    • Binary Cross-Entropy — standard loss function measuring dissimilarity between binary distributions
    • L2 Regularization — weight penalty reducing model variance and preventing overfitting

    Write custom classes for Linear Regression (Ordinary Least Squares) and Logistic Regression using NumPy, including vectorized cost functions, gradient descent updates, and L1/L2 regularization penalties.

    Done when: Your custom estimator achieves parity within 1% metric margin against Scikit-Learn on identical classification datasets.

    How to work through it

    1. Derive the Mean Squared Error and Binary Cross-Entropy loss functions
    2. Implement fit() and predict_proba() methods in pure NumPy
    3. Add Ridge (L2) and Lasso (L1) penalty terms to gradient calculations
    4. Compare accuracy and weights against Scikit-Learn's LogisticRegression
  • Train and tune Tree-based ensembles (Random Forest & XGBoost/LightGBM)
    ~10hBuild1 resource

    Tree ensembles remain the industry standard for tabular prediction problems in real-world ML systems.

    You'll learn

    • Gradient Boosting — ensemble method building successive trees to predict residual errors
    • Optuna — hyperparameter optimization framework utilizing Tree-structured Parzen Estimators
    • Stratified K-Fold — cross-validation split preserving label percentage ratios in every fold

    Train Decision Trees, Random Forests, and Gradient Boosted Decision Trees (XGBoost/LightGBM). Perform rigorous hyperparameter tuning using cross-validation and evaluate feature importance.

    Done when: The tuned gradient boosting model achieves a ROC-AUC > 0.85 on an imbalanced tabular fraud/churn dataset with zero data leakage.

    How to work through it

    1. Construct stratified k-fold cross-validation splits preventing data leakage
    2. Train baseline Random Forest and compute Gini impurity feature importances
    3. Configure XGBoost / LightGBM with early stopping on a validation set
    4. Perform Bayesian hyperparameter search using Optuna
  • Implement unsupervised clustering and dimensionality reduction
    ~6hBuild

    Unsupervised methods are critical for anomaly detection, feature engineering, and customer segmentation.

    You'll learn

    • Eigenvectors — directions along which a linear transformation acts by stretching
    • Silhouette Score — measure of how similar an object is to its own cluster vs other clusters
    • StandardScaler — feature normalization ensuring zero mean and unit variance

    Implement K-Means clustering and Principal Component Analysis (PCA) to segment high-dimensional data, evaluate cluster cohesion with Silhouette scores, and visualize projections.

    Done when: PCA successfully reduces a 50-feature dataset to explain >80% variance in 5 components, and K-Means identifies distinct, interpretable clusters.

    How to work through it

    1. Standardize features with zero mean and unit variance
    2. Compute covariance matrix and calculate eigenvectors/eigenvalues for PCA
    3. Implement K-Means++ initialization and iterative cluster assignment
    4. Evaluate cluster stability using Elbow method and Silhouette scores
5

Deep Learning Fundamentals & PyTorch

Transition to deep neural networks, computational graphs, backpropagation, and PyTorch architecture fundamentals.

  • Build a multi-layer perceptron (MLP) with custom PyTorch training loop
    ~10hBuild1 resource

    Understanding the explicit mechanics of PyTorch training loops prevents subtle bugs in complex deep learning pipelines.

    You'll learn

    • torch.autograd — PyTorch automatic differentiation engine computing gradients
    • DataLoader — multi-threaded batch generator for model training
    • Adam Optimizer — adaptive moment estimation gradient descent algorithm

    Construct neural network architectures using torch.nn.Module, write an explicit training loop with forward pass, loss calculation, backward propagation (loss.backward()), and optimizer stepping.

    Done when: The PyTorch MLP trains stably, achieves convergence on a non-linear dataset, and logs training and validation losses per epoch.

    How to work through it

    1. Create custom dataset and DataLoader classes with batching and shuffling
    2. Define PyTorch network subclassing torch.nn.Module with ReLU activations
    3. Write training loop iterating over batches, zeroing gradients, and calling backward()
    4. Track validation metrics and save model checkpoints
  • Train a Convolutional Neural Network (CNN) with transfer learning
    ~8hBuild1 resource

    Transfer learning is the primary industry paradigm for training high-performing neural models with limited data.

    You'll learn

    • Convolutional Layer — spatial feature extraction kernel sliding across images
    • Transfer Learning — adapting pre-trained weights from large datasets to target tasks
    • Learning Rate Scheduler — adjusting learning rate during training epochs

    Build an image classification model using a pre-trained ResNet/ConvNeXt backbone, freeze early layers, and fine-tune classifier heads using PyTorch torchvision.

    Done when: The fine-tuned CNN achieves >90% validation accuracy on a custom image classification dataset with proper data augmentations applied.

    How to work through it

    1. Implement image preprocessing and augmentation transforms (random crops, flips)
    2. Instantiate pre-trained ResNet backbone from torchvision.models
    3. Freeze base weights and replace final fully connected layer
    4. Fine-tune the model with learning rate scheduling and early stopping
6

Sequence Models, Transformers & LLM Fundamentals

Understand attention mechanisms, Transformer architectures, and modern Large Language Model (LLM) fine-tuning techniques.

  • Implement Scaled Dot-Product and Multi-Head Attention
    ~10hBuild1 resource

    The attention mechanism is the foundation of modern NLP, LLMs, and multimodal architectures.

    You'll learn

    • Self-Attention — mechanism relating different positions of a single sequence
    • Causal Masking — matrix masking preventing attention to subsequent sequence tokens
    • Softmax Temperature — scaling factor preserving gradient stability in dot products

    Code the self-attention mechanism $(Q, K, V)$ from scratch in PyTorch, apply causal masking, and assemble a single Transformer decoder layer.

    Done when: Your custom Multi-Head Attention module produces exact numerical tensor outputs matching torch.nn.MultiheadAttention on identical inputs.

    How to work through it

    1. Implement query, key, value matrix projections
    2. Compute scaled dot-product attention with softmax scaling
    3. Apply triangular causal mask to prevent looking at future tokens
    4. Implement multi-head tensor splitting, concatenation, and output projection
  • Fine-tune a Hugging Face Transformer with PEFT/LoRA
    ~10hBuild1 resource

    Parameter-efficient fine-tuning enables practical customization of modern foundation models on standard hardware.

    You'll learn

    • LoRA (Low-Rank Adaptation) — technique freezing base weights and training rank decomposition matrices
    • Tokenization — splitting raw text into numerical subword tokens
    • Hugging Face Trainer — unified API for PyTorch model training and evaluation

    Fine-tune an open-source Transformer model for sequence classification or text generation using Hugging Face Transformers, Datasets, and Low-Rank Adaptation (LoRA).

    Done when: The fine-tuned LoRA model trains with under 5% of total parameters updated and demonstrates measurable metric improvement on your target NLP dataset.

    How to work through it

    1. Tokenize and prepare text dataset with Hugging Face Datasets
    2. Configure LoRA adapter target modules using the PEFT library
    3. Run training using Hugging Face Trainer with evaluation logging
    4. Merge LoRA weights and run evaluation benchmarks against the base model
7

Containerization & Reproducible Environments

Package machine learning code, dependencies, and CUDA runtime configurations into standardized Docker containers. Can run concurrently with modeling phases.

  • Containerize a Python ML application with Docker and multi-stage builds
    ~6hBuild1 resource

    Containers guarantee environment reproducibility between local development, CI runners, and cloud clusters.

    You'll learn

    • Multi-stage builds — Docker pattern to minimize final image size by discarding build tools
    • Docker layer caching — ordering Dockerfile instructions to speed up builds
    • .dockerignore — excluding virtual environments and heavy datasets from build context

    Write a clean, secure Dockerfile for an ML application using multi-stage builds, non-root user permissions, and dependency caching layers.

    Done when: The Docker container builds cleanly, weighs under 500MB, and executes inference requests inside the isolated container.

    How to work through it

    1. Create a Dockerfile based on an official Python slim base image
    2. Implement multi-stage building to separate build tooling from final runtime
    3. Create a non-root system user inside the container for security
    4. Build, tag, and run the Docker image mapping local ports
  • Orchestrate multi-container local stack with Docker Compose
    ~6hBuild1 resource

    Real ML systems require multi-service orchestration for databases, caches, and serving engines.

    You'll learn

    • Docker Compose — tool for defining and running multi-container Docker applications
    • Container Networking — internal DNS resolution allowing containers to talk by service name
    • Healthchecks — automated container status checks determining readiness

    Configure a docker-compose.yml that orchestrates an ML API container, a Redis caching instance, and a PostgreSQL database on a shared internal network with health checks.

    Done when: Running docker compose up spins up all 3 services, passes health checks, and allows the API to read and write to the database and cache.

    How to work through it

    1. Write docker-compose.yml defining API, Redis, and Postgres services
    2. Configure environment variables and internal network bridges
    3. Add container health checks and restart policies
    4. Verify inter-container communication using service hostnames
8

High-Performance Model Serving & APIs

Expose trained models as production web services using FastAPI, asynchronous request handling, batching, and ONNX Runtime optimization.

  • Build a production-ready REST inference API with FastAPI and Pydantic
    ~8hBuild1 resource

    FastAPI is the industry standard framework for exposing ML models via clean REST APIs.

    You'll learn

    • Pydantic — data validation and parsing library utilizing Python type annotations
    • Lifespan Events — loading heavy ML models into memory once at server startup
    • Asynchronous I/O — non-blocking concurrency handling multiple requests simultaneously

    Construct an asynchronous FastAPI service to serve model predictions with strict input validation via Pydantic schemas, exception handlers, and auto-generated Swagger documentation.

    Done when: The API handles concurrent requests, validates malformed inputs returning HTTP 422 errors, and responds with predictions under 20ms p95 latency.

    How to work through it

    1. Define strict Pydantic schemas for input request features and output probabilities
    2. Load model weights during application startup using FastAPI lifespan events
    3. Implement POST /predict endpoint with asynchronous request handling
    4. Add custom exception handlers and structured logging
  • Optimize model inference latency using ONNX and dynamic batching
    ~8hBuild1 resource

    Production ML engineers must minimize compute costs and meet tight latency SLAs in serving systems.

    You'll learn

    • ONNX (Open Neural Network Exchange) — open ecosystem for interoperable AI models
    • ONNX Runtime — high-performance cross-platform engine for model inference
    • Dynamic Batching — grouping individual incoming requests into batches to maximize throughput

    Export PyTorch/Scikit-Learn models to ONNX format, configure ONNX Runtime with CPU/GPU execution providers, and benchmark latency throughput gains against standard PyTorch.

    Done when: The ONNX Runtime serving pipeline achieves at least a 2x inference speedup and lower memory footprint compared to native PyTorch runtime.

    How to work through it

    1. Export trained PyTorch model to open .onnx format via torch.onnx.export
    2. Verify ONNX computational graph using onnx.checker
    3. Initialize ONNX Runtime inference session with CPU/CUDA execution providers
    4. Benchmark throughput (requests/second) under concurrent load using Locust
9

Experiment Tracking & Model Registry (MLOps Core)

Implement systematic experiment tracking, parameter versioning, artifact storage, and model lifecycle governance using MLflow or Weights & Biases.

  • Instrument training pipelines with MLflow experiment tracking
    ~6hBuild1 resource

    Systematic experiment logging replaces ad-hoc spreadsheet tracking and guarantees experiment reproducibility.

    You'll learn

    • MLflow Tracking — API and UI for logging parameters, code versions, metrics, and artifacts
    • Artifact Store — object storage (S3, GCS, or local filesystem) holding model binaries
    • Model Artifact — serializable file containing learned parameters and metadata

    Integrate MLflow into training scripts to automatically track parameters, hyperparameter search runs, evaluation metrics curves, and save model artifacts.

    Done when: You can view a clean MLflow UI dashboard comparing at least 10 training runs with metric plots and saved artifact binaries.

    How to work through it

    1. Set up local or remote MLflow tracking server with backend and artifact store
    2. Wrap training code with mlflow.start_run() context managers
    3. Log hyperparameters, metrics per epoch, and ROC/PR curve plots
    4. Log model binaries using mlflow.pytorch.log_model or sklearn flavor
  • Implement Model Registry lifecycle stages and automated promotion
    ~6hBuild1 resource

    Model registries establish formal governance, preventing accidental deployment of inferior models.

    You'll learn

    • Model Registry — centralized store for collaborative model management and versioning
    • Model Lineage — audit trail tracking model origin, training code, and dataset version
    • Model Governance — formal rules controlling deployment readiness

    Register candidate models in the MLflow Model Registry, assign semantic versioning tags, and build an automated script that promotes models across Staging and Production based on validation metric thresholds.

    Done when: The automated gate promotes a model to 'Production' alias only if its accuracy surpasses the currently active production model.

    How to work through it

    1. Register trained model artifacts to the MLflow Model Registry
    2. Assign metadata tags, descriptions, and schema definitions
    3. Write automated promotion script comparing candidate metrics with production baseline
    4. Transition model alias/stage to Production upon passing test gates
10

Automated Pipelines, CI/CD & Orchestration

Automate data workflows and deployment pipelines using workflow orchestrators (Prefect/Airflow) and GitHub Actions CI/CD.

  • Build an automated ML training pipeline with Prefect
    ~8hBuild1 resource

    Workflow orchestrators replace brittle cron jobs with robust scheduling, dependency graphs, and state monitoring.

    You'll learn

    • DAG (Directed Acyclic Graph) — mathematical structure representing execution task workflows
    • Prefect — modern Python workflow orchestration and scheduling engine
    • Idempotency — property where an operation can be applied multiple times without changing the result

    Construct an orchestrator DAG (Directed Acyclic Graph) using Prefect tasks and flows that ingests data, validates schema, trains the model, and logs artifacts with automatic retries.

    Done when: The Prefect flow runs end-to-end on a scheduled trigger, handles transient task failures with retries, and notifies on completion.

    How to work through it

    1. Define modular pipeline steps using @task decorators
    2. Assemble full DAG workflow using the @flow decorator
    3. Configure task retry policies and caching mechanisms
    4. Deploy and run the flow using a local Prefect worker
  • Build a GitHub Actions CI/CD pipeline for model validation and container deployment
    ~8hBuild1 resource

    Continuous integration and deployment ensure that broken code or degraded models never reach production environments.

    You'll learn

    • GitHub Actions — automated CI/CD platform built into GitHub repositories
    • Smoke Testing — minimal test run verifying that critical software functions work
    • Continuous Delivery (CD) — automated release practice keeping code releasable at any moment

    Create a GitHub Actions workflow that triggers on pull requests to run unit tests, check code linting, run model smoke tests, build the Docker container, and push to a container registry.

    Done when: Pushing a commit to GitHub triggers the automated runner, passes all tests, and successfully builds the container image.

    How to work through it

    1. Create .github/workflows/ci.yml configuration file
    2. Configure jobs for Ruff linting, Pytest execution, and model sanity checks
    3. Configure Docker build and push action using repository secrets
    4. Add status badge to repository README reflecting pipeline health
11

Production Monitoring, Observability & Drift Detection

Implement data and concept drift detection, telemetry logging, and alerting systems for deployed models using Evidently AI and Prometheus.

  • Implement Data Drift and Concept Drift monitoring with Evidently AI
    ~8hBuild1 resource

    Real-world data changes over time, causing silent model accuracy degradation unless monitored proactively.

    You'll learn

    • Covariate Shift — change in the distribution of input features $P(X)$ over time
    • Concept Drift — change in the statistical relationship between features and target labels $P(Y|X)$
    • Kolmogorov-Smirnov Test — non-parametric test comparing continuous distributions

    Construct an automated drift analysis service comparing production live inference payloads against baseline training data distributions using statistical tests (Kolmogorov-Smirnov, Wasserstein distance).

    Done when: The drift pipeline detects synthetic data corruption/distribution shifts in production traffic and generates an HTML/JSON drift diagnostic report.

    How to work through it

    1. Collect baseline reference dataset from training split
    2. Simulate shifted production feature distributions (covariate shift)
    3. Run Evidently AI Drift Report calculating statistical distance per feature
    4. Set alert triggers when drift score exceeds defined threshold
  • Instrument model serving metrics with Prometheus and Grafana
    ~8hBuild1 resource

    Real-time metrics allow instant detection of server outages, latency spikes, and system health degradation.

    You'll learn

    • Prometheus — time-series monitoring and alerting database standard in cloud infrastructure
    • Grafana — multi-platform analytics and interactive dashboard visualization web app
    • Latency Percentiles (p95/p99) — metric showing response time for the slowest 5% or 1% of requests

    Add Prometheus metrics instrumentation to your FastAPI model server tracking prediction latency histograms, request counts, prediction value distributions, and error rates, then visualize them on a Grafana dashboard.

    Done when: You can view real-time Grafana dashboards displaying live requests/sec, p95/p99 inference latency, and error code rates under active load.

    How to work through it

    1. Install prometheus-fastapi-instrumentator in the FastAPI app
    2. Expose /metrics endpoint with custom histogram and counter metrics
    3. Configure local Prometheus instance to scrape the API endpoint
    4. Import and configure Grafana dashboard visualizing live inference telemetry
12

Capstone Project & Technical Interview Preparation

Synthesize every engineering and modeling competency into a single end-to-end production system and prepare for MLE hiring evaluations.

  • Build and document an end-to-end production ML system capstone
    ~16hBuild1 resource

    A comprehensive end-to-end system repository provides indisputable proof of your engineering and modeling capabilities.

    You'll learn

    • System Architecture Design — end-to-end structural mapping of interconnected software services
    • Technical Documentation — clear communication of system trade-offs, schemas, and instructions
    • Reproducible Build — ensuring anyone can spin up the full stack from a clean git clone

    Design, build, and deploy an end-to-end ML application featuring automated Prefect data ingestion, MLflow tracking, containerized FastAPI serving, CI/CD deployment, and Evidently drift monitoring.

    Done when: The project repository contains a complete architectural diagram, passing CI tests, reproducible Docker instructions, and a live deployed demonstration endpoint.

    How to work through it

    1. Define business problem, data ingestion pipeline, and validation gates
    2. Train and register an optimized model with experiment tracking
    3. Build containerized REST API with input validation and Prometheus metrics
    4. Set up CI/CD pipeline and write comprehensive system architecture documentation
  • Drill ML system design scenarios and coding interviews
    ~12hPractice1 resource

    ML Engineer hiring heavily tests your ability to navigate architectural trade-offs under real-world constraints.

    You'll learn

    • ML System Design Framework — structured approach to breaking down open-ended architecture problems
    • Cold Start Problem — handling inference for new users/items with zero historical interactions
    • Online vs Offline Features — trade-offs between pre-computed batch features and real-time streaming features

    Practice designing scalable ML systems for classic industry interview scenarios (e.g., recommendation feed, search ranking, fraud detection) focusing on data pipelines, latency, scale, and failure modes.

    Done when: You can whiteboard or write a structured 45-minute ML system design breakdown covering requirements, data flow, modeling trade-offs, serving architecture, and monitoring.

    How to work through it

    1. Study standard ML system design frameworks (requirements, data, modeling, serving, monitoring)
    2. Draft end-to-end designs for a Real-Time Recommendation System
    3. Draft end-to-end designs for an Ad Click-Through Rate (CTR) Prediction Service
    4. Review Python data structures, algorithms, and tensor manipulation coding drills

How the plan fits together

12 phases in 6 stages. Anything on the same row can be worked on at the same time.

An arrow points from a phase to the work it unlocks: before starting any phase, every phase with an arrow into it has to be finished first.

STARTSTAGE 2STAGE 3STAGE 4STAGE 5STAGE 61Programming & SoftwareEngineering Foundations3 tasks · ~18h2Applied Mathematics &Vectorized Computing3 tasks · ~26h3Data Engineering & TabularWrangling2 tasks · ~16h4Classical Machine Learning& Validation3 tasks · ~24h5Deep Learning Fundamentals& PyTorch2 tasks · ~18h6Sequence Models,Transformers & LLMFundamentals2 tasks · ~20h7Containerization &Reproducible Environments2 tasks · ~12h8High-Performance ModelServing & APIs2 tasks · ~16h9Experiment Tracking &Model Registry (MLOpsCore)2 tasks · ~12h10Automated Pipelines, CI/CD& Orchestration2 tasks · ~16h11Production Monitoring,Observability & DriftDetection2 tasks · ~16h12Capstone Project &Technical InterviewPreparation2 tasks · ~28h
Solid arrow
Must be finished before the phase it points to
Dashed arrow
Same rule, but the prerequisite sits more than one stage back

Resources

17 in this plan's library, beyond the links on individual tasks.

Learning & Reference

Core textbooks, documentation, and conceptual guides.

  • Designing Machine Learning Systems

    A comprehensive reference for end-to-end ML system design, feature storage, serving paradigms, and interview prep.

    huyenchip.com · O'Reilly Media · Book · Paid book (available via O'Reilly Learning subscription and book retailers); accompanying chapter outlines and references open-sourced by author · Advanced

  • Docker Documentation: Getting Started & Container Concepts

    Teaches container isolation, multi-stage builds, dependency locking, and configuring GPU base environments.

    docs.docker.com · Docker Inc. · Documentation · 100% Free · Beginner to Intermediate

  • Evidently AI Documentation & ML Observability Course

    Focuses on post-deployment monitoring, statistical drift tests, data quality checks, and telemetry integration.

    docs.evidentlyai.com · Evidently AI · Documentation · 100% Free open-source documentation and self-paced course · Intermediate

  • Learn PyTorch for Deep Learning: Zero to Mastery

    Guides learners through tensors, computational graphs, modular script organization, and GPU acceleration in PyTorch.

    learnpytorch.io · Daniel Bourke · Book · Free online book and open-source notebooks; optional paid video platform subscription · Intermediate

  • Machine Learning in Python with scikit-learn (MOOC)

    Teaches predictive modeling pipelines, preprocessing, cross-validation, and classical ML algorithms with strict validation standards.

    inria.github.io · Inria · Course · 100% Free to read and execute interactive notebooks; free platform access via FUN-MOOC · Beginner to Intermediate

  • Mathematics for Machine Learning

    Covers linear algebra, calculus, and optimization to prepare you for implementing operations with NumPy.

    mml-book.com · Cambridge University Press · Book · Free digital PDF version hosted by authors; paid print copy available · Intermediate

  • MLOps Zoomcamp

    A project-based course teaching workflow orchestration, automated training pipelines, CI/CD, and deployment.

    DataTalks.Club · Course · 100% Free (Self-paced materials, assignments, and cohort repository freely available) · Intermediate to Advanced

  • The Hugging Face NLP Course

    Covers sequence modeling, tokenizers, Transformers, transfer learning, and PEFT/LoRA fine-tuning.

    huggingface.co · Hugging Face · Course · 100% Free · Intermediate to Advanced

  • The Missing Semester of Your CS Education

    Bridges the gap between basic syntax and software craftsmanship by covering UNIX shell, Git, build systems, and debugging tools.

    missing.csail.mit.edu · Massachusetts Institute of Technology (MIT) · MIT OpenCourseWare · 100% Free (Open courseware notes, assignments, and video lectures) · Beginner

Open Source Frameworks

Essential libraries and tools for training and serving.

  • FastAPI Documentation & User Guide

    Teaches how to write asynchronous, high-throughput Python inference microservices with typing validation and dependency injection.

    fastapi.tiangolo.com · Sebastián Ramírez · Documentation · 100% Free · Intermediate

  • FastAPI Official Documentation

    Essential reference for building asynchronous, production-ready inference endpoints with automated validation and documentation.

    fastapi.tiangolo.com · Sebastián Ramírez · Documentation · Free · Intermediate

  • MLflow Official Documentation & Tutorials

    Provides guides on hyperparameter logging, metric visualization, artifact tracking, and model registry management.

    mlflow.org · Linux Foundation / Databricks · Documentation · 100% Free · Intermediate

  • ONNX Runtime Documentation

    Explains how to export models to ONNX format, configure hardware execution providers, and achieve low-latency predictions.

    onnxruntime.ai · Microsoft & The Linux Foundation AI & Data · Documentation · 100% Free · Intermediate to Advanced

  • Polars User Guide & Documentation

    Teaches high-performance DataFrame manipulation, lazy query optimization, and memory-efficient streaming.

    docs.pola.rs · Polars BV / Polars Open Source Project · Documentation · 100% Free · Intermediate

  • PyTorch Official Tutorials

    Practical hands-on guide covering tensors, automatic differentiation with autograd, and constructing modular neural network modules.

    pytorch.org · PyTorch Foundation · Interactive Documentation · Free · Intermediate

  • Scikit-Learn User Guide

    The gold standard reference for classical algorithms, validation curves, cross-validation protocols, and preprocessing pipelines.

    scikit-learn.org · Scikit-Learn · Documentation · Free · Intermediate

Portfolio & Career

Project repositories, system design templates, and interview prep.

  • Made With ML

    Demonstrates taking an ML application from design to deployment, providing an ideal blueprint for a capstone portfolio project.

    madewithml.com · Goku Mohandas · Course · 100% Free online lessons and GitHub code templates · Advanced