Learn Machine Learning

You'll learn

• Vector norm — mathematical measure of vector length and magnitude
• Dot product — algebraic operation returning the cosine similarity scaled by vector magnitudes
• Eigendecomposition — factorization of a matrix into canonical eigenvalues and eigenvectors
• Singular Value Decomposition (SVD) — matrix factorization generalizing eigendecomposition to non-square matrices

How to work through it

1. 1Review vector arithmetic, dot products, vector norms, and orthogonality
2. 2Study matrix transformations, rank, null space, and determinants
3. 3Calculate eigenvalues and eigenvectors for 2x2 and 3x3 matrices
4. 4Work through the geometric intuition of singular value decomposition (SVD)

You'll learn

• Gradient vector — vector of partial derivatives pointing in the direction of steepest ascent
• Jacobian matrix — matrix of all first-order partial derivatives of a vector-valued function
• Hessian matrix — square matrix of second-order partial derivatives describing local curvature
• Convexity — property where any local minimum is guaranteed to be a global minimum

How to work through it

1. 1Review single-variable derivatives and rules of differentiation
2. 2Compute partial derivatives for multivariate scalar fields
3. 3Derive the multi-variable chain rule and formulate gradient vectors
4. 4Construct Jacobian and Hessian matrices and analyze second-order convexity conditions

You'll learn

• Learning rate — scalar step size used to update parameters along the negative gradient
• Newton-Raphson method — second-order optimization using curvature information from the inverse Hessian
• Stochastic Gradient Descent (SGD) — gradient descent approximating the gradient over a single sample or mini-batch

How to work through it

1. 1Derive first-order gradient descent parameter updates from Taylor approximations
2. 2Analyze the impact of learning rate magnitudes on convergence
3. 3Study second-order Newton-Raphson optimization and Hessian inversion cost
4. 4Contrast batch gradient descent with stochastic and mini-batch sampling

You'll learn

• Gaussian elimination — algorithm for solving systems of linear equations via row reduction
• Finite differences — numerical approximation of derivatives using small delta steps

How to work through it

1. 1Create a Matrix class supporting addition, scalar multiplication, and dot products
2. 2Implement matrix transposition and row reduction algorithms
3. 3Implement Gaussian elimination to solve systems of linear equations Ax = b
4. 4Write a numerical gradient check function using finite differences

You'll learn

• Bayes' Theorem — mathematical formula for determining conditional probability based on prior knowledge
• Probability Density Function (PDF) — function specifying probability per unit length of a continuous random variable
• Covariance — statistical measure of the joint variability of two random variables

How to work through it

1. 1Define probability axioms, sample spaces, and independence
2. 2Derive and apply Bayes' Theorem for conditional hypothesis testing
3. 3Study discrete distributions (Bernoulli, Binomial) and continuous distributions (Gaussian, Uniform)
4. 4Derive mathematical definitions of variance, standard deviation, and covariance

You'll learn

• Maximum Likelihood Estimation (MLE) — method of estimating parameters of an assumed probability distribution given observed data
• Log-likelihood — logarithm of the likelihood function used to transform products of probabilities into sums
• Maximum A Posteriori (MAP) — Bayesian estimation method incorporating a prior probability distribution over parameters

How to work through it

1. 1Formulate likelihood and log-likelihood functions for independent and identically distributed (i.i.d.) data
2. 2Derive the analytical MLE for the Gaussian distribution parameter set
3. 3Derive the analytical MLE for the Bernoulli parameter set
4. 4Contrast MLE with Maximum A Posteriori (MAP) estimation incorporating prior distributions

You'll learn

• Broadcasting — NumPy mechanism allowing arithmetic operations on arrays of different shapes
• Vectorization — replacing explicit scalar loops with batch operations executed in compiled C
• ndarray — core n-dimensional array data structure in NumPy with contiguous memory allocation

How to work through it

1. 1Create and manipulate ndarrays with explicit data types (dtypes)
2. 2Master NumPy broadcasting rules across mismatched array dimensions
3. 3Apply fancy boolean indexing and multi-axis reductions (sum, mean, max)
4. 4Benchmark vectorized matrix routines against naive nested Python loops

You'll learn

• Synthetic data generation — creating controlled datasets with known ground truth parameters
• Log-likelihood surface — multi-dimensional geometric surface representing parameter likelihood

How to work through it

1. 1Generate synthetic data drawn from univariate and bivariate Gaussian distributions
2. 2Implement the vectorized log-likelihood function for Gaussian and Bernoulli models
3. 3Find the parameter estimates analytically and via numerical optimization using scipy.optimize
4. 4Plot the log-likelihood optimization path using Matplotlib

You'll learn

• Normal Equation — closed-form algebraic solution for the weights in linear regression
• Mean Squared Error (MSE) — loss function measuring the average squared difference between predictions and targets
• Residuals — difference between observed target values and predicted model values

How to work through it

1. 1Set up the matrix formulation of linear regression Y = Xw + epsilon
2. 2Derive the residual sum of squares (RSS) loss function in matrix notation
3. 3Differentiate the loss with respect to weight vector w and solve for normal equations
4. 4Derive the step-by-step batch gradient descent update rule for linear regression

You'll learn

• Ridge Regression (L2) — regularization adding a squared magnitude penalty to the loss function
• Lasso Regression (L1) — regularization adding an absolute magnitude penalty encouraging sparsity
• Multicollinearity — condition where independent variables in a regression model are highly correlated

How to work through it

1. 1Analyze ill-conditioned matrix inversion and multicollinearity in unregularized OLS
2. 2Derive the Ridge regression closed-form solution (X^T X + lambda I)^(-1) X^T y
3. 3Examine L1 sub-gradient optimization and coordinate descent for Lasso
4. 4Compare L1 and L2 penalty constraints geometrically using level curves

You'll learn

• Sigmoid function — S-shaped activation function mapping real values into probabilities between 0 and 1
• Binary Cross-Entropy — loss function measuring divergence between two probability distributions for binary outcomes
• Decision boundary — hyper-surface separating different predicted class regions in feature space

How to work through it

1. 1Define the logistic sigmoid function and its mathematical derivative
2. 2Formulate binary classification likelihood assuming a Bernoulli output distribution
3. 3Derive the Negative Log-Likelihood (Binary Cross-Entropy) loss function
4. 4Compute the analytical gradient of cross-entropy with respect to the weight vector

You'll learn

• Estimator API — design pattern standardizing model fitting, transformation, and prediction interfaces
• Gradient step verification — checking code correctness by comparing analytical and numerical gradients

How to work through it

1. 1Implement a base Estimator class with fit, predict, and score methods
2. 2Implement LinearRegression with both analytical normal equations and SGD optimizers
3. 3Implement RidgeRegression with regularized closed-form inversion
4. 4Implement LogisticRegression with vectorized gradient descent and binary threshold prediction
5. 5Write unit tests validating outputs against scikit-learn baselines

You'll learn

• Bias — error introduced by approximating a real-world problem with an overly simplistic model
• Variance — error introduced by model sensitivity to small fluctuations in training data
• Irreducible error — intrinsic noise in the data generation process that cannot be eliminated by any model

How to work through it

1. 1Write the mathematical definition of expected test error for a noisy target y = f(x) + epsilon
2. 2Expand and decompose the expectation into irreducible variance, model bias squared, and model variance
3. 3Map underfitting and overfitting states to high-bias and high-variance regimes
4. 4Plot simulated learning curves illustrating training error vs. validation error as sample size increases

You'll learn

• Data leakage — unintended transfer of information from the test set or future data into the training process
• Stratified K-Fold — cross-validation partitioning that preserves identical class label proportions across folds
• Target leakage — inclusion of features that duplicate or reveal the target variable during training

How to work through it

1. 1Analyze k-fold, stratified k-fold, and leave-one-out cross-validation mechanics
2. 2Study out-of-fold estimation and cross-validated score variance
3. 3Identify pre-processing data leakage (scaling, imputing, feature selecting before splitting)
4. 4Analyze temporal and group leakage in sequential or clustered datasets

You'll learn

• Confusion Matrix — table visualizing performance of a classification algorithm across true and false predictions
• ROC-AUC — metric measuring discrimination ability across all classification probability thresholds
• Precision-Recall curve — metric curve evaluating classification performance under severe class imbalance

How to work through it

1. 1Construct a confusion matrix and derive True Positive Rate, False Positive Rate, Precision, and Recall
2. 2Derive the harmonic mean formula for the F1-score and its generalized F-beta formulation
3. 3Trace the step-by-step algorithm for calculating ROC curves and ROC-AUC scores
4. 4Compare Mean Absolute Error (L1 loss) and Mean Squared Error (L2 loss) sensitivity to outliers

You'll learn

• Scikit-learn Pipeline — utility assembling several steps that can be cross-validated together while setting different parameters
• Bootstrap confidence interval — non-parametric method of estimating metric uncertainty via repeated resampling with replacement

How to work through it

1. 1Construct a scikit-learn Pipeline encapsulating feature scalers, encoders, and classifiers
2. 2Implement a stratified k-fold validation loop executing strictly within the pipeline
3. 3Calculate bootstrap confidence intervals for accuracy, F1, and AUC metrics
4. 4Plot cross-validated ROC curves with variance bands

You'll learn

• Curse of Dimensionality — phenomenon where data becomes sparse in high-dimensional space, degrading distance metrics
• k-d tree — space-partitioning data structure for organizing points in a k-dimensional space

How to work through it

1. 1Study distance metrics: Euclidean (L2), Manhattan (L1), and Cosine distance
2. 2Derive the mathematical geometry of hyper-spheres inscribed in hyper-cubes in high dimensions
3. 3Analyze the computational complexity of brute-force nearest-neighbor searches versus k-d trees
4. 4Evaluate the sensitivity of distance-based models to unscaled feature variables

You'll learn

• Gini Impurity — measure of how often a randomly chosen element from the set would be incorrectly labeled
• Shannon Entropy — quantitative measure of the average uncertainty or information content in a random variable
• Information Gain — reduction in entropy achieved by partitioning data according to a specific feature

How to work through it

1. 1Derive Shannon Entropy H(S) = - sum p_i log2(p_i) and Information Gain formulas
2. 2Derive Gini Impurity G(S) = 1 - sum (p_i)^2 and contrast with entropy
3. 3Understand greedy recursive splitting algorithms (CART and ID3)
4. 4Analyze stopping criteria, minimum leaf samples, maximum depth, and cost-complexity pruning

You'll learn

• Bagging (Bootstrap Aggregation) — ensemble technique training multiple base models on bootstrap subsets to reduce variance
• Random Forest — ensemble of decision trees utilizing random feature subsets to decorrelate individual trees
• Gradient Boosting — sequential ensemble technique fitting new base learners to the pseudo-residuals of previous models

How to work through it

1. 1Analyze variance reduction achieved by averaging independent estimators
2. 2Study Random Forest decorrelation via random feature subset selection at split points
3. 3Derive AdaBoost weight updates and exponential loss optimization
4. 4Derive Gradient Boosting parameter updates using pseudo-residuals as target gradients

You'll learn

• Recursive partitioning — algorithmic technique of continuously splitting data into smaller subsets
• Pseudo-residuals — negative gradient of the loss function with respect to current ensemble predictions

How to work through it

1. 1Implement a Node class storing split feature index, threshold value, and leaf predictions
2. 2Implement recursive best-split search minimizing mean squared error
3. 3Build a DecisionTreeRegressor class with configurable max_depth and min_samples_split
4. 4Build a GradientBoostedRegressor that iteratively fits regression trees to residual errors
5. 5Train the scratch boosted model on a non-linear sinusoidal function

You'll learn

• Support Vectors — training data points lying on or inside the margin boundary that uniquely define the decision boundary
• Slack variable — penalty variable permitting margin violations in non-linearly separable datasets
• Margin — perpendicular distance between the decision hyperplane and the closest data points

How to work through it

1. 1Define hyperplane geometry and geometric distance from a data point to a hyperplane
2. 2Formulate the hard-margin SVM optimization problem as a quadratic program
3. 3Introduce slack variables (xi) to derive the soft-margin SVM with penalty parameter C
4. 4Analyze the role of the C hyperparameter in controlling margin width vs. training errors

You'll learn

• Lagrangian Duality — mathematical transformation converting a constrained primal problem into an equivalent dual optimization problem
• KKT Conditions — first-order necessary conditions for a solution in non-linear programming to be optimal
• Kernel Trick — method of computing dot products in high-dimensional feature spaces implicitly using kernel functions
• Radial Basis Function (RBF) — kernel function measuring similarity based on squared Euclidean distance

How to work through it

1. 1Formulate the Lagrangian function for the primal SVM optimization problem
2. 2Apply KKT conditions and differentiate with respect to primal variables (w, b)
3. 3Substitute optimal conditions back into the Lagrangian to construct the dual objective function
4. 4Analyze Mercer's conditions and mathematical formulas for RBF and Polynomial kernels

You'll learn

• Gram matrix — matrix of all possible pairwise inner products between sample vectors in feature space
• Quadratic Programming — optimization method for minimizing a quadratic objective function subject to linear constraints

How to work through it

1. 1Set up the Gram matrix using vectorized linear and RBF kernel functions
2. 2Formulate the quadratic programming matrices (P, q, G, h, A, b) required by cvxopt
3. 3Solve for Lagrange multipliers (alpha) and extract non-zero support vectors
4. 4Compute the intercept parameter b and implement the decision function for new points
5. 5Visualize the decision boundary and highlighted support vectors using Matplotlib

You'll learn

• Principal Component Analysis (PCA) — orthogonal linear transformation transforming data into linearly uncorrelated variables
• Explained Variance Ratio — proportion of the dataset's total variance accounted for by each principal component
• Reconstruction error — squared difference between original high-dimensional data and its low-rank approximation

How to work through it

1. 1Formulate the variance maximization objective for a 1D linear projection
2. 2Apply Lagrange multipliers with unit vector constraint ||u|| = 1 to derive the eigenvector equation
3. 3Relate the data matrix SVD (X = U Sigma V^T) directly to the covariance matrix decomposition
4. 4Calculate the proportion of explained variance from singular values

You'll learn

• k-Means — unsupervised clustering algorithm partitioning n observations into k clusters
• k-means++ — initialization algorithm choosing initial seeds with probability proportional to squared distance from nearest existing center
• Silhouette Score — metric evaluating clustering quality by measuring cohesion within clusters versus separation from other clusters

How to work through it

1. 1Formulate the k-Means objective function (Within-Cluster Sum of Squares)
2. 2Derive the two-step iterative optimization: cluster assignment step and centroid update step
3. 3Analyze k-means++ probabilistic distance-weighted initialization
4. 4Study agglomerative hierarchical clustering distance link criteria (single, complete, average, Ward)

You'll learn

• Gaussian Mixture Model (GMM) — probabilistic model assuming all data points are generated from a mixture of finite Gaussian distributions
• Expectation-Maximization (EM) — iterative optimization algorithm for finding maximum likelihood estimates in latent variable models
• Responsibility — posterior probability that a specific mixture component generated a given observation

How to work through it

1. 1Define the generative probability model for Gaussian mixtures with latent indicator variables
2. 2Analyze why standard MLE fails on mixture distributions due to sums inside the logarithm
3. 3Derive the Expectation step (calculating responsibilities / posterior class probabilities)
4. 4Derive the Maximization step (updating means, covariances, and mixing coefficients)

You'll learn

• Latent variable — variable that is not directly observed but inferred through mathematical modeling
• Log-likelihood convergence — monitoring the increase in log-likelihood across EM iterations to detect convergence

How to work through it

1. 1Implement PCA class calculating mean centering, SVD decomposition, and transform/inverse_transform methods
2. 2Implement GMM class with randomized/k-means++ initialization of means and covariances
3. 3Write the vectorized E-step computing Gaussian densities and responsibilities
4. 4Write the vectorized M-step updating parameters and track log-likelihood convergence
5. 5Plot cluster confidence ellipses for fitted Gaussian components

You'll learn

• Universal Approximation Theorem — theorem stating that a feedforward network with a single hidden layer can approximate any continuous function
• ReLU (Rectified Linear Unit) — activation function defined as f(x) = max(0, x)
• Softmax — function that normalizes a vector of K real numbers into a probability distribution over K classes

How to work through it

1. 1Examine the architecture of Multi-Layer Perceptrons (MLPs) with dense weight layers and biases
2. 2Study non-linear activation functions and their mathematical derivatives
3. 3Analyze the vanishing gradient problem in deep saturating networks (Sigmoid/Tanh)
4. 4Derive the multi-class Softmax activation function and categorical cross-entropy loss

You'll learn

• Backpropagation — algorithm for calculating gradients of composite functions using the chain rule along a computational graph
• Computational graph — directed acyclic graph where nodes represent variables or mathematical operations
• Reverse-mode differentiation — algorithmic technique computing derivatives of scalar outputs with respect to all inputs in a single backward pass

How to work through it

1. 1Represent composite mathematical functions as directed acyclic computational graphs
2. 2Contrast forward-mode and reverse-mode automatic differentiation computational complexity
3. 3Derive layer-wise backward error delta vectors using the multi-variable chain rule
4. 4Derive weight gradient outer products dL/dW = delta * a^(l-1)^T and bias gradients

You'll learn

• Adam (Adaptive Moment Estimation) — optimization algorithm combining momentum and adaptive learning rates
• Batch Normalization — technique normalizing layer inputs across a mini-batch to stabilize training
• Dropout — regularization method randomly zeroing a fraction of activation units during training

How to work through it

1. 1Analyze momentum and Nesterov accelerated momentum mechanics
2. 2Derive RMSprop exponential moving average of squared gradients
3. 3Derive the complete Adam optimizer update rule with bias corrections
4. 4Examine Batch Normalization forward transform and its effect on internal covariate shift

You'll learn

• Automatic differentiation engine — software subsystem that dynamically builds execution graphs to compute gradients
• MNIST — standard benchmark dataset of 70,000 28x28 grayscale images of handwritten digits

How to work through it

1. 1Implement a Tensor class that stores data, grad, and references to creator operations
2. 2Implement backward methods for addition, matrix multiplication, ReLU, and Cross-Entropy
3. 3Create modular Linear, Sequential, and ReLU layer abstractions
4. 4Implement the Adam optimizer class with moving average state buffers
5. 5Train and evaluate your framework on the MNIST digit classification benchmark

You'll learn

• torch.nn.Module — base class for all neural network modules in PyTorch
• DataLoader — PyTorch utility providing batching, shuffling, and multi-process data loading
• Autograd in PyTorch — PyTorch's automatic differentiation engine that records operations on tensors

How to work through it

1. 1Learn torch.Tensor manipulation, device placement (.to(device)), and autograd mechanics
2. 2Subclass torch.utils.data.Dataset and configure batching with DataLoader
3. 3Build neural network architectures by subclassing torch.nn.Module
4. 4Write decoupled training, validation, and early stopping loops

You'll learn

• Convolutional Layer — layer applying learnable sliding filters to extract local spatial features
• Receptive field — specific region of the input space that affects a particular unit's activation
• Residual connection — skip connection that adds the input of a layer directly to its output

How to work through it

1. 1Derive the 2D cross-correlation / convolution operation over multi-channel feature maps
2. 2Calculate receptive field expansion through successive convolutional layers
3. 3Analyze Max-Pooling and Average-Pooling dimension reduction mechanics
4. 4Study residual learning and skip connections (identity mappings in ResNet)

You'll learn

• Self-Attention — attention mechanism relating different positions of a single sequence to compute a representation
• Query, Key, Value (Q, K, V) — linear projections used to compute attention weights and aggregate contextual representations
• Multi-Head Attention — running attention in parallel across multiple subspace representation heads

How to work through it

1. 1Analyze the limitations of recurrent architectures (sequential bottleneck)
2. 2Derive the Scaled Dot-Product Attention mathematical formulation
3. 3Study Multi-Head Attention splitting, projection matrices, and concatenation
4. 4Analyze sinusoidal positional encodings and layer normalization placement

You'll learn

• CIFAR-10 — benchmark dataset of 60,000 32x32 color images in 10 classes
• Transformer Encoder — stack of self-attention and feed-forward layers processing bidirectional contextual representations

How to work through it

1. 1Construct a custom ResidualBlock class with 2D convolutions, batch normalization, and skip connections
2. 2Build and train a full ResNet model on the CIFAR-10 image dataset
3. 3Implement a standalone MultiHeadAttention class using torch.matmul and softmax
4. 4Combine attention with feed-forward sub-layers and residual layer normalization into a TransformerEncoder

You'll learn

• Shapley values — average marginal contribution of a feature value across all possible feature coalitions
• Permutation Feature Importance — measure of feature importance based on increase in prediction error after shuffling feature values
• Partial Dependence Plot (PDP) — plot showing the marginal effect of one or two features on the predicted outcome

How to work through it

1. 1Study cooperative game theory axioms: efficiency, symmetry, dummy player, additivity
2. 2Derive Shapley values and the TreeSHAP / KernelSHAP approximations
3. 3Implement Permutation Feature Importance measuring out-of-fold metric degradation
4. 4Construct Partial Dependence Plots to visualize non-linear marginal feature effects

You'll learn

• 5x2 cv paired t-test — statistical test designed specifically for comparing machine learning algorithms with controlled Type I error
• McNemar's test — non-parametric statistical test applied to paired nominal data based on disagreement matrices
• Type I error — false positive conclusion that a statistically significant difference exists when it does not

How to work through it

1. 1Analyze the false positive rate inflation of naive repeated t-tests on cross-validation folds
2. 2Study Dietterich's 5x2 cross-validated paired t-test for comparing learning algorithms
3. 3Formulate McNemar's test for paired nominal classification outcomes
4. 4Apply the non-parametric Wilcoxon signed-rank test across multiple benchmark datasets

You'll learn

• Covariate shift — change in the distribution of the input features P(X) while P(Y|X) remains unchanged
• Concept drift — change in the statistical relationship between features and target P(Y|X) over time
• Population Stability Index (PSI) — metric measuring how much a variable has shifted in distribution between two samples

How to work through it

1. 1Differentiate between covariate shift P(X), label shift P(Y), and concept drift P(Y|X)
2. 2Calculate the Population Stability Index (PSI) across numerical feature bins
3. 3Run two-sample Kolmogorov-Smirnov (KS) tests on continuous distributions
4. 4Analyze strategies for model recalibration and retraining triggers upon detected drift

You'll learn

• Nested cross-validation — validation protocol with separate inner loops for tuning and outer loops for unbiased performance estimation
• Adversarial validation — methodology of testing whether training and test distributions can be distinguished by a binary classifier

How to work through it

1. 1Set up a nested cross-validation pipeline (5 outer folds for evaluation, 3 inner folds for hyperparameter tuning)
2. 2Train and tune Logistic Regression, LightGBM/XGBoost, and a PyTorch MLP on the benchmark dataset
3. 3Perform paired 5x2 cv statistical tests to verify whether tree or deep learning improvements are statistically significant
4. 4Run SHAP explainability analyses to audit feature importance and check for target or proxy leakage
5. 5Inject synthetic covariate drift into the test fold and measure model performance degradation curves