Data Scientist — Competency Roadmap

You'll learn

• Virtual environments — isolated runtime directories preventing dependency collisions across projects
• pip / uv — package installers that fetch and resolve dependencies from PyPI
• VS Code Python interpreter — setting the editor to target your specific virtual environment

How to work through it

1. 1Install Python 3.11+ and VS Code
2. 2Configure a terminal shell and install uv or miniconda
3. 3Create a dedicated virtual environment named 'ds-core'
4. 4Verify package installation and interpreter selection in VS Code

You'll learn

• List comprehensions — concise syntax for transforming and filtering iterable collections
• Dictionary hash maps — key-value lookups with O(1) average time complexity
• Type hints — annotations specifying expected parameter and return types for clarity

How to work through it

1. 1Practise indexing, slicing, and modifying lists and dicts
2. 2Write conditional filtering scripts using list and dict comprehensions
3. 3Build custom functions with default arguments and type hints
4. 4Write unit tests using assert statements to verify outputs

You'll learn

• Object-Oriented Programming (OOP) — structuring code into reusable classes with bundled state and behaviour
• Custom exceptions — raising domain-specific runtime errors to prevent silent data corruption
• Dunder methods — double-underscore methods that hook into Python's built-in operators

How to work through it

1. 1Define classes with attributes, methods, and constructor initialisation
2. 2Implement defensive programming with try/except/raise blocks
3. 3Override standard dunder methods like __len__ and __getitem__
4. 4Write a small modular script importing your custom class

You'll learn

• pytest — automated test framework for writing concise, expressive Python tests
• argparse — standard library module for creating command-line interfaces
• Regular expressions (re module) — pattern matching engine for parsing unstructured text streams

How to work through it

1. 1Create a modular folder structure with an __init__.py package layout
2. 2Implement regex-based log parsing functions
3. 3Write unit tests in a tests/ directory targeting edge cases and malformed lines
4. 4Run pytest to verify all automated checks pass

You'll learn

• Relational Normalization — structuring schemas to eliminate redundancy and maintain integrity
• Foreign Key constraints — relational rules guaranteeing referential integrity between tables
• Aggregation operators — SQL functions like COUNT, SUM, AVG combined with grouping sets

How to work through it

1. 1Install SQLite / DBeaver or connect to PostgreSQL via Docker
2. 2Create normalized tables for an e-commerce data model
3. 3Populate tables with dummy transactional records
4. 4Write aggregations using GROUP BY, ORDER BY, and multiple INNER/LEFT JOINs

You'll learn

• Common Table Expressions (CTEs) — temporary named result sets that make queries readable and modular
• Window Functions (OVER / PARTITION BY) — calculations across a set of table rows related to the current row without collapsing them
• Execution Plans — internal query optimiser trees detailing table scans, index lookups, and join costs

How to work through it

1. 1Write chained WITH statements (CTEs) to structure multi-stage queries
2. 2Implement partition-based window calculations with ROW_NUMBER and DENSE_RANK
3. 3Calculate lead/lag delta metrics across time partitions
4. 4Verify query execution plans using EXPLAIN ANALYZE

You'll learn

• SQLAlchemy — database abstraction library and Object Relational Mapper for Python
• DuckDB — fast in-process analytical SQL database engine optimised for OLAP queries
• Parameterised Queries — separating query structure from user inputs to prevent SQL injection

How to work through it

1. 1Install SQLAlchemy and DuckDB in your Python environment
2. 2Establish an engine connection and connection pooling
3. 3Write parameterised SQL statements using SQLAlchemy text constructs
4. 4Handle connection teardown and query timeouts defensively

You'll learn

• Pandas DataFrames — in-memory two-dimensional labeled tabular structures
• Polars — lightning-fast columnar data frame library written in Rust utilizing lazy evaluation
• Tidy Data Principles — standard tabular layout where each variable is a column and each observation is a row

How to work through it

1. 1Load and inspect diverse tabular formats (CSV, Parquet, JSON)
2. 2Handle missing values using domain-appropriate imputation or deletion
3. 3Perform grouping, aggregations, and reshaping via melt/pivot
4. 4Benchmark operation runtimes between Pandas and Polars

You'll learn

• Grammar of Graphics — principled framework for mapping data features to geometric visual aesthetic attributes
• Plotly Express — high-level declarative interactive graphing library
• Information Density & Chart Junk — visual design concepts ensuring maximum insight with minimal distraction

How to work through it

1. 1Plot univariate distributions using histograms, KDEs, and boxplots
2. 2Visualise bivariate relationships with scatter plots, pairplots, and heatmaps
3. 3Add interactive hover tooltips and dynamic selectors using Plotly
4. 4Apply clean theme formatting, readable axes labels, and color-blind accessible palettes

You'll learn

• Jupyter Notebooks — interactive computational environment combining code, equations, and narrative text
• Data Profiling — automated statistical inspection of data quality, null distributions, and cardinality
• Outlier Detection — identifying anomalies using IQR, Z-scores, and domain boundaries

How to work through it

1. 1Select an authentic, messy open dataset
2. 2Perform comprehensive data quality checks (outliers, skewness, missingness)
3. 3Generate diagnostic visualizations with markdown explanations of findings
4. 4Draft a concluding 1-page executive memo summarizing actionable business takeaways

You'll learn

• NumPy ndarray — multidimensional array data structure supporting vectorised operations in C
• Broadcasting — automatic element-wise alignment of arrays with differing but compatible shapes
• Vector Norms (L1, L2) — geometric measures of vector magnitude and distance

How to work through it

1. 1Study geometric interpretations of vectors, matrices, and linear combinations
2. 2Perform array manipulation, slicing, and broadcasting with NumPy
3. 3Implement matrix multiplication and dot products manually and via np.dot/@
4. 4Benchmark loop-based Python against vectorised NumPy arrays

You'll learn

• Eigenvectors & Eigenvalues — vectors whose direction remains unchanged during a linear transformation, scaled by the eigenvalue
• Principal Component Analysis (PCA) — orthogonal linear transformation finding directions of maximum variance
• Singular Value Decomposition (SVD) — factorisation of a real matrix into singular vectors and singular values

How to work through it

1. 1Calculate covariance matrices from tabular features
2. 2Compute eigenvalues and eigenvectors using np.linalg.eig
3. 3Project data onto principal component vectors
4. 4Compare explained variance ratios against scikit-learn's PCA

You'll learn

• Loss Functions — mathematical representations of prediction error to be minimised
• Gradients — vectors of partial derivatives pointing in the direction of steepest function ascent
• Learning Rate — step-size hyperparameter controlling numerical convergence speed in optimisation

How to work through it

1. 1Review limits, derivatives, partial derivatives, and gradients
2. 2Formulate the Mean Squared Error (MSE) loss function and its partial derivatives
3. 3Write an iterative loop updating parameter weights via gradient steps
4. 4Plot the loss convergence curve across training epochs

You'll learn

• Central Limit Theorem (CLT) — theorem stating that sample means approach a normal distribution as sample size grows
• Probability Density Function (PDF) — function describing the relative likelihood of a continuous random variable
• Expected Value & Variance — fundamental measures of central tendency and dispersion for random variables

How to work through it

1. 1Model probability density (PDF) and cumulative distribution functions (CDF) using scipy.stats
2. 2Simulate sampling distributions using random number generators
3. 3Verify the Central Limit Theorem empirically through repeated sampling
4. 4Evaluate skewness, kurtosis, and distribution fits on real datasets

You'll learn

• p-value — probability of observing data at least as extreme as the actual result, assuming the null hypothesis is true
• Type I and Type II Errors — false positive (alpha) vs false negative (beta) statistical decisions
• Statistical Power (1 - beta) — probability that a test correctly rejects a false null hypothesis

How to work through it

1. 1Formulate null ($H_0$) and alternative ($H_1$) hypotheses clearly
2. 2Run independent and paired Student's t-tests using scipy.stats
3. 3Execute Chi-square tests of independence on categorical cross-tabs
4. 4Construct 95% confidence intervals using standard errors and t-critical values

You'll learn

• Bootstrapping — statistical method estimating sample properties by repeated sampling with replacement
• Permutation Testing — non-parametric significance test constructing null distributions by shuffling group labels
• Non-parametric Statistics — methods that do not rely on assumptions that data belong to any specific parametric family

How to work through it

1. 1Write empirical resampling functions with replacement for bootstrapping
2. 2Generate empirical confidence intervals for medians and trimmed means
3. 3Implement label permutation loops to construct empirical null distributions
4. 4Compare bootstrap confidence intervals against standard analytical formulas

You'll learn

• Data Leakage — sharing information between training and holdout datasets that produces falsely optimistic metrics
• Target Encoding — encoding categorical features with the mean of the target variable using cross-validation smoothing
• Scikit-Learn Pipeline — composite estimator encapsulating preprocessing steps and models into an atomic unit

How to work through it

1. 1Build transformers using FunctionTransformer and BaseEstimator
2. 2Combine heterogeneous column transformations with ColumnTransformer
3. 3Address multicollinearity using Variance Inflation Factors (VIF)
4. 4Chain preprocessing and estimators within a unified sklearn.pipeline.Pipeline

You'll learn

• Bias-Variance Tradeoff — balancing underfitting error from simple assumptions against overfitting error from sensitivity to noise
• L1 (Lasso) vs L2 (Ridge) Regularisation — penalty terms driving coefficients strictly to zero (L1) or shrinking them smoothly (L2)
• ROC-AUC & PR-AUC — threshold-agnostic evaluation curves for binary classification performance

How to work through it

1. 1Fit Linear and Logistic Regression models on transformed tabular data
2. 2Apply Ridge and Lasso regularisation penalties to control overfitting
3. 3Tune regularisation strength using RidgeCV and LogisticRegressionCV
4. 4Evaluate classification metrics: Precision, Recall, F1, ROC-AUC, and PR-AUC

You'll learn

• Ensemble Learning (Bagging) — training multiple models independently on bootstrap samples and averaging their predictions
• Gini Impurity & Information Gain — splitting criteria used to find optimal partitions in decision trees
• Silhouette Coefficient — measure of how similar an object is to its own cluster compared to other clusters

How to work through it

1. 1Train Decision Trees and visualize decision boundaries
2. 2Construct Random Forest ensembles and measure feature importances
3. 3Run K-Means clustering and compute inertia across k-values
4. 4Evaluate clustering density and noise separation using DBSCAN and Silhouette scores

You'll learn

• Nested Cross-Validation — model selection technique preventing optimistic hyperparameter tuning bias
• Optuna — automated hyperparameter optimization framework using Bayesian sampling
• Joblib serialization — fast persistence of Python objects and Scikit-Learn pipelines to disk

How to work through it

1. 1Structure a clean repository with data, src, and notebook directories
2. 2Implement Stratified K-Fold cross-validation
3. 3Perform systematic hyperparameter tuning using Optuna or GridSearchCV
4. 4Serialize final trained pipeline using joblib and document performance on holdout data

You'll learn

• Minimum Detectable Effect (MDE) — smallest relative shift in a metric an experiment is powered to detect
• CUPED (Controlled-experiment Using Pre-Experiment Data) — variance reduction technique using pre-experiment covariates to increase sensitivity
• Sample Ratio Mismatch (SRM) — diagnostic test detecting sample allocation bugs in online experiments

How to work through it

1. 1Calculate sample size requirements using alpha, statistical power, and MDE
2. 2Implement user-level hashing for consistent experimental bucket assignment
3. 3Run variance reduction on pre-experiment continuous metrics using CUPED
4. 4Check for Sample Ratio Mismatch (SRM) using Chi-square goodness-of-fit

You'll learn

• Confounding Variable — variable that influences both the dependent and independent variables, causing a spurious association
• Propensity Score Matching (PSM) — statistical matching technique attempting to estimate causal treatment effects by balancing covariates
• Difference-in-Differences (DiD) — quasi-experimental design comparing before-after changes in treatment vs control groups

How to work through it

1. 1Study Directed Acyclic Graphs (DAGs) and confounding bias
2. 2Estimate propensity scores using logistic regression and perform IPW weighting
3. 3Implement Difference-in-Differences regression interaction models
4. 4Conduct placebo tests to evaluate parallel pre-trends validity

You'll learn

• Gradient Boosting — iterative ensemble technique where each new model predicts the negative gradient of the loss function
• LightGBM — fast gradient boosting framework using histogram-based binning and leaf-wise tree growth
• Class Imbalance Techniques — strategies (cost-sensitive learning, threshold moving) for heavily skewed target distributions

How to work through it

1. 1Study the mathematical derivation of gradient boosting and second-order Taylor approximations
2. 2Implement LightGBM and XGBoost classifiers with native categorical support
3. 3Handle class imbalance using class weights, SMOTE, or scale_pos_weight
4. 4Tune learning rates, subsampling, and tree depth using Optuna

You'll learn

• Shapley Values — method from coalitional game theory allocating payout fairly among contributing features
• SHAP (SHapley Additive exPlanations) — unified framework interpreting predictions by computing marginal feature contributions
• Global vs Local Interpretability — explaining overall model behavior across a dataset vs explaining a single specific prediction

How to work through it

1. 1Study the axiomatic properties of Shapley values from game theory
2. 2Compute TreeSHAP explanations on a trained LightGBM model
3. 3Generate global feature importance summary plots and interaction plots
4. 4Build local prediction breakdown water-fall plots for individual customer cases

You'll learn

• Isolation Forest — unsupervised tree algorithm isolating anomalies based on short path lengths
• Probability Calibration — adjusting model output scores to match empirical real-world probabilities
• Cost-Benefit Utility Analysis — mapping classification thresholds directly to financial dollar returns instead of raw accuracy

How to work through it

1. 1Engineer temporal aggregation and velocity features from transaction streams
2. 2Fit an Isolation Forest to score unlabeled anomalies
3. 3Train a cost-sensitive LightGBM classifier incorporating anomaly features
4. 4Apply Platt Scaling or Isotonic Regression for probability calibration

You'll learn

• Computational Graph — directed graph expressing mathematical operations and tracking gradients during backpropagation
• torch.nn.Module — base class for all neural network modules in PyTorch
• Stochastic Gradient Descent (SGD) & Adam — adaptive learning rate optimisation algorithms for neural networks

How to work through it

1. 1Understand forward passes, computational graphs, and backward passes (Autograd)
2. 2Build custom Dataset and DataLoader classes in PyTorch
3. 3Define neural network architectures subclassing torch.nn.Module
4. 4Write modular training and evaluation loops including optimizer steps and loss tracking

You'll learn

• Self-Attention Mechanism — architectural layer computing dynamic relationship weights between all tokens in a sequence
• Sentence Transformers — fine-tuned transformer networks mapping variable-length text into dense semantic vector spaces
• Vector Embeddings & FAISS — high-dimensional numerical representations and library for fast approximate nearest neighbour search

How to work through it

1. 1Study the self-attention mechanism and transformer encoder/decoder layouts
2. 2Tokenize text datasets using Hugging Face AutoTokenizer
3. 3Extract sentence embeddings using modern Pretrained Sentence Transformers
4. 4Build a similarity search index over embeddings using FAISS or cosine distance

You'll learn

• Retrieval-Augmented Generation (RAG) — framework combining external document retrieval with LLM generation to reduce hallucinations
• Chunking Strategies — methods for splitting text into semantically cohesive passages while preserving context
• Hallucination Mitigation — prompt engineering and grounding techniques ensuring outputs cite source documents accurately

How to work through it

1. 1Chunk and preprocess a domain-specific text dataset (e.g., financial filings)
2. 2Store and index embedded chunks in a local vector database
3. 3Implement similarity retrieval and re-ranking for user queries
4. 4Pass retrieved contexts into an LLM prompt and test retrieval accuracy

You'll learn

• FastAPI — modern, high-performance web framework for building APIs with Python
• Pydantic — data validation and settings management using Python type annotations
• Model Inference Latency — time taken to receive data, preprocess, score, and return a prediction

How to work through it

1. 1Define input/output data schemas using Pydantic models
2. 2Write asynchronous endpoints handling GET and POST requests in FastAPI
3. 3Load ML model pipelines into memory during application startup
4. 4Test endpoints locally using automated Swagger documentation and pytest

You'll learn

• Docker & Containers — lightweight standalone executable packages containing everything needed to run an application
• GitHub Actions — automated continuous integration and continuous delivery (CI/CD) platform
• Image Layer Optimization — structuring Dockerfile commands to leverage build caching and minimise image size

How to work through it

1. 1Write optimized multi-stage Dockerfiles for Python applications
2. 2Build, tag, and run Docker containers locally with port binding
3. 3Configure GitHub Actions workflows (.github/workflows/ci.yml) to run tests automatically on push
4. 4Verify that the containerised inference API passes automated health checks

You'll learn

• Workflow Orchestrator — system managing complex computational graphs, task dependencies, schedules, and failure alerts
• Data Lineage — audit trail tracking how data flows and transforms from raw ingestion to reporting
• Idempotency — property of an operation whereby it produces the exact same result even if executed multiple times

How to work through it

1. 1Define tasks and workflows using Prefect decorators (@task, @flow)
2. 2Implement task retries, error notifications, and state change handlers
3. 3Run automated data quality checks during execution using Great Expectations or Pydantic
4. 4Schedule pipeline runs and inspect execution logs through the UI

You'll learn

• Minto Pyramid Principle — top-down communication structure stating conclusions first followed by supporting arguments
• Expected Monetary Value (EMV) — statistical calculation weighting financial outcomes by their probabilities
• Executive Memo — concise business document focused on actionable decisions and risk analysis

How to work through it

1. 1Map confusion matrix outcomes to concrete financial unit economics
2. 2Calculate expected monetary value across decision thresholds
3. 3Model uncertainty and sensitivity scenarios (Best, Base, Worst case)
4. 4Draft a structured executive decision memo using the Minto Pyramid Principle

You'll learn

• Slide Takeaway Titles — framing slide headings as complete takeaway assertions rather than passive categories
• Visual Hierarchy — arranging chart elements to guide the viewer's eye to the key insight immediately
• Audience Calibration — tailoring technical depth appropriately between C-suite and engineering stakeholders

How to work through it

1. 1Select an existing completed project from your portfolio
2. 2Structure slides using clear takeaway titles rather than topic headers
3. 3Design decluttered visual charts highlighting strategic conclusions
4. 4Record a 10-minute presentation simulating an executive briefing

You'll learn

• Cookiecutter Data Science — standard, structured directory layout for reproducible data science projects
• Technical Documentation — clear, professional exposition of project goals, constraints, and architecture in markdown
• Portfolio Curation — presenting a focused set of high-quality projects rather than dozens of unmaintained tutorials

How to work through it

1. 1Standardize project layouts using Cookiecutter Data Science standards
2. 2Write detailed READMEs with badges, architecture diagrams, and impact summaries
3. 3Ensure all virtual environments and Dockerfiles reproduce dependencies cleanly
4. 4Deploy at least one interactive demo to a free cloud tier (e.g., Streamlit Community Cloud or HuggingFace Spaces)

You'll learn

• Technical Coding Screening — timed live coding tests assessing fluency in SQL, Python, and data manipulation
• Machine Learning Concept Defense — articulating trade-offs and mathematical justifications clearly under pressure
• Case Study Frameworks — structured methods for breaking down ambiguous business problems into analytical steps

How to work through it

1. 1Practice SQL aggregation and window functions on LeetCode / StrataScratch
2. 2Practice array/string data manipulation problems in Python
3. 3Conduct mock reviews explaining bias-variance, regularisation, and hypothesis testing out loud
4. 4Review common experimental design and metric definition case questions

You'll learn

• Take-Home Assessment Strategy — allocating time effectively across data cleaning, modeling, and communication
• Code Review Standards — formatting code according to PEP8, modular structure, and automated testing
• Trade-off Documentation — clearly explaining what was prioritised and what would be done with more time

How to work through it

1. 1Download an unfamiliar complex open dataset and problem prompt
2. 2Time-box execution to replicate realistic take-home deadlines
3. 3Perform cleaning, modeling, and validation following strict modular standards
4. 4Assemble the final deliverables: code repository, executive PDF memo, and brief slide deck