AI Engineer — Competency Roadmap

You'll learn

• uv — extremely fast Python package and virtual environment manager
• ruff — high-performance Python linter and formatter
• mypy — static type checker for type-annotated Python

How to work through it

1. 1Install uv package manager
2. 2Initialize a new Python project with pyproject.toml
3. 3Configure ruff and mypy in pyproject.toml
4. 4Write and run a typed Python script

You'll learn

• asyncio — Python built-in library for writing concurrent code using async/await
• httpx — asynchronous HTTP client for Python
• Pydantic — data validation and settings management using Python type hints

How to work through it

1. 1Define Pydantic models for incoming weather JSON schemas
2. 2Implement async HTTP calls using httpx.AsyncClient
3. 3Gather concurrent requests using asyncio.gather
4. 4Add error handling for schema mismatches and network timeouts

You'll learn

• pytest — testing framework for Python
• pytest-asyncio — plugin for testing asyncio coroutines
• pytest-mock — thin wrapper around standard mock library

How to work through it

1. 1Install and configure pytest and pytest-asyncio
2. 2Write fixtures to mock external HTTP responses
3. 3Test positive and edge-case schema validations
4. 4Measure test coverage via pytest-cov

You'll learn

• Typer — library for building CLI applications based on Python type hints
• GitHub Actions — automated continuous integration and continuous delivery platform

How to work through it

1. 1Structure codebase into a modular src package layout
2. 2Add CLI argument parsing using Typer
3. 3Write GitHub Actions workflow for linting and test suite
4. 4Push working repository with README documentation

You'll learn

• tiktoken — fast BPE tokeniser for use with OpenAI models
• Byte-Pair Encoding (BPE) — subword tokenization algorithm used in modern LLMs
• Context Window — maximum sequence length of tokens a model can process at once

How to work through it

1. 1Install tiktoken and open-source tokenizer libraries
2. 2Encode strings to raw token IDs and decode them back
3. 3Calculate pricing based on current OpenAI/Anthropic model tier rates
4. 4Analyze edge cases with special characters and multi-lingual text

You'll learn

• Few-Shot Prompting — providing targeted input-output examples inside the prompt
• Chain-of-Thought (CoT) — prompting technique instructing models to produce intermediate reasoning steps
• Temperature / Top_p — sampling parameters controlling output randomness and diversity

How to work through it

1. 1Configure OpenAI and Anthropic SDK client instances
2. 2Construct structured system prompts with few-shot demonstration examples
3. 3Implement Chain-of-Thought reasoning steps in prompt templates
4. 4Compare response determinism by varying temperature and top_p

You'll learn

• Instructor — library for structured LLM outputs built on top of Pydantic
• JSON Schema Mode — LLM API parameter forcing deterministic JSON compliance
• Self-Correction Loop — pattern of passing schema validation error messages back to the LLM for correction

How to work through it

1. 1Define nested Pydantic schema with validators and field constraints
2. 2Use Instructor or native OpenAI Structured Outputs to enforce schema
3. 3Implement retry hooks for schema validation failures
4. 4Test validation boundary cases with malformed or incomplete inputs

You'll learn

• Embedding Vector — high-dimensional numerical representation of semantic meaning
• Cosine Similarity — metric measuring the cosine of the angle between two non-zero vectors
• sentence-transformers — Python framework for state-of-the-art text and image embeddings

How to work through it

1. 1Generate embeddings using sentence-transformers or local embedding models
2. 2Write a NumPy function calculating cosine similarity between query and corpus vectors
3. 3Implement argsort-based top-k retrieval
4. 4Benchmark retrieval speed vs linear scan as corpus size grows

You'll learn

• Chunking — process of splitting long texts into smaller segments for embedding models
• Sliding Window Overlap — retaining trailing tokens from previous chunk to preserve context continuity
• Recursive Character Splitting — dividing text hierarchically by paragraphs, newlines, and sentences

How to work through it

1. 1Extract raw text from PDF and Markdown files
2. 2Implement fixed character chunking with sliding window overlap
3. 3Implement recursive chunking respecting paragraph and header boundaries
4. 4Evaluate chunk preservation on complex table and list structures

You'll learn

• HNSW (Hierarchical Navigable Small World) — graph-based approximate nearest neighbor index algorithm
• Qdrant — open-source vector similarity search engine with extended filtering support
• Metadata Filtering — combining relational attributes with vector similarity in search

How to work through it

1. 1Run Qdrant or Chroma via Docker or embedded engine
2. 2Define collection schema with distance metric (Cosine/Dot)
3. 3Batch upsert embeddings with structured metadata payloads
4. 4Execute filtered semantic queries combining vector distance with metadata predicates

You'll learn

• BM25 (Best Matching 25) — probabilistic ranking function used in information retrieval
• Reciprocal Rank Fusion (RRF) — rank aggregation algorithm that merges multiple ranked result lists
• Hybrid Search — combining sparse lexical search and dense vector search

How to work through it

1. 1Index documents into a BM25 sparse keyword index
2. 2Index the same documents into a vector embedding store
3. 3Implement Reciprocal Rank Fusion (RRF) algorithm to merge ranked lists
4. 4Run comparative evaluation on keyword-heavy vs semantic queries

You'll learn

• Cross-Encoder — model that processes query and passage together with full cross-attention
• Bi-Encoder — model that embeds query and passage independently into vector space
• Two-stage Retrieval — pattern of fast retrieval followed by compute-intensive reranking

How to work through it

1. 1Retrieve top-k candidates (e.g. k=25) via hybrid search
2. 2Pass query-document pairs to cross-encoder reranker
3. 3Sort by reranker relevance score and slice top-n (e.g. n=5)
4. 4Measure latency and relevance delta before and after reranking

You'll learn

• Grounding — constraining LLM generation strictly to provided context chunks
• Citation Verification — algorithmic validation that claims link directly to retrieved sources
• Hallucination Mitigation — prompt and architectural techniques to suppress fabricated statements

How to work through it

1. 1Construct context prompt including chunk IDs and source titles
2. 2Implement prompt guardrails instructing model to decline out-of-context queries
3. 3Extract structured citation references in the model response
4. 4Add automated check verifying cited chunks actually exist in context

You'll learn

• Tool Calling / Function Calling — protocol allowing LLMs to request execution of specific external code functions
• Schema Generation — converting programming language type definitions into OpenAPI/JSON schemas
• Tool Message Role — dedicated message format for feeding tool execution output back into LLM context

How to work through it

1. 1Write typed Python helper functions with docstrings
2. 2Generate JSON schemas automatically using Pydantic or docstring parsers
3. 3Send user prompt with available tool definitions to model
4. 4Parse model tool_calls, execute corresponding local functions, and return outputs

You'll learn

• ReAct Pattern — prompting and execution architecture intertwining reasoning traces with task-specific actions
• Agent State Management — preserving historical context and intermediate tool observations across iterations
• Infinite Loop Prevention — setting hard iteration and timeout boundaries in autonomous control flows

How to work through it

1. 1Define agent state object (message history, scratchpad, loop count)
2. 2Implement Thought -> Action -> Action Input -> Observation loop cycle
3. 3Add safety limits (maximum iterations, token budget cap)
4. 4Implement graceful handling for malformed tool arguments

You'll learn

• Text-to-SQL — generating valid relational database queries from natural language
• Read-Only Guardrails — restricting agent capabilities to prevent accidental data modification or injection
• Schema Pruning — dynamically injecting only relevant table definitions into context

How to work through it

1. 1Provide database schema introspection tool to agent
2. 2Create read-only SQL execution tool with parameter limits
3. 3Implement error-catching feedback loop for SQL syntax errors
4. 4Format final result with summarized table and generated markdown charts

You'll learn

• Golden Dataset — curated benchmark set of inputs with verified reference outputs
• Synthetic Data Generation — using generative models to bootstrap test suites and training data
• Multi-Hop Reasoning — questions requiring information synthesis across multiple separate passages

How to work through it

1. 1Extract distinct chunks and document sections
2. 2Prompt a frontier model to generate questions and expected answers grounded in specific chunks
3. 3Filter out ambiguous or low-quality generated questions
4. 4Export evaluation dataset in standard format (e.g. JSONL)

You'll learn

• LLM-as-a-Judge — using a powerful model to score the quality and factual consistency of other model outputs
• Faithfulness Metric — fraction of claims in generated answer that can be inferred from context
• Context Recall — measure of whether all information needed to answer the question was retrieved

How to work through it

1. 1Install and configure Ragas evaluation framework or custom judge prompts
2. 2Execute RAG pipeline on golden dataset queries to collect retrieved context and answers
3. 3Compute Faithfulness (hallucination check) and Answer Relevance scores
4. 4Compute Context Precision and Recall against ground truth

You'll learn

• Prompt Injection — adversarial technique causing an LLM to ignore system instructions
• PII Masking — detecting and redacting sensitive personal data prior to model inference
• Input Guardrails — defensive validation layers filtering untrusted user text

How to work through it

1. 1Define input moderation rules and regex/entity PII masking
2. 2Implement lightweight prompt injection detector classifier
3. 3Add safety check layer before passing prompts to primary LLM
4. 4Test with known jailbreak datasets (e.g. DAN prompts)

You'll learn

• Server-Sent Events (SSE) — standard for streaming unidirectional text events over HTTP
• Time To First Token (TTFT) — latency metric measuring elapsed time before the first token arrives
• FastAPI — modern, fast web framework for building APIs with Python

How to work through it

1. 1Create FastAPI app with async route handlers
2. 2Implement async generator yielding SSE data chunks
3. 3Handle client disconnections and cancel upstream LLM requests
4. 4Add CORS and authentication middleware

You'll learn

• Semantic Caching — caching responses based on vector similarity rather than exact string equality
• Redis — in-memory data store frequently used for caching and vector operations
• Token Bucket Rate Limiting — algorithm enforcing request throughput constraints per user

How to work through it

1. 1Set up Redis instance with RedisVL or vector search index
2. 2Compute embedding for incoming user query
3. 3Perform vector similarity search against cached query keys with similarity threshold (e.g. >0.95)
4. 4Return cached output on cache hit; call LLM and store result on cache miss

You'll learn

• Langfuse / Phoenix — open-source observability and tracing platforms for LLM applications
• Trace Spans — individual timed segments in a distributed operation tree
• Prompt Versioning — tracking changes to prompt templates alongside production metrics

How to work through it

1. 1Run Langfuse or Arize Phoenix locally or via cloud tier
2. 2Instrument FastAPI application and LLM calls using tracing decorators
3. 3Log metadata including user IDs, session tags, and prompt version hashes
4. 4Track cost per session and token usage over time

You'll learn

• Docker Multi-Stage Build — creating minimal container images by separating build and runtime environments
• Secrets Management — secure injection of API tokens into container runtimes
• Health Check Endpoint — HTTP route reporting service readiness to load balancers

How to work through it

1. 1Write optimized multi-stage Dockerfile minimizing image size
2. 2Configure environment secrets for API keys and database credentials
3. 3Deploy container to cloud host with health check endpoints
4. 4Run load test using Locust or wrk to determine throughput limits

You'll learn

• GGUF — binary format for storing models optimized for fast CPU/GPU inference with llama.cpp
• Quantization (Q4/Q8) — reducing model weight precision from FP16 to 4/8-bit integers to save memory
• VRAM Offloading — splitting model layers between system RAM and GPU memory

How to work through it

1. 1Install Ollama and run a lightweight model (e.g. Llama-3.2-3B or Mistral-7B)
2. 2Download GGUF model files from Hugging Face Hub
3. 3Run inference using llama.cpp CLI with CPU/GPU offloading parameters
4. 4Measure tokens/second generation speed and VRAM allocation

You'll learn

• vLLM — high-throughput and memory-efficient LLM serving engine
• PagedAttention — memory management algorithm for KV cache inspired by virtual memory paging
• Continuous Batching — dynamic iteration-level batching of incoming LLM requests

How to work through it

1. 1Deploy a vLLM container serving an open model (e.g. Qwen2.5-7B)
2. 2Configure max model length and GPU memory utilization parameters
3. 3Benchmark concurrent request throughput using vLLM benchmarking scripts
4. 4Expose the OpenAI-compatible API to downstream applications

You'll learn

• Instruction Tuning — training models to follow natural language commands and task specifications
• Hugging Face Datasets — library for easily sharing and loading machine learning datasets
• Alpaca/ShareGPT Format — standardized JSON formats for conversational training datasets

How to work through it

1. 1Gather raw domain text or synthetic conversational pairs
2. 2Format data into Alpaca or ShareGPT JSON schemas with system, user, and assistant roles
3. 3Validate formatting and remove duplicates using pandas and Hugging Face Datasets
4. 4Plot sequence length histograms to set optimal max_seq_length

You'll learn

• LoRA (Low-Rank Adaptation) — technique freezing base model weights and training small low-rank adapter matrices
• QLoRA — quantized LoRA allowing fine-tuning of 8B+ models on consumer GPUs (e.g. 16GB VRAM)
• Unsloth — optimized framework for fast fine-tuning and export of LLMs

How to work through it

1. 1Set up Unsloth or Hugging Face TRL (SFTTrainer) environment
2. 2Load base model in 4-bit precision with BitsAndBytes
3. 3Attach LoRA target modules (q_proj, k_proj, v_proj, o_proj)
4. 4Run training loop with gradient accumulation and learning rate warmup
5. 5Save and test the resulting LoRA adapter weights

You'll learn

• Catastrophic Forgetting — tendency of fine-tuned models to lose general pre-trained knowledge
• Domain Adaptation — modifying base model behavior for specific industry vocabulary and constraints
• Holdout Test Set — unseen evaluation data preserved strictly for final validation

How to work through it

1. 1Run base model and fine-tuned model on identical holdout test set
2. 2Compute domain schema adherence rate (e.g. percentage of valid JSON produced)
3. 3Run general reasoning benchmark (e.g. MMLU subset) to check for catastrophic forgetting
4. 4Synthesize findings into an evaluation comparison report

You'll learn

• Vercel AI SDK — TypeScript toolkit for building interactive AI-powered web applications
• React Server Components — React paradigm for server-side data fetching and rendering
• Streaming UI — rendering incoming token buffers incrementally into the DOM

How to work through it

1. 1Initialize Next.js project with Tailwind CSS and TypeScript
2. 2Use `useChat` hook from Vercel AI SDK connected to Python backend or Next.js route handler
3. 3Implement syntax highlighting for code blocks using react-markdown
4. 4Add automatic scroll-to-bottom behavior with manual scroll lock

You'll learn

• Generative UI — dynamically generating interactive user interface components from model outputs
• Component Streaming — streaming structured UI component properties over the wire

How to work through it

1. 1Define tool calling schemas for specific UI widget components
2. 2Parse tool call payloads in Next.js frontend
3. 3Dynamically mount interactive React components (e.g. Recharts) based on tool arguments
4. 4Handle user interactions on rendered widgets that send state updates back to the LLM

You'll learn

• System Architecture Design — end-to-end topology planning for multi-component AI systems
• Technical Documentation — crafting clear architecture diagrams and evaluation methodology reports

How to work through it

1. 1Draft system architecture specification and data flow diagrams
2. 2Implement Python backend with hybrid RAG, agent tools, and observability
3. 3Build TypeScript / Next.js frontend with generative UI
4. 4Run automated evaluation benchmark and document performance in repository README
5. 5Deploy to production cloud infrastructure and record demo walkthrough

You'll learn

• AI System Design — structured architectural methodology for complex AI platforms
• Back-of-the-Envelope Estimation — calculating token throughput, latency budgets, and operational costs
• Graceful Degradation — fallback patterns when frontier model APIs experience outages or rate limits

How to work through it

1. 1Study standard AI system design blueprints and scaling patterns
2. 2Break down requirements into functional and non-functional goals
3. 3Calculate back-of-the-envelope token costs, storage requirements, and QPS limits
4. 4Map out failure modes, circuit breakers, and evaluation loops