M21 · Level IV — Autonomous & Embodied Intelligence
Reinforcement Learning & Sequential Decision Systems
Understand learning through interaction as a formal decision problem, then engineer value functions, policies, temporal-difference learning, policy gradients, actor-critic methods, offline learning and model-based decision systems.
Topics
Technical concepts
- 01Sequential Decision Problem
- 02Markov Decision Processes
- 03Reward, Return and Long-Horizon Credit Assignment
- 04Bellman Expectation and Optimality Equations
- 05Dynamic Programming
- 06Fixed Points, Contraction and Convergence
- 07Exploration and Exploitation
- 08Monte Carlo Learning
- 09Temporal-Difference Learning
- 10Eligibility Traces and TD(lambda)
- 11SARSA: On-Policy Temporal-Difference Control
- 12Q-Learning: Off-Policy Control
- 13From Tables to Function Approximation
- 14Deep Q Learning as a Systems Pattern
- 15Experience Replay and Dataset Dynamics
- 16Policy Gradient Principle
- 17Log-Derivative Trick and REINFORCE
- 18Baselines, Advantages and Variance Reduction
- 19Actor-Critic Architecture
- 20Entropy Regularisation and Exploration
- 21Online vs Offline Reinforcement Learning
- 22Distribution Shift in Offline RL
- 23Conservative Value Estimation
- 24Model-Based Reinforcement Learning
- 25Compounding Model Error
- 26Model Predictive Control as a Bridge
- 27Constrained and Safe Reinforcement Learning
- 28Reward Design and Specification
- 29Partial Observability and Belief States
- 30POMDP Formulation
- 31Multi-Agent Reinforcement Learning
- 32Hierarchical and Temporally Extended Actions
- 33Generalisation Across Environments
- 34Model-Free vs Model-Based Reasoning
- 35Systems Bridge: RL at Runtime and Scale
- 36Evaluation Bridge: Measuring RL Systems
- 37Quantum Technology Bridge: RL and Hybrid Decision Loops
Practical implementation
- 01Laboratory I: MDP and Bellman Simulator
- 02Laboratory II: Exploration and Bandits
- 03Laboratory III: Monte Carlo vs Temporal-Difference
- 04Laboratory IV: SARSA vs Q-Learning
- 05Laboratory V: Neural Value Function
- 06Laboratory VI: Policy Gradient Estimator
- 07Laboratory VII: Actor-Critic
- 08Laboratory VIII: Offline RL Support Analysis
- 09Laboratory IX: Learned Dynamics and Model Predictive Control
- 10Laboratory X: Constrained Decision System
Tools and technologies
Project
End-to-end RL system with evaluation, uncertainty and constraints
Expected learning outcomes
- 01Can formalise an MDP and explain its assumptions.
- 02Can derive and implement Bellman updates.
- 03Can use dynamic programming when a model is available.
- 04Can implement and analyse Monte Carlo and TD learning.
- 05Can distinguish SARSA and Q-learning correctly.
- 06Can explain function approximation and its instability risks.
- 07Can implement and diagnose policy-gradient/actor-critic learning.
- 08Can analyse exploration and regret-style behaviour.
- 09Can detect offline data-support limitations.
- 10Can use model-based planning while accounting for model error.
- 11Can specify and evaluate decision constraints.
- 12Can defend an RL system with reproducible evidence.
Assessment
- 01Daily practice
- 02Laboratories
- 03Research reproduction
- 04Integrated project
- 05Technical viva
- 06Engineering record
Source document: TUTOR Frontier Module 21 Reinforcement Learning Sequential Decision