TUTOR

M21 · Level IV — Autonomous & Embodied Intelligence

Reinforcement Learning & Sequential Decision Systems

Understand learning through interaction as a formal decision problem, then engineer value functions, policies, temporal-difference learning, policy gradients, actor-critic methods, offline learning and model-based decision systems.

Topics

  • State
  • Action
  • Transition
  • Reward
  • Return
  • Value
  • Policy
  • Planning
  • Learning
  • Safe Decision

Technical concepts

  • 01Sequential Decision Problem
  • 02Markov Decision Processes
  • 03Reward, Return and Long-Horizon Credit Assignment
  • 04Bellman Expectation and Optimality Equations
  • 05Dynamic Programming
  • 06Fixed Points, Contraction and Convergence
  • 07Exploration and Exploitation
  • 08Monte Carlo Learning
  • 09Temporal-Difference Learning
  • 10Eligibility Traces and TD(lambda)
  • 11SARSA: On-Policy Temporal-Difference Control
  • 12Q-Learning: Off-Policy Control
  • 13From Tables to Function Approximation
  • 14Deep Q Learning as a Systems Pattern
  • 15Experience Replay and Dataset Dynamics
  • 16Policy Gradient Principle
  • 17Log-Derivative Trick and REINFORCE
  • 18Baselines, Advantages and Variance Reduction
  • 19Actor-Critic Architecture
  • 20Entropy Regularisation and Exploration
  • 21Online vs Offline Reinforcement Learning
  • 22Distribution Shift in Offline RL
  • 23Conservative Value Estimation
  • 24Model-Based Reinforcement Learning
  • 25Compounding Model Error
  • 26Model Predictive Control as a Bridge
  • 27Constrained and Safe Reinforcement Learning
  • 28Reward Design and Specification
  • 29Partial Observability and Belief States
  • 30POMDP Formulation
  • 31Multi-Agent Reinforcement Learning
  • 32Hierarchical and Temporally Extended Actions
  • 33Generalisation Across Environments
  • 34Model-Free vs Model-Based Reasoning
  • 35Systems Bridge: RL at Runtime and Scale
  • 36Evaluation Bridge: Measuring RL Systems
  • 37Quantum Technology Bridge: RL and Hybrid Decision Loops

Practical implementation

  • 01Laboratory I: MDP and Bellman Simulator
  • 02Laboratory II: Exploration and Bandits
  • 03Laboratory III: Monte Carlo vs Temporal-Difference
  • 04Laboratory IV: SARSA vs Q-Learning
  • 05Laboratory V: Neural Value Function
  • 06Laboratory VI: Policy Gradient Estimator
  • 07Laboratory VII: Actor-Critic
  • 08Laboratory VIII: Offline RL Support Analysis
  • 09Laboratory IX: Learned Dynamics and Model Predictive Control
  • 10Laboratory X: Constrained Decision System

Tools and technologies

  • simulators/environments
  • reproducible trajectories
  • dynamic programming
  • Monte Carlo temporal-difference methods
  • policy optimisation
  • offline RL
  • model-based experiments

Project

End-to-end RL system with evaluation, uncertainty and constraints

Expected learning outcomes

  • 01Can formalise an MDP and explain its assumptions.
  • 02Can derive and implement Bellman updates.
  • 03Can use dynamic programming when a model is available.
  • 04Can implement and analyse Monte Carlo and TD learning.
  • 05Can distinguish SARSA and Q-learning correctly.
  • 06Can explain function approximation and its instability risks.
  • 07Can implement and diagnose policy-gradient/actor-critic learning.
  • 08Can analyse exploration and regret-style behaviour.
  • 09Can detect offline data-support limitations.
  • 10Can use model-based planning while accounting for model error.
  • 11Can specify and evaluate decision constraints.
  • 12Can defend an RL system with reproducible evidence.

Assessment

  • 01Daily practice
  • 02Laboratories
  • 03Research reproduction
  • 04Integrated project
  • 05Technical viva
  • 06Engineering record

Source document: TUTOR Frontier Module 21 Reinforcement Learning Sequential Decision