M11 · Level II — Machine Intelligence
Transformers & Representation Learning
Derive transformers from first principles and understand how contextual representations are produced, trained, evaluated and scaled.
Topics
Technical concepts
- 01From Symbols to Computable Sequences
- 02Embeddings: Turning Discrete IDs into Continuous State
- 03Positional Information
- 04Why Attention?
- 05Query, Key and Value: The Core Attention Construction
- 06Softmax Attention and Probability Geometry
- 07Masking: Constraining the Information Path
- 08Multi-Head Attention
- 09The Position-Wise Feed-Forward Network
- 10Full Transformer Block as a Computational Graph
- 11Attention Complexity and the Scaling Problem
- 12Efficient Attention as a Systems Question
- 13Sequence Prediction Objectives
- 14Masked and Denoising Objectives
- 15Encoder-Only, Decoder-Only and Encoder-Decoder Systems
- 16Cross-Attention and Conditional Information Transfer
- 17Context Length and Positional Generalisation
- 18Static vs Contextual Representation
- 19Representation Geometry and Similarity
- 20Attention Diagnostics
- 21Training a Transformer: Systems View
- 22Parameterisation and Scaling as Experimental Questions
- 23In-Context Learning as a Representation and Inference Problem
- 24Prompt Structure as Input Programming
- 25Decoding: Turning Logits into Generated Sequences
- 26Pretraining as Representation Formation
- 27Transfer and Fine-Tuning Bridge
- 28Quantum Technology Bridge: Attention, Kernels and Hybrid Subroutines
Practical implementation
- 01Laboratory I: Attention from First Principles
- 02Laboratory II: Transformer Block from Scratch
- 03Laboratory III: Positional Encoding Ablation
- 04Laboratory IV: Attention Scaling Benchmark
- 05Laboratory V: Character-to-Context Transformer
- 06Laboratory VI: Contextual Representation Probe
- 07Laboratory VII: Masking and Information Leakage
- 08Laboratory VIII: Representation Robustness
Tools and technologies
Project
Small end-to-end transformer system + evidence packet
Expected learning outcomes
- 01Can derive scaled dot-product attention with correct dimensions.
- 02Can implement attention and a transformer block from primitive tensor operations.
- 03Can prove and test masking behaviour.
- 04Can explain positional information and run a controlled ablation.
- 05Can measure attention scaling in time and memory.
- 06Can analyse contextual representation with a defined probe or perturbation.
- 07Can reproduce a small research claim with documented deviations.
- 08Can defend the system and its limitations in a technical viva.
Assessment
- 01Daily practice
- 02Laboratories
- 03Research reproduction
- 04Integrated project
- 05Technical viva
- 06Engineering record
Source document: TUTOR Frontier Module 11 Transformers Representation Learning