TUTOR

M11 · Level II — Machine Intelligence

Transformers & Representation Learning

Derive transformers from first principles and understand how contextual representations are produced, trained, evaluated and scaled.

Topics

  • Symbols
  • Tokens
  • Vectors
  • Context
  • Attention
  • Transformer Blocks
  • Objectives
  • Representation
  • Scaling

Technical concepts

  • 01From Symbols to Computable Sequences
  • 02Embeddings: Turning Discrete IDs into Continuous State
  • 03Positional Information
  • 04Why Attention?
  • 05Query, Key and Value: The Core Attention Construction
  • 06Softmax Attention and Probability Geometry
  • 07Masking: Constraining the Information Path
  • 08Multi-Head Attention
  • 09The Position-Wise Feed-Forward Network
  • 10Full Transformer Block as a Computational Graph
  • 11Attention Complexity and the Scaling Problem
  • 12Efficient Attention as a Systems Question
  • 13Sequence Prediction Objectives
  • 14Masked and Denoising Objectives
  • 15Encoder-Only, Decoder-Only and Encoder-Decoder Systems
  • 16Cross-Attention and Conditional Information Transfer
  • 17Context Length and Positional Generalisation
  • 18Static vs Contextual Representation
  • 19Representation Geometry and Similarity
  • 20Attention Diagnostics
  • 21Training a Transformer: Systems View
  • 22Parameterisation and Scaling as Experimental Questions
  • 23In-Context Learning as a Representation and Inference Problem
  • 24Prompt Structure as Input Programming
  • 25Decoding: Turning Logits into Generated Sequences
  • 26Pretraining as Representation Formation
  • 27Transfer and Fine-Tuning Bridge
  • 28Quantum Technology Bridge: Attention, Kernels and Hybrid Subroutines

Practical implementation

  • 01Laboratory I: Attention from First Principles
  • 02Laboratory II: Transformer Block from Scratch
  • 03Laboratory III: Positional Encoding Ablation
  • 04Laboratory IV: Attention Scaling Benchmark
  • 05Laboratory V: Character-to-Context Transformer
  • 06Laboratory VI: Contextual Representation Probe
  • 07Laboratory VII: Masking and Information Leakage
  • 08Laboratory VIII: Representation Robustness

Tools and technologies

  • Linux VM/container
  • scratch attention implementation
  • framework implementation reproducible experiments
  • technical viva

Project

Small end-to-end transformer system + evidence packet

Expected learning outcomes

  • 01Can derive scaled dot-product attention with correct dimensions.
  • 02Can implement attention and a transformer block from primitive tensor operations.
  • 03Can prove and test masking behaviour.
  • 04Can explain positional information and run a controlled ablation.
  • 05Can measure attention scaling in time and memory.
  • 06Can analyse contextual representation with a defined probe or perturbation.
  • 07Can reproduce a small research claim with documented deviations.
  • 08Can defend the system and its limitations in a technical viva.

Assessment

  • 01Daily practice
  • 02Laboratories
  • 03Research reproduction
  • 04Integrated project
  • 05Technical viva
  • 06Engineering record

Source document: TUTOR Frontier Module 11 Transformers Representation Learning