TUTOR

M08 · Level II — Machine Intelligence

Generalisation & Experiment Design

Determine whether an intelligent system has learned transferable structure rather than accidental properties of finite data.

Topics

  • Learning Beyond the Sample
  • Cross-Validation
  • Statistical Uncertainty
  • Robustness
  • Reproducibility
  • Quantum-Efficiency Benchmarking

Technical concepts

  • 01Generalisation: What Does “Works” Mean?
  • 02Hypotheses, Controls & Experimental Questions
  • 03Experimental Units & Independence
  • 04Train, Validation & Test as Information Boundaries
  • 05Cross-Validation from First Principles
  • 06Stratified, Grouped, Time-Aware & Nested Evaluation
  • 07Learning Curves
  • 08Performance Distributions, Not Single Scores
  • 09Bootstrap for Performance Uncertainty
  • 10Confidence Intervals & Interval Reasoning
  • 11Paired Model Comparisons
  • 12Repeated Evaluation & Randomness
  • 13Hyperparameter Search as Experimental Design
  • 14Multiple Comparisons & Researcher Degrees of Freedom
  • 15Ablation Studies
  • 16Calibration & Reliability
  • 17Thresholds, Utility & Abstention
  • 18Metric Selection Under Imbalance
  • 19Robustness: Perturb the Input
  • 20Distribution Shift
  • 21Error Slicing & Subpopulation Analysis
  • 22Statistical Power & Sample Size
  • 23Reproducible Experiment Specification
  • 24Quantum Computing Efficiency Bridge

Practical implementation

  • 01Laboratory 01 — Generalisation Benchmark Harness
  • 02Laboratory 02 — Cross-Validation & Bootstrap Study
  • 03Laboratory 03 — Ablation & Hyperparameter Study
  • 04Laboratory 04 — Robustness & Shift Matrix
  • 05Laboratory 05 — Calibration & Abstention
  • 06Laboratory 06 — Reproducibility Audit

Tools and technologies

  • VM/container
  • controlled experiments
  • statistical analysis
  • stress testing

Project

Laboratory 06 — Reproducibility Audit

Expected learning outcomes

  • 01Formulate testable hypotheses.
  • 02Design leakage-safe evaluation.
  • 03Use cross-validation appropriately.
  • 04Quantify uncertainty.
  • 05Compare systems fairly.
  • 06Analyse calibration.
  • 07Stress robustness.
  • 08Reason about shift.
  • 09Benchmark emerging compute methods rigorously.

Assessment

  • 01Daily practice — 20%
  • 02Laboratories — 35%
  • 03Research reproduction — 10%
  • 04Integrated experiment — 20%
  • 05Technical viva — 10%
  • 06Engineering record — 5%

Source document: TUTOR Frontier Module 08 Generalisation Experiment Design