M08 · Level II — Machine Intelligence
Generalisation & Experiment Design
Determine whether an intelligent system has learned transferable structure rather than accidental properties of finite data.
Topics
Technical concepts
- 01Generalisation: What Does “Works” Mean?
- 02Hypotheses, Controls & Experimental Questions
- 03Experimental Units & Independence
- 04Train, Validation & Test as Information Boundaries
- 05Cross-Validation from First Principles
- 06Stratified, Grouped, Time-Aware & Nested Evaluation
- 07Learning Curves
- 08Performance Distributions, Not Single Scores
- 09Bootstrap for Performance Uncertainty
- 10Confidence Intervals & Interval Reasoning
- 11Paired Model Comparisons
- 12Repeated Evaluation & Randomness
- 13Hyperparameter Search as Experimental Design
- 14Multiple Comparisons & Researcher Degrees of Freedom
- 15Ablation Studies
- 16Calibration & Reliability
- 17Thresholds, Utility & Abstention
- 18Metric Selection Under Imbalance
- 19Robustness: Perturb the Input
- 20Distribution Shift
- 21Error Slicing & Subpopulation Analysis
- 22Statistical Power & Sample Size
- 23Reproducible Experiment Specification
- 24Quantum Computing Efficiency Bridge
Practical implementation
- 01Laboratory 01 — Generalisation Benchmark Harness
- 02Laboratory 02 — Cross-Validation & Bootstrap Study
- 03Laboratory 03 — Ablation & Hyperparameter Study
- 04Laboratory 04 — Robustness & Shift Matrix
- 05Laboratory 05 — Calibration & Abstention
- 06Laboratory 06 — Reproducibility Audit
Tools and technologies
Project
Laboratory 06 — Reproducibility Audit
Expected learning outcomes
- 01Formulate testable hypotheses.
- 02Design leakage-safe evaluation.
- 03Use cross-validation appropriately.
- 04Quantify uncertainty.
- 05Compare systems fairly.
- 06Analyse calibration.
- 07Stress robustness.
- 08Reason about shift.
- 09Benchmark emerging compute methods rigorously.
Assessment
- 01Daily practice — 20%
- 02Laboratories — 35%
- 03Research reproduction — 10%
- 04Integrated experiment — 20%
- 05Technical viva — 10%
- 06Engineering record — 5%
Source document: TUTOR Frontier Module 08 Generalisation Experiment Design