TUTOR

M09 · Level II — Machine Intelligence

Neural Networks from First Principles

Derive neural networks as compositional mathematical systems, then engineer reliable training, diagnostics and deployment.

Topics

  • Computation
  • Representation
  • Differentiation
  • Learning
  • Training Dynamics
  • Diagnostics
  • Systems

Technical concepts

  • 01First Principle: What a Neural Network Computes
  • 02The Affine Map: The Core Computational Primitive
  • 03Scalar Neuron, Perceptron and Decision Geometry
  • 04Activation Functions as Controlled Nonlinearity
  • 05Multilayer Composition and Representation
  • 06Expressivity and the Universal Approximation Idea
  • 07Computational Graphs: The Language of Backpropagation
  • 08The Chain Rule: Local Sensitivity Becomes Global Sensitivity
  • 09Reverse-Mode Automatic Differentiation and Backpropagation
  • 10Loss Functions Turn Behaviour into a Learning Signal
  • 11Gradient Descent as Parameter Dynamics
  • 12Initialisation and Symmetry Breaking
  • 13Normalisation and Internal Scale Control
  • 14Regularisation: Constraining Learned Solutions
  • 15Training Dynamics: What Changes During Learning?
  • 16Numerical Stability and Finite Precision
  • 17Minibatches, Stochasticity and Throughput
  • 18Capacity, Underfitting and Overfitting
  • 19Learned Representations
  • 20Vision Bridge: From Pixels to Feature Maps
  • 21Language and Sequence Bridge
  • 22Transformer Bridge: The Neural Core Before Attention
  • 23Implementation I: Neural Network in NumPy
  • 24Implementation II: Framework Mapping with PyTorch
  • 25Gradient Checking and Verification
  • 26The Reproducible Training Harness
  • 27Failure Lab I: Dead or Saturated Activations
  • 28Failure Lab II: Vanishing and Exploding Gradients
  • 29Failure Lab III: When the Network Is Not the Problem
  • 30Controlled Neural-Network Experiments
  • 31Interpretability and Diagnostic Probes
  • 32Inference: The Trained Network as a Runtime System
  • 33Quantum Technology Bridge: Where It May Intersect with Neural Systems

Practical implementation

  • 01Laboratory Matrix

Tools and technologies

  • VM/container
  • reproducible experiments
  • NumPy first
  • framework second
  • technical viva

Project

Laboratory Matrix

Expected learning outcomes

  • 01By completion, the learner should be capable of writing a neural network from first principles, proving the gradient for a small graph, detecting numerical pathologies, comparing training interventions under controlled experiments, and translating the same principles into vision and language settings.
  • 02Every implementation must be traceable to a mathematical statement, measurable objective, gradient pathway and failure mode.

Assessment

  • 01Daily practice
  • 02Laboratories
  • 03Research reproduction
  • 04Integrated project
  • 05Technical viva
  • 06Engineering record

Source document: TUTOR Frontier Module 09 Neural Networks from First Principles