Lecture 13 — Neural networks: from motivation to practice

Lecture 13 — Neural networks: from motivation to practice#

⚠️ AI-synthesized; not fully reviewed by course staff. Treat as a study aid; released slides, notebooks, and lecture recordings are authoritative.

Tue Mar 3, 2026 · Part 2 — Modern AI & lab work

What this lecture is about#

A neural network is, at its core, the same recipe from L7: pick a function class, define a loss, optimize. What’s new is that the function class is now massively expressive (a deep stack of linear maps interleaved with nonlinearities), the optimizer needs care (initialization, normalization, learning rate, gradient handling), and the inductive bias is encoded in the architecture rather than in a hand-crafted feature set.

The lecture is built in two halves.

The first half is conceptual: what is a neural network, why does it work, and how does it relate to the linear / kernel / tree models from labs 0-5?

  • Motivation. Linear models can’t approximate arbitrary functions. Trees and forests can but generalize poorly with high-dimensional structured inputs (images, sequences). Kernels can but scale badly. Neural networks combine expressiveness (universal approximation), parametric deployment (fixed-size model regardless of training-set size), and automatic differentiation (any differentiable loss + architecture is trainable).

  • Anatomy. Layers, activations, depth, parameters. A layer is “apply a linear map, then a non-linearity.” Depth lets you compose features into higher-level features.

  • Architecture as inductive bias. This is the central conceptual move. CNNs encode “translation-equivariance and locality matter” — the right bias for natural images (pooling layers add translation invariance at the readout). RNNs encode “sequential structure matters.” Transformers use self-attention, which is permutation equivariant on its own — and then add positional encoding precisely to break that symmetry so token order matters. The architecture is not incidental; it is the prior.

  • Loss matching task type. Squared error → Gaussian (regression). Cross-entropy → categorical (classification). Hinge → margin-based. NLL → arbitrary parametric. The choice is informed by L7’s framework.

The second half is practical: how do you actually train one of these without it failing silently?

  • Initialization. Bad init → vanishing or exploding gradients → silent failure. Kaiming / Xavier / orthogonal are not interchangeable.

  • Normalization. BatchNorm, LayerNorm, GroupNorm. They stabilize the optimization landscape.

  • Residual connections. They let gradients flow through deep stacks; they also change what gets learned.

  • Optimizers. SGD with momentum, Adam, AdamW. Each makes different assumptions about the loss surface.

  • Learning rate schedules. Constant, step, cosine, warmup. Tied tightly to batch size.

  • Gradient clipping and accumulation. When gradients spike (RNNs, large LRs); when memory limits batch size.

  • Monitoring. Loss curves, gradient norms, weight statistics, activation distributions. A healthy training run looks a particular way.

  • Regularization. Weight decay, dropout, early stopping, augmentation. Each implies a different prior.

The companion notebook nn_fundamentals_lab.ipynb runs MNIST experiments where you can see each of these knobs change something visible.

Why it matters#

Three reasons this lecture is the hinge of the course:

Almost every model in Parts 3-5 is a neural network. Clinical NLP (L20), medical imaging (L22), survival models (L23), protein language models (L27) — they are all variations on this template. If you internalize the architecture-as-inductive-bias idea here, the rest of the course becomes “what’s the right inductive bias for this modality.”

Training mechanics are the model. Two NNs with the same architecture trained with different learning-rate schedules will end up at different solutions. This is not a footnote; it is one of the central facts of modern ML and a reason “reproducibility crisis” exists. You need to know the optimizer is part of the model so that when you read a paper, you know which knobs to ask about.

Monitoring is non-optional. Expressive function classes can fit anything including noise — they fail silently in a way that simpler models don’t. The L10 inductive-bias / generalization story has its practical face here: you have to watch the training run, you can’t just check the final loss.

Things you should walk away believing#

  • Universal approximation makes the function class powerful; the optimizer and architecture together decide what you actually learn.

  • Architecture choice is a prior. Pick it for the data modality; don’t default to “what worked on MNIST.”

  • The training pipeline (init, norm, optimizer, LR, schedule, batch, clipping, monitoring, regularization) is the model. Don’t blame “training instability” — diagnose which piece broke.

  • Smaller models are easier to train but encode weaker priors. Bigger models with careful inductive bias usually win once you can afford them.

  • Always have a sanity baseline (linear or shallow NN) and a non-trivial loss curve in your peripheral vision while training.

How this connects to the rest of the course#

  • L14 (LLMs) is the largest-scale instance of this lecture: a transformer is a stack of attention + MLP blocks, optimized with the recipe above.

  • L15 (foundation models) is L14 abstracted: pre-train once, adapt to many tasks.

  • L20 (clinical NLP), L22 (imaging), L25-L27 (biological AI) are domain-specific instantiations of the same recipe with different inductive biases.

  • L24 (fairness) returns to “the optimizer’s bias decides which fair-or-unfair solution it converges to.”

Study guide#

Key terms, self-check questions, and additional resources for active recall.

Key Terms#

Term

Definition

Universal Approximation

A sufficiently wide (or deep) neural network can approximate any continuous function to arbitrary precision. Justifies using NNs as a general-purpose function class.

Layer

The fundamental building block: h = σ(Wx + b). W ∈ ℝ^{d_out × d_in}, b ∈ ℝ^{d_out}, σ is a nonlinear activation. Parameter count: d_out·d_in + d_out.

Activation Function

The nonlinearity σ applied element-wise. Without it, composing layers collapses to a single linear map. ReLU, GELU, sigmoid, tanh are common.

ReLU

σ(z) = max(0, z). Default activation; simple, non-saturating, but can produce “dead neurons” (stuck at 0 for all inputs).

GELU

Gaussian Error Linear Unit. Smooth approximation of ReLU; preferred in transformers (BERT, GPT).

Depth vs. Width

Depth (more layers) is far more parameter-efficient than width (more neurons per layer) for high expressiveness in practice.

Inductive Bias via Architecture

FC/MLP: no structural prior. CNN: translation equivariance (locality) at conv layers; pooling adds invariance. RNN: sequential ordering. Self-attention: permutation equivariant; full transformers add positional encoding to break this symmetry, so the deployed transformer is not permutation equivariant — order matters.

Xavier / Glorot Init

Two related conventions: Xavier “fan-in” sets weight variance = 1/d_in; the original Glorot recommendation is 2/(d_in+d_out). The two give different values; the “fan-in” form is more common in modern code. Designed for tanh/sigmoid.

Kaiming / He Init

Weight variance = 2/d_in. Designed for ReLU; accounts for ReLU zeroing half the neurons.

Batch Normalization

Normalizes layer inputs to zero mean and unit variance per mini-batch. Reduces internal covariate shift.

Layer Normalization

Normalizes across the feature dimension per sample. Preferred in transformers; works with variable sequence lengths.

Residual Connection

h_{ℓ+1} = h_ℓ + F(h_ℓ). “Skip” connections solve the degradation problem; create gradient highways; enable very deep networks (ResNet).

Adam / AdamW

Adaptive gradient optimizer with per-parameter first and second moment estimates. AdamW adds decoupled weight decay. Default for most DL; start at η = 10⁻³.

Learning Rate Schedule

Varies η during training. Warmup + cosine decay is the modern default. A constant LR violates Robbins–Monro convergence conditions.

Gradient Clipping

Caps gradient norms at a threshold (e.g., 1.0) to prevent destructive updates from gradient spikes.

Gradient Accumulation

Accumulate gradients over k micro-batches before updating, simulating a larger effective batch size when GPU memory is limited.

Linear Scaling Rule

When multiplying batch size by k, multiply LR by k (Goyal et al., 2017). Derived for SGD; breaks down for Adam at large batch sizes.

Weight Decay / L2 Regularization

Adds λ‖θ‖² to loss. Built into AdamW. Primary regularizer for most modern architectures.

Dropout

Randomly zeros neurons with probability p during training. Forces redundancy; reduces co-adaptation. p ∈ [0.1, 0.5].

Why Neural Networks? Three Requirements#

Requirement

Why It Matters for Health ML

Arbitrarily expressive (universal)

Health data is high-D and nonlinear; no simple model class suffices.

Parametric (fixed-size θ)

Deployable in clinical systems; model size doesn’t grow with the dataset.

Efficiently auto-differentiable

Scalable optimization via backprop; O(forward-pass) cost for gradients.

Architecture Encodes Assumptions (Inductive Bias)#

Layer Type

Inductive Bias

Health Modality

Rule of Thumb

FC / MLP

No structural prior

Tabular EHR, lab values

Baseline for structured data

CNN

Translation equivariance + locality (pooling adds invariance)

Medical images, ECG signals

First choice for images

RNN / LSTM

Sequential ordering

Clinical notes, time series

Use if order is crucial

Transformer

Self-attention is permutation equivariant; positional encoding restores order sensitivity, so the full model is not permutation equivariant

Any; esp. notes, sequences

Fewest structural assumptions; default when unsure

Practical Optimization: The Full Toolkit#

Training recipe

  1. Choose architecture matching your data modality.

  2. Initialize weights with Kaiming (ReLU) or Xavier (tanh/sigmoid).

  3. Add BatchNorm (CNNs) or LayerNorm (Transformers).

  4. Add residual connections if depth > 5-6.

  5. Use AdamW optimizer with η = 10⁻³ as default.

  6. Use cosine decay or linear-warmup + cosine-decay LR schedule.

  7. Set gradient clipping to 1.0.

  8. Monitor: loss, gradient norm, activation statistics, train/val gap.

  9. Regularize with weight decay first; add dropout if still overfitting.

Loss Functions and Their Probabilistic Interpretations#

Loss

Task

Probabilistic Interpretation

Binary Cross-Entropy

Binary classification

NLL of sigmoid output under Bernoulli model

Categorical Cross-Entropy

Multi-class

NLL of softmax output under Categorical model

MSE

Regression

NLL under Gaussian noise

MAE

Regression (robust to outliers)

NLL under Laplace noise

Focal Loss

Extreme class imbalance

Cross-entropy down-weighting easy examples

Self-Check Questions#

  1. Compute the output of a 2-layer MLP with ReLU on x = [1, 2, 3]ᵀ, W₁ = [[0.5, −1, 0.3], [0.2, 0.8, −0.4]], b₁ = [0.1, −0.2], W₂ = [[1, −1]], b₂ = [0]. Show each step.

  2. Why does composing multiple linear layers without nonlinearities reduce to a single linear layer? Prove it for 2 layers.

  3. Explain the vanishing gradient problem. Which activations cause it, and why does ReLU help?

  4. Why does weight initialization matter? What goes wrong with all-zeros initialization?

  5. A model trained on hospital A’s chest X-rays has 95% accuracy but 70% on hospital B. You increase L2 regularization. Will this help? Why or why not?

  6. The linear-scaling rule says to multiply LR by k when multiplying batch size by k. Under what conditions does this break down?

Additional Resources#