Lecture 12 — Lab Day #2

Lecture 12 — Lab Day #2#

⚠️ AI-synthesized; not fully reviewed by course staff. Treat as a study aid; released slides, notebooks, and lecture recordings are authoritative.

Thu Feb 26, 2026 · Part 2 — Modern AI & lab work

What this class meeting was about#

Second of the two consecutive lab/Q&A days. No new lecture material. Students kept working through the first lab chunk (labs 0-5, released Sun Feb 22, covering ML model types). The second lab chunk — ML losses — would not release until the following Monday (Mar 2), so the labs in scope on this day are the same as on Lab Day #1.

What this day adds beyond Lab Day #1:

  • Time pressure to actually finish. The next chunk lands Monday; carry-over is allowed but not encouraged.

  • Cross-comparison. With a few days of work behind them, students can now compare results across model classes on the same dataset and start to feel the bias-variance picture from L10 rather than just believe it.

  • Clinic-style debugging. This is the day where most students hit the “my logistic regression is doing weirdly poorly” or “my tree is at 100% training accuracy and 60% test accuracy” moments, and the in-class time is for debugging those rather than introducing new content.

What you should walk away with#

In addition to everything from Lab Day #1:

  • A debugging muscle for “is this a data problem or a model problem?” — the single most useful skill in applied ML.

  • A working comparison table for the six baselines (which performs best on your dataset, why, and how confident you are in that ranking).

  • An understanding that no single number (accuracy, AUROC, F1) is the right summary of a binary classifier — connecting back to L8.

How this connects to the rest of the course#

  • L13 (neural networks) is now the last NN-related lab you’ve seen plus the lecture you haven’t. The contrast between “I trained a small NN in lab5 and it kind of worked” and “here is why architecture and optimization actually matter” is what L13 is built around.

  • L16 (Lab Day #3) will pull in the loss-types chunk (lab 6-11) released the following Monday and the data-types chunk (lab 12-16) released on Mar 6; some carry-over from labs 0-5 is expected.

Study guide#

Key terms, self-check questions, and additional resources for active recall.

Common Debugging Patterns to Build#

Symptom

Common Causes

Train acc ≫ test acc

Overfitting — model too complex, regularization too weak, leakage between splits, or k too small in k-NN.

Train acc ≈ test acc, both low

Underfitting — model class too simple, or task fundamentally hard with the available features.

Wildly variable test acc across runs

High variance — small training set, or stochastic algorithm without seed control.

Logistic regression worse than naive baseline

Feature scaling missing, or label encoding wrong.

k-NN catastrophically slow

High dimensionality / large training set; consider approximate-NN or first dim-reduction.

Tree splits look weird

Feature has near-zero variance, or majority class dominates due to imbalance.

Self-Check Questions#

  1. You see Train AUROC = 0.95 and Test AUROC = 0.65. Walk through the causes you’d investigate, in order.

  2. A lab-mate fits a logistic regression on raw Hounsfield unit values without scaling. The model “works” (AUROC 0.7) — but should they trust it? Why or why not?

  3. A random forest with 100 trees does better than a single decision tree. Identify which mechanism (bagging, feature subsampling, both) is most responsible, and explain.

  4. You evaluate two models with the same AUROC. Calibration plots show one is well-calibrated and one is shifted upward by 0.2 across the score range. Which is “better”? (Trick question — discuss both senses.)

Additional Resources#

Same as L11. In particular: