Lecture 22 — Medical imaging: modeling

Lecture 22 — Medical imaging: modeling#

⚠️ AI-synthesized; not fully reviewed by course staff. Treat as a study aid; released slides, notebooks, and lecture recordings are authoritative.

Thu Apr 9, 2026 · Part 3 — Health data modalities · §3.3 Medical imaging

What this lecture is about#

Now that you understand the imaging data (L21), this lecture is about the modeling toolkit. Imaging is the domain where architectural inductive bias does the most work: a CNN’s locality and translation-equivariance assumptions give you a reasonable model with very little data, and a U-Net’s encoder-decoder skip-connection structure is the prior for medical-image segmentation.

We work through the modeling toolbox in the order most projects use it.

Transfer learning from ImageNet. The default starting point. You’d think natural-image features wouldn’t transfer to grayscale chest X-rays — they do, surprisingly well, especially the early layers (which learn edge and texture detectors). Later layers re-learn for the medical domain.
Domain-specific pretraining. RadImageNet (CT/MRI/X-ray), MedSAM (segmentation), CheXzero. Helps when task and domain match the pretraining; can hurt when they don’t.
Self-supervised pretraining. Contrastive (SimCLR, MoCo) and masked-image (MAE) approaches that use unlabeled medical images. Useful when labels are scarce but unlabeled data abound.
Chest X-ray classification (CheXNet, DenseNet-121). The canonical CXR pipeline: ImageNet-pretrained DenseNet, fine-tuned for multi-label CXR pathology. Read the original paper, it set the template.
Segmentation and U-Net. The U-shape: encoder downsamples, decoder upsamples, skip connections preserve spatial detail. Skip connections are the key — without them, the decoder can’t recover boundaries. nnU-Net is the modern auto-configured U-Net that wins most medical-segmentation challenges.
Segmentation losses and metrics. Dice loss (= 1 - Dice coefficient) is the workhorse loss. Weighted cross-entropy when classes are imbalanced. Boundary-aware losses like focal-Tversky and the boundary loss of Kervadec et al. for rare structures; HD95 (95th-percentile Hausdorff distance) is a boundary-aware metric, not a loss — it’s non-differentiable, so you optimize a differentiable surrogate and report HD95 alongside Dice. The metric you optimize ≠ the metric you report.
Multi-instance learning (MIL) for pathology. Whole-slide images are too big for end-to-end training. The MIL formulation: split into patches, encode patches, aggregate to a slide-level prediction. Attention-MIL (Ilse et al.) is the standard aggregator.
Modality-specific augmentation. What’s a valid augmentation for medical images? Random rotation makes sense for histopathology (rotation-invariant) but not for CXRs (heart is on the left). Random intensity changes can simulate scanner variation but can also break the semantic meaning of Hounsfield units. Augmentation is a domain choice.
Class imbalance. Most pathology is rare. Loss reweighting, sampling strategies, and threshold tuning all matter.
Multi-site generalization. The hard problem from L21. Mitigation strategies: federated learning, harmonization (ComBat for radiomics), domain-adversarial training, test-time adaptation. All have caveats.
Garden of forking paths in evaluation. Pre-register your evaluation pipeline. Otherwise the choices made downstream of model training will inflate apparent performance.

Why it matters#

Three reasons this lecture is the practical core of medical-imaging ML:

Architecture choice is the model. A linear classifier on raw pixels won’t work; a deep CNN will. A 2D CNN sliced over volumetric CT will work less well than a 3D CNN that respects the volume structure. A patch classifier on histopathology won’t work; an MIL-aggregated patch classifier will. The architectural prior carries most of the modeling weight.

Segmentation and classification have different evaluation cultures. Classification papers report AUROC; segmentation papers report Dice. They are not interchangeable. A “great Dice on validation” can come with terrible boundary localization (HD95) — and vice versa.

Multi-site generalization is the open problem. A model that is excellent at one hospital and useless at another isn’t a model that “didn’t generalize”; it’s a model whose distribution-shift properties weren’t characterized. The lecture closes with this because it’s the most common failure mode in practice.

Things you should walk away believing#

ImageNet transfer is the conventional default starting point even for grayscale medical images, and it often helps — but the gain is smaller than commonly assumed (Raghu et al., Transfusion, NeurIPS 2019, showed much of the apparent benefit comes from initialization-scale effects rather than from learned features).
Domain-specific pretraining is helpful when task and domain match. It is not a free win.
U-Net is the right structural prior for segmentation. Skip connections are essential, not decorative.
Dice loss + Dice score is the workhorse pair for segmentation; pick boundary-aware metrics for boundary-sensitive tasks.
Multi-instance learning is forced by gigapixel pathology data; attention-MIL is the standard aggregator.
Modality-specific augmentation is a domain choice. Don’t blindly apply ImageNet augmentations.
Multi-site generalization fails by default. Demonstrate it; don’t assume it.

How this connects to the rest of the course#

L13 (NN training mechanics) is the substrate; this lecture is the imaging-specific specialization.
L15 (foundation models) — RadImageNet, MedSAM, BiomedCLIP are imaging foundation models worth examining through the L15 lens.
L24 (fairness) — imaging models inherit subgroup performance gaps when trained on biased datasets.
L10 (generalization, domain shift) — imaging is the modality where the L10 abstract claims become most concrete.
L27 (modern biological AI) — equivariant networks for 3D molecular structure are an extension of imaging-style inductive bias to molecules.

Study guide#

Key terms, self-check questions, and additional resources for active recall.

Key Terms#

Term	Definition
Transfer learning	Start from ImageNet (or domain) pre-trained weights, fine-tune on the target task. The default starting point in medical imaging.
Domain-specific pretraining	Pre-train on a medical-imaging corpus (RadImageNet, MedSAM). Helps when task and domain match.
Self-supervised pretraining	Use unlabeled images (contrastive: SimCLR, MoCo; masked-image: MAE) to learn representations.
DenseNet-121	The canonical CXR-classification backbone (Rajpurkar et al., CheXNet). Dense skip connections.
U-Net	Encoder-decoder with skip connections between corresponding levels. The default segmentation architecture.
Skip connection	Direct path from encoder layer ℓ to decoder layer L−ℓ; preserves spatial detail through the bottleneck. Essential, not decorative.
Dice loss / Dice coefficient	2·\|A∩B\| / (\|A\|+\|B\|). Standard segmentation loss / metric.
HD95	95th-percentile Hausdorff distance — boundary-aware segmentation metric.
MIL aggregator	Function that pools per-patch predictions to a slide-level decision (max, mean, attention-MIL). Attention-MIL (Ilse et al. 2018) is the standard.
Augmentation	Synthetic transformations (rotation, flip, brightness) applied during training. Choice must respect the data: rotations OK on histopathology, not on CXR (heart is on the left).
Multi-site generalization	Deployment across hospitals / scanners / populations. Fails by default unless demonstrated.
Garden of forking paths	Many implicit decisions in the analysis pipeline (preprocessing, splits, augmentation, evaluation metric). Inflates apparent performance unless pre-registered.
nnU-Net	Self-configuring U-Net pipeline (Isensee et al. 2021). Wins most medical-segmentation challenges.

Architecture Choice Is the Model#

You can’t shortcut the architectural prior

Linear classifier on raw pixels won’t work.
Deep CNN will work — but a 2D CNN sliced over volumetric CT works less well than a 3D CNN that respects the volume.
Patch classifier on histopathology won’t work; an MIL-aggregated patch classifier will. The architectural prior carries most of the modeling weight.

Segmentation vs. Classification Cultures#

They are not interchangeable

Classification papers report AUROC; segmentation papers report Dice. They measure different things. A great Dice on validation can come with terrible boundary localization (HD95) — and vice versa. Always look at the metric matched to the task, and ideally a boundary-aware metric for boundary-sensitive tasks.

Self-Check Questions#

ImageNet transfer is surprisingly effective for CXR classification. Why? Which layers tend to transfer well?
U-Net’s skip connections are described as “essential, not decorative.” Walk through what fails without them.
Dice loss is the default for segmentation. Why not just plain cross-entropy on per-pixel labels? When does Dice fail?
MIL with attention pooling: name one advantage over simple mean-pooling per slide.
A model trained on CXRs from hospital A drops in performance on hospital B. Name three mitigation strategies and the assumption each makes.