📝 Lab 13: Ordinal Sequences (Text Data)

📝 Lab 13: Ordinal Sequences (Text Data)#

BINF 4002 – Machine Learning for Health

Learning Objectives#

  1. Understand how text is represented as ordinal sequences of discrete tokens

  2. Load and explore a real text classification dataset

  3. Build baselines using Bag-of-Words and TF-IDF tabularization + classical ML

  4. Understand word embeddings and why dense representations outperform sparse ones

  5. Implement and train a recurrent neural network (LSTM) for sequence classification

  6. Fine-tune a pre-trained Transformer (DistilBERT) via transfer learning

  7. Compare tabularization, recurrent, and Transformer approaches

Why Text / Ordinal Sequences in Healthcare?#

Clinical text is one of the richest data sources in medicine:

  • Clinical notes document patient histories, assessments, and plans in free text

  • Radiology reports describe imaging findings and impressions

  • Pathology reports detail biopsy and specimen analysis

  • Discharge summaries synthesize hospital stays for downstream care

Despite containing critical information, most clinical text is unstructured — it doesn’t fit neatly into rows and columns. Learning to process text as ordered sequences of tokens is essential for extracting actionable information from electronic health records.

Dataset: IMDB Movie Reviews#

We use the IMDB sentiment classification dataset (50K movie reviews, binary positive/negative) as a clean, well-understood benchmark. While not clinical, the NLP techniques are identical to those used for clinical text classification (e.g., detecting suicide risk from notes, classifying radiology findings, extracting diagnoses from discharge summaries).

We will note clinical parallels throughout the lab.

Set-up#

Install Dependencies#

This lab requires PyTorch, HuggingFace Datasets, and HuggingFace Transformers. Run the cell below to install everything. (This may take 1–2 minutes on Colab.)

# ── Install dependencies ──────────────────────────────────────────────────
import subprocess, sys

def install(pkg):
    subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", pkg])

try:
    import torch
    print(f"✅ PyTorch {torch.__version__}")
except ImportError:
    install("torch")
    import torch
    print(f"✅ PyTorch {torch.__version__} installed")

try:
    import datasets
    print(f"✅ HuggingFace Datasets {datasets.__version__}")
except ImportError:
    install("datasets")
    import datasets
    print(f"✅ HuggingFace Datasets {datasets.__version__} installed")

try:
    import transformers
    print(f"✅ HuggingFace Transformers {transformers.__version__}")
except ImportError:
    install("transformers")
    import transformers
    print(f"✅ HuggingFace Transformers {transformers.__version__} installed")

try:
    import sklearn
    print(f"✅ scikit-learn {sklearn.__version__}")
except ImportError:
    install("scikit-learn")

Imports#

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings, re, collections
warnings.filterwarnings('ignore')

plt.rcParams['figure.dpi'] = 120
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.right'] = False

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset
from torch.nn.utils.rnn import pad_sequence

from datasets import load_dataset

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import roc_auc_score, roc_curve, classification_report
from sklearn.decomposition import TruncatedSVD

print("✅ All imports successful")
print(f"   PyTorch: {torch.__version__}")
print(f"   CUDA available: {torch.cuda.is_available()}")

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"   Using device: {device}")

Part 1 — What Are Ordinal Sequences?#

An ordinal sequence is data where:

  1. Each sample is a variable-length sequence of discrete tokens: \(\mathbf{x} = (x_1, x_2, \ldots, x_T)\)

  2. The order of tokens carries meaning — rearranging them changes the semantics

Text is the canonical example: a sentence is a sequence of words (or subwords), and word order is critical to meaning.

Why Order Matters#

Consider these clinical notes — same words, very different meanings:

Note

Interpretation

“Patient denied chest pain, but reported a headache”

No chest pain, headache present

“Patient reported chest pain, but denied a headache””

Chest pain present, no headache

A model that ignores word order (like Bag-of-Words) would see both as containing “patient,” “chest pain” and “headache” — and might classify them identically. Sequence-aware models can distinguish them.

From Text to Numbers#

Models need numerical input. The pipeline is:

\[\text{Raw text} \xrightarrow{\text{tokenize}} \text{Token sequence} \xrightarrow{\text{encode}} \text{Integer sequence} \xrightarrow{\text{embed}} \text{Vector sequence}\]

  1. Tokenization: split text into tokens (words, subwords, or characters)

  2. Encoding: map each unique token to an integer ID via a vocabulary

  3. Embedding: map each integer ID to a dense vector (learned or pre-trained)

# ── Tokenization demo: same words, different order ────────────────────────
# Simple whitespace tokenizer (real tokenizers handle punctuation, casing, etc.)

def simple_tokenize(text):
    return re.findall(r'\b\w+\b', text.lower())

# Clinical text examples where order matters
examples = [
    "patient denied chest pain and shortness of breath",
    "patient reported chest pain and shortness of breath",
    "no evidence of malignancy in the tissue sample",
    "evidence of malignancy in the tissue sample",
]

print("── Tokenization and the importance of order ──\n")
for text in examples:
    tokens = simple_tokenize(text)
    print(f"Text:   '{text}'")
    print(f"Tokens: {tokens}")
    print(f"Length: {len(tokens)} tokens\n")

# Show that BoW loses the distinction
from collections import Counter
bow1 = Counter(simple_tokenize(examples[0]))
bow2 = Counter(simple_tokenize(examples[1]))
shared = set(bow1.keys()) & set(bow2.keys())
diff = (set(bow1.keys()) | set(bow2.keys())) - shared
print(f"\n── Bag-of-Words comparison ──")
print(f"Sentence 1 tokens: {set(bow1.keys())}")
print(f"Sentence 2 tokens: {set(bow2.keys())}")
print(f"Shared tokens: {shared}")
print(f"Differing tokens: {diff}")
print(f"Only 1 word differs — but the clinical meaning is opposite!")

# ── Building a vocabulary and encoding text as integers ────────────────────
# This is what every NLP model does under the hood

corpus = [
    "the patient has a fever",
    "the patient denied chest pain",
    "chest x-ray shows no abnormality",
    "patient reports mild fever and cough",
]

# Step 1: Tokenize
tokenized = [simple_tokenize(sent) for sent in corpus]

# Step 2: Build vocabulary (token → integer)
all_tokens = [tok for sent in tokenized for tok in sent]
vocab = {"<PAD>": 0, "<UNK>": 1}  # special tokens
for tok in sorted(set(all_tokens)):
    if tok not in vocab:
        vocab[tok] = len(vocab)

print(f"Vocabulary size: {len(vocab)}")
print(f"Vocabulary: {vocab}")

# Step 3: Encode sentences as integer sequences
encoded = [[vocab.get(tok, 1) for tok in sent] for sent in tokenized]
print(f"\n── Encoded sequences ──")
for sent, enc in zip(corpus, encoded):
    print(f"  '{sent}'")
    print(f"   → {enc}  (length {len(enc)})")

# Step 4: Pad to same length for batching
max_len = max(len(e) for e in encoded)
padded = [e + [0] * (max_len - len(e)) for e in encoded]
padded_tensor = torch.tensor(padded)
print(f"\nPadded tensor shape: {padded_tensor.shape}  (batch_size × max_length)")
print(padded_tensor)

🤔 Reflection 1.1 — Text as Ordinal Sequences#

  1. We used a simple whitespace tokenizer above. Real tokenizers (like WordPiece or BPE) split words into subwords — e.g., “unhappiness” → [“un”, “##happiness”]. Why might subword tokenization be better than word-level tokenization for clinical text, which contains many rare medical terms (e.g., “cardiomyopathy”, “thrombocytopenia”)?

  2. We padded all sequences to the same length with zeros. What is the computational cost of padding? If your longest document has 5,000 tokens but the median is 200, what strategies could you use to avoid wasting computation on padding?

  3. In our toy vocabulary, unknown words map to <UNK>. In clinical text, new drug names, abbreviations, and misspellings appear constantly. How does the <UNK> token affect model performance? How do subword tokenizers help mitigate this?

  4. Compare text sequences to the molecular graphs from Lab 12. Both are variable-length structured data. What are the key structural differences? (Hint: think about topology.)

# ══ SOLUTION — Reflection 1.1 ═══════════════════════════════════════════════
# 1. Word-level tokenization assigns <UNK> to any word not in the vocabulary.
#    Medical terms like "thrombocytopenia" might be rare enough to be OOV (out-of-
#    vocabulary). Subword tokenization breaks it into known pieces: "thrombo" + "cyto"
#    + "penia" — each piece carries meaning (thrombo=clot, cyto=cell, penia=deficiency).
#    This allows the model to generalize to unseen medical terms by composing known
#    morphemes, dramatically reducing the OOV problem.
#
# 2. Padding wastes memory and computation (LSTM processes pad tokens; attention
#    computes over them). Strategies: (a) truncate to a max length (e.g., 512),
#    (b) batch sequences of similar length together (dynamic batching / bucketing),
#    (c) use packed sequences in PyTorch (nn.utils.rnn.pack_padded_sequence),
#    (d) use attention masks to ignore padding in Transformer models.
#
# 3. <UNK> collapses ALL unknown words to a single representation — "aspirin" and
#    "pembrolizumab" become identical if both are OOV. This is catastrophic for
#    clinical NLP where drug names matter. Subword tokenizers virtually eliminate
#    OOV by breaking unknown words into known subword pieces, preserving at least
#    partial semantic information.
#
# 4. Text is a LINEAR sequence (each token has at most 2 neighbors: previous and
#    next). Graphs have arbitrary topology (a node can connect to any number of
#    other nodes). Text has a natural ordering; graphs are permutation-invariant
#    over nodes. However, both share the challenge of variable length and the need
#    for aggregation (pooling node embeddings in graphs, pooling token embeddings
#    in text) to produce fixed-size representations for classification.
print("See comments above for solution.")

Part 2 — Loading and Exploring Text Data#

We load the IMDB movie review dataset — 50K reviews labeled positive or negative. To keep training fast on Colab, we subsample to 8K train / 1K validation / 1K test.

# ── Load and subsample the IMDB dataset ────────────────────────────────────
print("Loading IMDB dataset from HuggingFace...")
raw_dataset = load_dataset("imdb")

# Subsample for Colab speed
np.random.seed(42)
train_indices = np.random.choice(len(raw_dataset['train']), size=8000, replace=False)
test_indices = np.random.choice(len(raw_dataset['test']), size=2000, replace=False)

train_texts = [raw_dataset['train'][int(i)]['text'] for i in train_indices]
train_labels = [raw_dataset['train'][int(i)]['label'] for i in train_indices]

# Split test into val and test
val_texts = [raw_dataset['test'][int(i)]['text'] for i in test_indices[:1000]]
val_labels = [raw_dataset['test'][int(i)]['label'] for i in test_indices[:1000]]
test_texts = [raw_dataset['test'][int(i)]['text'] for i in test_indices[1000:]]
test_labels = [raw_dataset['test'][int(i)]['label'] for i in test_indices[1000:]]

y_train = np.array(train_labels)
y_val = np.array(val_labels)
y_test = np.array(test_labels)

print(f"\nTrain: {len(train_texts)}, Val: {len(val_texts)}, Test: {len(test_texts)}")
print(f"Train positive rate: {y_train.mean():.3f}")
print(f"Val positive rate:   {y_val.mean():.3f}")

# ── Explore the text data ──────────────────────────────────────────────────
# Look at a few examples
for i in [0, 1]:
    label = "Positive" if train_labels[i] == 1 else "Negative"
    text_preview = train_texts[i][:300].replace('\n', ' ')
    print(f"── Example {i+1} ({label}) ──")
    print(f"   {text_preview}...")
    print()

# Text length distribution
train_lengths = [len(simple_tokenize(t)) for t in train_texts]

fig, axes = plt.subplots(1, 3, figsize=(15, 4))

axes[0].hist(train_lengths, bins=50, color='#3498db', edgecolor='white', alpha=0.8)
axes[0].set_xlabel('Number of Words')
axes[0].set_ylabel('Count')
axes[0].set_title('Review Length Distribution')
axes[0].axvline(np.median(train_lengths), color='#e74c3c', linestyle='--',
                label=f'Median: {np.median(train_lengths):.0f}')
axes[0].legend()

# Log scale to see the tail
axes[1].hist(train_lengths, bins=50, color='#3498db', edgecolor='white', alpha=0.8)
axes[1].set_xlabel('Number of Words')
axes[1].set_ylabel('Count (log)')
axes[1].set_title('Review Length (log scale)')
axes[1].set_yscale('log')

# Label distribution
counts = pd.Series(y_train).value_counts().sort_index()
axes[2].bar(['Negative (0)', 'Positive (1)'], counts.values,
           color=['#e74c3c', '#3498db'], edgecolor='white')
axes[2].set_ylabel('Count')
axes[2].set_title('Label Distribution')
for i, v in enumerate(counts.values):
    axes[2].text(i, v + 20, str(v), ha='center', fontsize=11)

plt.tight_layout()
plt.show()

print(f"\nWords per review: mean={np.mean(train_lengths):.0f}, "
      f"median={np.median(train_lengths):.0f}, "
      f"min={np.min(train_lengths)}, max={np.max(train_lengths)}")

🤔 Reflection 2.1 — Understanding Text Data Properties#

  1. The review lengths vary enormously (some are 50 words, others 2,000+). How does this compare to the variable molecule sizes in Lab 12? Which is more extreme?

  2. Unlike the BBBP dataset (75% positive), IMDB is roughly balanced. When would you expect class imbalance in clinical text classification tasks? (Name two examples.)

  3. Movie reviews express sentiment through word choice, sarcasm, and narrative structure. Clinical notes express findings through medical vocabulary, negation (“no evidence of”), and structured sections (HPI, Assessment, Plan). What challenges unique to clinical text might cause models trained on general English to perform poorly?

  4. Some reviews contain HTML tags (e.g., <br />). In clinical text, you might encounter templates, copy-pasted sections, and structured data mixed with free text. How might you preprocess clinical text differently from movie reviews?

# ══ SOLUTION — Reflection 2.1 ═══════════════════════════════════════════════
# 1. Both vary significantly, but text length variation is more extreme. Molecules
#    in BBBP ranged from ~5 to ~130 atoms (26x range). Reviews range from ~10 to
#    ~2500 words (250x range). Clinical notes can be even more extreme — a brief
#    follow-up note might be 50 words while a discharge summary is 5,000+.
#
# 2. Clinical text imbalance is very common: (a) detecting rare adverse drug events
#    from clinical notes (~1% positive rate), (b) identifying suicide risk mentions
#    in psychiatric notes (~5% positive). Most clinical NLP tasks have severe class
#    imbalance because serious events are (thankfully) rare.
#
# 3. Clinical text challenges: (a) heavy use of negation and hedging ("no evidence
#    of," "unlikely to be"), (b) abbreviations and jargon ("pt c/o SOB x 3d"),
#    (c) domain-specific vocabulary, (d) inconsistent formatting across providers,
#    (e) copy-paste artifacts (repeated text from previous notes), (f) implicit
#    knowledge (clinicians omit "obvious" context). Models trained on general English
#    miss these patterns — which is why domain-specific pre-training (e.g., ClinicalBERT,
#    PubMedBERT) is so important.
#
# 4. Clinical preprocessing: (a) handle de-identification artifacts ([NAME], [DATE]),
#    (b) normalize abbreviations, (c) detect and handle copy-pasted sections,
#    (d) separate structured templates from free text, (e) preserve negation scope
#    (don't just remove "no" as a stopword!). For movie reviews, simple HTML stripping
#    and lowercasing usually suffice.
print("See comments above for solution.")

Part 3 — Baseline: Tabularization with Bag-of-Words and TF-IDF#

Just as we converted graphs into fixed-length feature vectors in Lab 12, we can convert variable-length text into fixed-length vectors by discarding word order.

Bag-of-Words (BoW)#

Count how many times each vocabulary word appears in each document:

\[\text{BoW}(\mathbf{x})_j = \sum_{t=1}^{T} \mathbb{1}[x_t = j]\]

For a vocabulary of size \(V\), each document becomes a \(V\)-dimensional count vector.

TF-IDF (Term Frequency — Inverse Document Frequency)#

BoW weights all words equally, but common words like “the” and “is” are uninformative. TF-IDF down-weights words that appear in many documents:

\[\text{TF-IDF}(t, d) = \text{TF}(t, d) \times \log\frac{N}{\text{DF}(t)}\]

where \(\text{TF}(t, d)\) is term frequency in document \(d\), \(N\) is total documents, and \(\text{DF}(t)\) is the number of documents containing term \(t\).

Both representations throw away word order — this is the key trade-off.

# This is an exercise cell with TODOs (??? placeholders) for student work.
# It is skipped in the book build to keep downstream cells runnable.
# Open the notebook in Colab to complete the exercise:
# Exercise cell skipped in book build
print("[exercise cell skipped — open in Colab to complete the TODOs]")

# ══ SOLUTION — build_bow_matrix ══════════════════════════════════════════════
def build_bow_matrix(documents, vocab):
    n_docs = len(documents)
    vocab_size = len(vocab)
    matrix = np.zeros((n_docs, vocab_size))

    for i, doc in enumerate(documents):
        tokens = simple_tokenize(doc)
        for token in tokens:
            if token in vocab:
                matrix[i, vocab[token]] += 1

    return matrix

# Rebuild with solution
X_train_bow = build_bow_matrix(train_texts, bow_vocab)
X_val_bow = build_bow_matrix(val_texts, bow_vocab)
X_test_bow = build_bow_matrix(test_texts, bow_vocab)

print(f"BoW matrix shape: {X_train_bow.shape}")
print(f"Sparsity: {(X_train_bow == 0).mean():.1%} zeros")
print(f"Top-5 vocab: {list(bow_vocab.keys())[:5]}")

# ── TF-IDF using sklearn (the standard approach) ─────────────────────────
tfidf = TfidfVectorizer(max_features=10000, min_df=3, max_df=0.95,
                        ngram_range=(1, 2))  # unigrams + bigrams
X_train_tfidf = tfidf.fit_transform(train_texts)
X_val_tfidf = tfidf.transform(val_texts)
X_test_tfidf = tfidf.transform(test_texts)

print(f"TF-IDF matrix shape: {X_train_tfidf.shape}")
print(f"Sparsity: {(1 - X_train_tfidf.nnz / np.prod(X_train_tfidf.shape)):.1%} zeros")

# Show top TF-IDF features for a positive and negative review
feature_names = tfidf.get_feature_names_out()
for label_val, label_name in [(1, "Positive"), (0, "Negative")]:
    idx = np.where(y_train == label_val)[0][0]
    row = X_train_tfidf[idx].toarray().flatten()
    top_feat_idx = row.argsort()[-8:][::-1]
    top_feats = [(feature_names[j], row[j]) for j in top_feat_idx]
    print(f"\nTop TF-IDF features for a {label_name} review:")
    for feat, score in top_feats:
        print(f"  {feat:20s} {score:.3f}")

# ── Classical ML on tabularized text ──────────────────────────────────────
print("═══ Tabularization Baselines ═══\n")

# BoW baselines
print("── Bag-of-Words (5K vocab) ──")
bow_models = {
    'LR + BoW': LogisticRegression(max_iter=1000, C=1.0, random_state=42),
    'RF + BoW': RandomForestClassifier(n_estimators=200, max_depth=20, random_state=42),
}
bow_results = {}
for name, model in bow_models.items():
    model.fit(X_train_bow, y_train)
    val_proba = model.predict_proba(X_val_bow)[:, 1]
    val_auc = roc_auc_score(y_val, val_proba)
    bow_results[name] = {'model': model, 'val_auc': val_auc}
    print(f"  {name:25s}  Val AUROC = {val_auc:.4f}")

# TF-IDF baselines
print("\n── TF-IDF (10K features, unigrams + bigrams) ──")
tfidf_models = {
    'LR + TF-IDF': LogisticRegression(max_iter=1000, C=1.0, random_state=42),
    'RF + TF-IDF': RandomForestClassifier(n_estimators=200, max_depth=20, random_state=42),
    'GBT + TF-IDF': GradientBoostingClassifier(n_estimators=200, max_depth=5, random_state=42),
}
tfidf_results = {}
for name, model in tfidf_models.items():
    model.fit(X_train_tfidf, y_train)
    val_proba = model.predict_proba(X_val_tfidf)[:, 1]
    val_auc = roc_auc_score(y_val, val_proba)
    tfidf_results[name] = {'model': model, 'val_auc': val_auc}
    print(f"  {name:25s}  Val AUROC = {val_auc:.4f}")

🤔 Reflection 3.1 — Tabularization Trade-offs#

  1. Compare BoW vs. TF-IDF performance. Why does TF-IDF typically outperform raw counts? What is the effect of adding bigrams (2-word sequences like “not good”) to the TF-IDF?

  2. The TF-IDF matrix has 10,000 columns but is >99% sparse. Why? What does this sparsity mean for model choice? (Hint: which classical ML models handle high-dimensional sparse data well, and which don’t?)

  3. BoW and TF-IDF both discard word order. The phrase “not bad” should be positive, but BoW treats “not” and “bad” independently (both are negative signals). How much does this hurt in practice? Why might TF-IDF with bigrams partially mitigate this?

  4. In clinical NLP, TF-IDF + Logistic Regression is often a strong baseline. Why might simpler models be especially competitive in the clinical domain compared to movie reviews? (Hint: think about the nature of clinical language vs. colloquial language.)

# ══ SOLUTION — Reflection 3.1 ═══════════════════════════════════════════════
# 1. TF-IDF outperforms raw counts because it down-weights common, uninformative
#    words ("the", "a", "is") while up-weighting discriminative words ("excellent",
#    "terrible"). Bigrams capture some word order — "not good" becomes a single
#    feature distinct from "not" and "good" separately. This helps with negation,
#    which is especially important in clinical text ("no evidence of").
#
# 2. Each document uses only ~100-500 unique words out of 10,000 vocabulary items.
#    Most features are zero for any given document. Logistic regression handles this
#    well (it learns sparse weights). Random forests struggle with very high-dimensional
#    sparse data (random feature subsampling rarely selects the informative features).
#    This is why LR typically beats RF on text data.
#
# 3. In practice, the loss from discarding order is moderate for many tasks. Sentiment
#    is often well-captured by which words appear (positive words = positive review).
#    But for tasks requiring understanding of negation, comparison, or causality, order
#    matters more. Bigrams help because "not good" and "not bad" become distinct features.
#    However, they can't capture longer-range dependencies ("I thought this would be good
#    but it was actually terrible").
#
# 4. Clinical language is more formulaic and keyword-rich than colloquial language.
#    Clinicians use specific terminology: "metastatic" directly signals cancer stage,
#    "code blue" signals cardiac arrest. There's less sarcasm, metaphor, or stylistic
#    variation. This means keyword-based models capture more of the signal. Additionally,
#    clinical NLP datasets are often small, favoring simpler models that don't overfit.
print("See comments above for solution.")

Part 4 — Word Embeddings: From Sparse to Dense#

BoW/TF-IDF vectors are sparse and high-dimensional — each word is a one-hot vector with no notion of similarity. “Good” and “great” are as far apart as “good” and “terrible.”

Word embeddings map each token to a dense, low-dimensional vector (typically 50–300 dimensions) where semantically similar words are nearby:

\[\text{embed}: \text{vocab index } j \mapsto \mathbf{e}_j \in \mathbb{R}^d\]

The embedding matrix \(\mathbf{E} \in \mathbb{R}^{V \times d}\) is either:

  • Learned from scratch during training (what we’ll do for the LSTM)

  • Pre-trained on a large corpus (Word2Vec, GloVe, or the embeddings inside BERT)

Why Embeddings Matter#

Representation

Dimensionality

“good” vs “great”

“good” vs “terrible”

One-hot

\(V = 10{,}000\)

distance = \(\sqrt{2}\)

distance = \(\sqrt{2}\)

Embedding

\(d = 128\)

distance ≈ 0.3 (close!)

distance ≈ 1.8 (far!)

Embeddings encode semantic relationships: \(\text{king} - \text{man} + \text{woman} \approx \text{queen}\)

# ── Visualize: TF-IDF (sparse) vs. embedding (dense) representations ─────
# Use SVD to project TF-IDF into 2D for visualization
svd = TruncatedSVD(n_components=2, random_state=42)
tfidf_2d = svd.fit_transform(X_train_tfidf[:500])

fig, axes = plt.subplots(1, 2, figsize=(13, 5))

# TF-IDF in 2D
colors = ['#e74c3c' if l == 0 else '#3498db' for l in y_train[:500]]
axes[0].scatter(tfidf_2d[:, 0], tfidf_2d[:, 1], c=colors, alpha=0.4, s=15)
axes[0].set_title('TF-IDF (SVD → 2D)', fontsize=12)
axes[0].set_xlabel('Component 1'); axes[0].set_ylabel('Component 2')
red_patch = plt.Line2D([0], [0], marker='o', color='w', markerfacecolor='#e74c3c',
                        markersize=8, label='Negative')
blue_patch = plt.Line2D([0], [0], marker='o', color='w', markerfacecolor='#3498db',
                         markersize=8, label='Positive')
axes[0].legend(handles=[red_patch, blue_patch])

# Show sparse vs dense comparison
sparse_vec = np.zeros(50)
sparse_vec[7] = 1; sparse_vec[23] = 1; sparse_vec[41] = 1
dense_vec = np.random.randn(50) * 0.3

axes[1].bar(range(50), sparse_vec, color='#e74c3c', alpha=0.7, label='Sparse (one-hot)')
axes[1].bar(range(50), dense_vec, color='#3498db', alpha=0.5, label='Dense (embedding)')
axes[1].set_title('Sparse vs Dense Representation', fontsize=12)
axes[1].set_xlabel('Dimension'); axes[1].set_ylabel('Value')
axes[1].legend()

plt.tight_layout()
plt.show()

🤔 Reflection 4.1 — Embeddings and Representation#

  1. In one-hot encoding, every pair of words is equidistant. Why is this a problem for generalization? If the model learns that “excellent” is positive, how does it know that “outstanding” is also positive without seeing it in training?

  2. Embeddings are learned during training (or pre-trained). What happens to the embeddings of rare words that appear only a few times in training? How does this relate to the problem of rare medical terms in clinical NLP?

  3. The embedding matrix \(\mathbf{E} \in \mathbb{R}^{V \times d}\) has \(V \times d\) parameters. With \(V = 10{,}000\) and \(d = 128\), that’s 1.28M parameters just for the embedding layer. On a dataset of 8K documents, is this likely to overfit? What strategies could help?

  4. Pre-trained embeddings (GloVe, Word2Vec) are trained on billions of words. Why might these be less useful for clinical text than for movie reviews? What is the alternative?

# ══ SOLUTION — Reflection 4.1 ═══════════════════════════════════════════════
# 1. With one-hot encoding, the model must independently learn that each synonym is
#    positive. Having seen "excellent = positive" gives zero information about
#    "outstanding." With embeddings, "excellent" and "outstanding" are nearby in
#    vector space, so learning about one transfers to the other. This is a form of
#    parameter sharing — similar words share similar representations.
#
# 2. Rare words have poorly trained embeddings because they're updated by very few
#    gradient steps. They tend to stay near their random initialization. This is a
#    major problem in clinical NLP where medical terms may appear only a handful of
#    times. Solutions: (a) use pre-trained domain-specific embeddings (BioWordVec),
#    (b) use subword tokenization so rare words decompose into well-trained pieces,
#    (c) use pre-trained language models that have seen these terms in large corpora.
#
# 3. Yes, 1.28M embedding parameters with only 8K training examples is a recipe for
#    overfitting. Strategies: (a) use pre-trained embeddings and freeze them,
#    (b) reduce vocabulary size, (c) reduce embedding dimension, (d) add dropout
#    after the embedding layer, (e) use regularization. In practice, pre-trained
#    embeddings or pre-trained transformers are the most effective solutions.
#
# 4. Pre-trained embeddings from general corpora (Wikipedia, news) may not capture
#    domain-specific semantics. In clinical text, "discharge" means leaving the
#    hospital (not firing an employee), "positive" often means disease-present (not
#    good). The alternative is domain-specific pre-training: BioWordVec (trained on
#    PubMed), ClinicalBERT (trained on clinical notes), PubMedBERT, etc.
print("See comments above for solution.")

Part 5 — Recurrent Neural Networks: LSTM#

Unlike BoW/TF-IDF, recurrent neural networks (RNNs) process sequences in order, maintaining a hidden state \(\mathbf{h}_t\) that accumulates information:

\[\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t)\]

The LSTM (Long Short-Term Memory) is the most widely used RNN variant. It uses gates to control what information to keep, forget, and output:

  • Forget gate \(\mathbf{f}_t\): what to discard from the cell state

  • Input gate \(\mathbf{i}_t\): what new information to store

  • Output gate \(\mathbf{o}_t\): what to output as the hidden state

After processing the entire sequence, the final hidden state \(\mathbf{h}_T\) is used as the sequence representation for classification.

Our Pipeline#

\[\text{Tokens} \xrightarrow{\text{Embedding}} \text{Dense vectors} \xrightarrow{\text{LSTM}} \text{Hidden states} \xrightarrow{\text{Pool last}} \mathbf{h}_T \xrightarrow{\text{Linear}} \hat{y}\]
# ── Prepare data for PyTorch ───────────────────────────────────────────────
# Build vocabulary from training data
print("Building vocabulary...")
token_counts_full = Counter()
for text in train_texts:
    token_counts_full.update(simple_tokenize(text))

# Keep top 15K tokens
VOCAB_SIZE = 15000
MAX_LEN = 256  # truncate to 256 tokens for speed

vocab_full = {"<PAD>": 0, "<UNK>": 1}
for tok, _ in token_counts_full.most_common(VOCAB_SIZE - 2):
    vocab_full[tok] = len(vocab_full)

print(f"Vocabulary size: {len(vocab_full)}")

def encode_texts(texts, vocab, max_len):
    # Tokenize, encode, truncate, and pad a list of texts.
    encoded = []
    lengths = []
    for text in texts:
        tokens = simple_tokenize(text)[:max_len]
        ids = [vocab.get(tok, 1) for tok in tokens]  # 1 = <UNK>
        lengths.append(len(ids))
        # Pad to max_len
        ids = ids + [0] * (max_len - len(ids))
        encoded.append(ids)
    return torch.tensor(encoded, dtype=torch.long), torch.tensor(lengths, dtype=torch.long)

X_train_enc, train_lens = encode_texts(train_texts, vocab_full, MAX_LEN)
X_val_enc, val_lens = encode_texts(val_texts, vocab_full, MAX_LEN)
X_test_enc, test_lens = encode_texts(test_texts, vocab_full, MAX_LEN)

y_train_t = torch.tensor(y_train, dtype=torch.float32)
y_val_t = torch.tensor(y_val, dtype=torch.float32)
y_test_t = torch.tensor(y_test, dtype=torch.float32)

# Create DataLoaders
train_ds = TensorDataset(X_train_enc, train_lens, y_train_t)
val_ds = TensorDataset(X_val_enc, val_lens, y_val_t)
test_ds = TensorDataset(X_test_enc, test_lens, y_test_t)

train_dl = DataLoader(train_ds, batch_size=64, shuffle=True)
val_dl = DataLoader(val_ds, batch_size=64, shuffle=False)
test_dl = DataLoader(test_ds, batch_size=64, shuffle=False)

print(f"\nEncoded tensor shape: {X_train_enc.shape} (n_samples x max_len)")
print(f"Example encoded: {X_train_enc[0, :15]}...")

# This is an exercise cell with TODOs (??? placeholders) for student work.
# It is skipped in the book build to keep downstream cells runnable.
# Open the notebook in Colab to complete the exercise:
# Exercise cell skipped in book build
print("[exercise cell skipped — open in Colab to complete the TODOs]")

# ══ SOLUTION — LSTMClassifier ════════════════════════════════════════════════
class LSTMClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim=128, hidden_dim=128, dropout=0.3):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True,
                            bidirectional=True, num_layers=1)
        self.dropout = nn.Dropout(dropout)
        self.classifier = nn.Linear(hidden_dim * 2, 1)

    def forward(self, input_ids, lengths):
        embeds = self.embedding(input_ids)
        embeds = self.dropout(embeds)

        packed = nn.utils.rnn.pack_padded_sequence(
            embeds, lengths.cpu().clamp(min=1), batch_first=True, enforce_sorted=False)

        packed_out, (h_n, c_n) = self.lstm(packed)

        # h_n: (2, batch, hidden_dim) for bidirectional
        h_final = torch.cat([h_n[0], h_n[1]], dim=1)
        h_final = self.dropout(h_final)

        out = torch.sigmoid(self.classifier(h_final)).squeeze(-1)
        return out

lstm_model = LSTMClassifier(vocab_size=len(vocab_full))
print(lstm_model)
print(f"\nTotal parameters: {sum(p.numel() for p in lstm_model.parameters()):,}")

# ── Training loop ─────────────────────────────────────────────────────────
def train_epoch_lstm(model, loader, optimizer, criterion):
    model.train()
    total_loss = 0
    n = 0
    for input_ids, lengths, labels in loader:
        input_ids, lengths, labels = input_ids.to(device), lengths.to(device), labels.to(device)
        optimizer.zero_grad()
        out = model(input_ids, lengths)
        loss = criterion(out, labels)
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)  # gradient clipping
        optimizer.step()
        total_loss += loss.item() * len(labels)
        n += len(labels)
    return total_loss / n

@torch.no_grad()
def evaluate_lstm(model, loader):
    model.eval()
    all_preds, all_labels = [], []
    total_loss = 0
    n = 0
    criterion = nn.BCELoss()
    for input_ids, lengths, labels in loader:
        input_ids, lengths, labels = input_ids.to(device), lengths.to(device), labels.to(device)
        out = model(input_ids, lengths)
        total_loss += criterion(out, labels).item() * len(labels)
        all_preds.append(out.cpu().numpy())
        all_labels.append(labels.cpu().numpy())
        n += len(labels)
    preds = np.concatenate(all_preds)
    labels = np.concatenate(all_labels)
    auc = roc_auc_score(labels, preds)
    return auc, total_loss / n

def train_lstm_model(model, train_dl, val_dl, lr=1e-3, epochs=15, patience=5):
    model = model.to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=lr)
    criterion = nn.BCELoss()

    history = {'train_loss': [], 'val_loss': [], 'train_auc': [], 'val_auc': []}
    best_val_auc = 0
    best_state = None
    wait = 0

    for epoch in range(1, epochs + 1):
        train_loss = train_epoch_lstm(model, train_dl, optimizer, criterion)
        train_auc, _ = evaluate_lstm(model, train_dl)
        val_auc, val_loss = evaluate_lstm(model, val_dl)

        history['train_loss'].append(train_loss)
        history['val_loss'].append(val_loss)
        history['train_auc'].append(train_auc)
        history['val_auc'].append(val_auc)

        if val_auc > best_val_auc:
            best_val_auc = val_auc
            best_state = {k: v.clone() for k, v in model.state_dict().items()}
            wait = 0
        else:
            wait += 1

        print(f"Epoch {epoch:2d} | Train Loss: {train_loss:.4f} | "
              f"Train AUC: {train_auc:.4f} | Val AUC: {val_auc:.4f} | Best: {best_val_auc:.4f}")

        if wait >= patience:
            print(f"Early stopping at epoch {epoch}")
            break

    model.load_state_dict(best_state)
    return model, history

# ── Train the LSTM ─────────────────────────────────────────────────────────
print("=" * 60)
print("Training Bidirectional LSTM")
print("=" * 60)
lstm_model = LSTMClassifier(vocab_size=len(vocab_full), embed_dim=128,
                            hidden_dim=128, dropout=0.3)
lstm_model, lstm_history = train_lstm_model(lstm_model, train_dl, val_dl,
                                            lr=1e-3, epochs=15, patience=5)

# Plot training curves
fig, axes = plt.subplots(1, 2, figsize=(13, 4))

axes[0].plot(lstm_history['train_loss'], label='Train', color='#3498db')
axes[0].plot(lstm_history['val_loss'], label='Val', color='#e74c3c')
axes[0].set_xlabel('Epoch'); axes[0].set_ylabel('BCE Loss')
axes[0].set_title('LSTM — Loss Curves'); axes[0].legend()

axes[1].plot(lstm_history['train_auc'], label='Train', color='#3498db')
axes[1].plot(lstm_history['val_auc'], label='Val', color='#e74c3c')
axes[1].set_xlabel('Epoch'); axes[1].set_ylabel('AUROC')
axes[1].set_title('LSTM — AUROC Curves'); axes[1].legend()

plt.tight_layout()
plt.show()

🤔 Reflection 5.1 — Recurrent Neural Networks#

  1. The LSTM has ~2.1M parameters (1.92M in the embedding alone). Compare this to the number of training examples (8K). What is the parameters-to-samples ratio? How does this compare to the classical ML baselines?

  2. We used bidirectional LSTM, which reads the text both left-to-right and right-to-left. Why is this important for text understanding? Give a clinical example where backward context helps interpret a word.

  3. We used pack_padded_sequence and gradient clipping. What problem does each solve? What would happen if we removed them?

  4. The LSTM processes tokens sequentially — it cannot be parallelized across timesteps. For a 256-token sequence, the forward pass requires 256 sequential steps. How does this compare to the Transformer architecture we’ll see in Part 6? What are the implications for training speed?

# ══ SOLUTION — Reflection 5.1 ═══════════════════════════════════════════════
# 1. ~2.1M parameters / 8K samples = ~260 parameters per sample. This is extremely
#    high — classical models with 10K TF-IDF features have far fewer effective
#    parameters (LR has 10K weights, RF even fewer per tree). This ratio makes
#    overfitting very likely, which is why we see the train-val gap in the curves.
#    Regularization (dropout), early stopping, and pre-trained embeddings all help.
#
# 2. Bidirectional reading captures both preceding and following context. Clinical
#    example: "No signs of [pneumonia] were observed." The word "pneumonia" is
#    modified by "No signs of" (preceding context, left-to-right) AND "were observed"
#    (following context, right-to-left). A forward-only LSTM at "pneumonia" hasn't
#    yet seen "were observed," so it might miss that this is a descriptive clause.
#    The backward LSTM at "pneumonia" has already processed "were observed."
#
# 3. pack_padded_sequence tells the LSTM to skip padding tokens, saving computation
#    and preventing the model from learning spurious patterns from padding. Gradient
#    clipping prevents exploding gradients, a known problem with RNNs where gradients
#    can grow exponentially over long sequences. Without clipping, training can
#    diverge (loss → NaN). Without packing, the LSTM wastes computation on padding
#    and the hidden states get "polluted" by processing many zero vectors.
#
# 4. LSTMs are inherently sequential: h_t depends on h_{t-1}, so the 256 steps
#    cannot be parallelized. Transformers use self-attention, which computes all
#    positions in parallel — making them much faster on GPUs despite having more
#    parameters. This is the key architectural advantage that made Transformers
#    dominant: parallelizable training enables scaling to much larger models and
#    datasets. The trade-off is that attention is O(T^2) in memory.
print("See comments above for solution.")

Part 6 — Transformers and Transfer Learning#

Transformers replaced RNNs as the dominant sequence model. The key innovation is self-attention, which allows each token to directly attend to every other token:

\[\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}\right)\mathbf{V}\]

This allows the model to capture long-range dependencies in a single step (vs. the LSTM which must propagate information sequentially through all intermediate tokens).

Transfer Learning: Pre-train, then Fine-tune#

Modern NLP follows a two-stage approach:

  1. Pre-train a large Transformer on massive text corpora (billions of words) using self-supervised objectives (predict masked words, predict next sentence)

  2. Fine-tune the pre-trained model on your specific task with a small labeled dataset

This is analogous to using pre-trained Morgan fingerprints in Lab 12 — except that Transformers learn the “fingerprint extraction” from raw text.

We use DistilBERT, a lightweight version of BERT (66M params vs. 110M) that retains ~95% of BERT’s performance while being 60% faster.

Clinical parallel: ClinicalBERT, PubMedBERT, and BioGPT follow this same paradigm but are pre-trained on medical text, giving much better performance on clinical NLP tasks.

# This is an exercise cell with TODOs (??? placeholders) for student work.
# It is skipped in the book build to keep downstream cells runnable.
# Open the notebook in Colab to complete the exercise:
# Exercise cell skipped in book build
print("[exercise cell skipped — open in Colab to complete the TODOs]")

# ══ SOLUTION — scaled_dot_product_attention ══════════════════════════════════
def scaled_dot_product_attention(Q, K, V):
    d_k = Q.shape[-1]
    scores = Q @ K.T
    scores = scores / np.sqrt(d_k)

    # Softmax (numerically stable)
    exp_scores = np.exp(scores - scores.max(axis=-1, keepdims=True))
    weights = exp_scores / exp_scores.sum(axis=-1, keepdims=True)

    output = weights @ V
    return output, weights

# Test
np.random.seed(42)
seq_len, d_k, d_v = 4, 8, 8
Q = np.random.randn(seq_len, d_k).astype(np.float32)
K = np.random.randn(seq_len, d_k).astype(np.float32)
V = np.random.randn(seq_len, d_v).astype(np.float32)

output, weights = scaled_dot_product_attention(Q, K, V)
print(f"Output shape: {output.shape}")
print(f"Attention weights (row sums): {weights.sum(axis=1).round(4)}")
print(f"\nAttention weight matrix:\n{weights.round(3)}")

# ── Visualize attention weights ────────────────────────────────────────────
# Show how attention allows every token to "look at" every other token

tokens = ["patient", "denied", "chest", "pain"]

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

# Attention heatmap
im = axes[0].imshow(weights, cmap='Blues', vmin=0)
axes[0].set_xticks(range(4)); axes[0].set_yticks(range(4))
axes[0].set_xticklabels(tokens); axes[0].set_yticklabels(tokens)
axes[0].set_xlabel("Key (attended to)")
axes[0].set_ylabel("Query (attending from)")
axes[0].set_title("Self-Attention Weights", fontsize=12)
for i in range(4):
    for j in range(4):
        axes[0].text(j, i, f'{weights[i,j]:.2f}', ha='center', va='center',
                     fontsize=11, color='white' if weights[i,j] > 0.4 else 'black')
plt.colorbar(im, ax=axes[0], shrink=0.8)

# Compare: LSTM vs Transformer information flow
axes[1].set_xlim(-0.5, 4.5); axes[1].set_ylim(-1.5, 2.5)
axes[1].set_aspect('equal')
# LSTM: sequential arrows
for i in range(4):
    axes[1].text(i, 2, tokens[i], ha='center', va='center',
                bbox=dict(boxstyle='round', facecolor='#3498db', alpha=0.3), fontsize=10)
    if i < 3:
        axes[1].annotate('', xy=(i+0.6, 2), xytext=(i+0.4, 2),
                        arrowprops=dict(arrowstyle='->', color='#3498db', lw=2))
axes[1].text(2, 2.4, 'LSTM: Sequential', ha='center', fontsize=11, fontweight='bold')

# Transformer: all-to-all arrows
for i in range(4):
    axes[1].text(i, 0, tokens[i], ha='center', va='center',
                bbox=dict(boxstyle='round', facecolor='#e74c3c', alpha=0.3), fontsize=10)
for i in range(4):
    for j in range(4):
        if i != j:
            axes[1].annotate('', xy=(j, 0.25), xytext=(i, -0.25),
                            arrowprops=dict(arrowstyle='->', color='#e74c3c',
                                           alpha=0.3, lw=1))
axes[1].text(2, -1.2, 'Transformer: All-to-All', ha='center', fontsize=11, fontweight='bold')
axes[1].set_title("Information Flow Comparison", fontsize=12)
axes[1].axis('off')

plt.tight_layout()
plt.show()

# ── Fine-tune DistilBERT ──────────────────────────────────────────────────
from transformers import DistilBertTokenizer, DistilBertForSequenceClassification
from transformers import Trainer, TrainingArguments
import datasets as hf_datasets

print("Loading DistilBERT tokenizer and model...")
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model_bert = DistilBertForSequenceClassification.from_pretrained(
    'distilbert-base-uncased', num_labels=2)

print(f"Model parameters: {sum(p.numel() for p in model_bert.parameters()):,}")

# Tokenize with HuggingFace tokenizer
def tokenize_function(examples):
    return tokenizer(examples['text'], truncation=True, padding='max_length',
                     max_length=256)

# Create HuggingFace datasets
train_hf = hf_datasets.Dataset.from_dict({'text': train_texts, 'label': train_labels})
val_hf = hf_datasets.Dataset.from_dict({'text': val_texts, 'label': val_labels})
test_hf = hf_datasets.Dataset.from_dict({'text': test_texts, 'label': test_labels})

print("Tokenizing...")
train_hf = train_hf.map(tokenize_function, batched=True, batch_size=256)
val_hf = val_hf.map(tokenize_function, batched=True, batch_size=256)
test_hf = test_hf.map(tokenize_function, batched=True, batch_size=256)

# Set format for PyTorch
train_hf.set_format('torch', columns=['input_ids', 'attention_mask', 'label'])
val_hf.set_format('torch', columns=['input_ids', 'attention_mask', 'label'])
test_hf.set_format('torch', columns=['input_ids', 'attention_mask', 'label'])
print("✅ Data ready for fine-tuning")

# ── Train with HuggingFace Trainer ─────────────────────────────────────────
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    probs = torch.softmax(torch.tensor(logits), dim=-1)[:, 1].numpy()
    auc = roc_auc_score(labels, probs)
    preds = np.argmax(logits, axis=-1)
    acc = (preds == labels).mean()
    return {'auroc': auc, 'accuracy': acc}

training_args = TrainingArguments(
    output_dir='/tmp/distilbert-imdb',
    num_train_epochs=3,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=64,
    learning_rate=2e-5,
    weight_decay=0.01,
    eval_strategy='epoch',
    save_strategy='epoch',
    load_best_model_at_end=True,
    metric_for_best_model='auroc',
    logging_steps=50,
    report_to='none',
    seed=42,
)

trainer = Trainer(
    model=model_bert,
    args=training_args,
    train_dataset=train_hf,
    eval_dataset=val_hf,
    compute_metrics=compute_metrics,
)

print("Fine-tuning DistilBERT (3 epochs)...")
print("This may take 5-10 minutes on Colab with GPU, longer on CPU.\n")
trainer.train()

# Get validation results
val_results = trainer.evaluate()
print(f"\nDistilBERT Val AUROC: {val_results['eval_auroc']:.4f}")

🤔 Reflection 6.1 — Transformers and Transfer Learning#

  1. DistilBERT has ~67M parameters — 30x more than the LSTM. Yet it often performs better even on small datasets. How can a much larger model avoid overfitting on only 8K examples? What role does pre-training play?

  2. We fine-tuned for only 3 epochs with a very small learning rate (2e-5). Why is a small learning rate crucial for fine-tuning? What would happen with lr=0.01?

  3. The self-attention mechanism has \(O(T^2)\) complexity in sequence length \(T\). For a 256-token sequence, the attention matrix is 256x256 = 65K entries. For a 4,096-token clinical note, it would be 16M entries. What strategies exist to handle very long documents? (Name at least two approaches.)

  4. We used distilbert-base-uncased, which was pre-trained on English Wikipedia and BookCorpus. For clinical text, what pre-trained model would you choose instead? Why does domain-specific pre-training matter so much?

# ══ SOLUTION — Reflection 6.1 ═══════════════════════════════════════════════
# 1. Pre-training is the key. The 67M parameters were already trained on billions
#    of words, so they encode rich language understanding. Fine-tuning only ADJUSTS
#    these parameters slightly (small learning rate, few epochs). The model doesn't
#    need to learn language from scratch — it only needs to learn the task-specific
#    decision boundary. This is why transfer learning is so powerful: the effective
#    "training data" includes the pre-training corpus, not just the 8K fine-tuning
#    examples. Analogy: a medical resident with 8 years of training doesn't need
#    many examples to learn a new diagnostic criterion.
#
# 2. A small learning rate prevents "catastrophic forgetting" — destroying the
#    valuable pre-trained representations. With lr=0.01, the first gradient update
#    would massively change the pre-trained weights, essentially reinitializing
#    the model. With lr=2e-5, the model makes tiny adjustments that preserve the
#    general language understanding while adapting the classification head.
#
# 3. Strategies for long sequences: (a) chunking — split the document into 512-token
#    windows, encode each, then aggregate (mean/max/attention pool), (b) sparse
#    attention — Longformer, BigBird use local + global attention patterns to reduce
#    O(T^2) to O(T), (c) hierarchical models — encode sentences individually, then
#    encode the sequence of sentence embeddings, (d) retrieval-augmented approaches —
#    retrieve relevant chunks rather than processing the entire document.
#
# 4. For clinical text: ClinicalBERT (pre-trained on MIMIC clinical notes),
#    PubMedBERT (pre-trained on PubMed abstracts), or BioLinkBERT. Domain-specific
#    pre-training matters because: (a) medical vocabulary is poorly represented in
#    Wikipedia ("cardiomyopathy" might be split into many subwords), (b) medical
#    language has different syntax and semantics (abbreviations, negation patterns),
#    (c) the model needs to understand domain concepts (drug interactions, anatomical
#    relationships) that aren't present in general text.
print("See comments above for solution.")

Part 7 — Model Comparison and Final Test Evaluation#

We now compare all approaches on the held-out test set. As in previous labs, this is the first time we touch the test set.

# ── Collect all validation results ─────────────────────────────────────────
print("=== Validation Set Results (for model selection) ===\n")

all_val_results = {}

for name, res in bow_results.items():
    all_val_results[name] = res['val_auc']
for name, res in tfidf_results.items():
    all_val_results[name] = res['val_auc']

# LSTM
lstm_val_auc, _ = evaluate_lstm(lstm_model, val_dl)
all_val_results['BiLSTM'] = lstm_val_auc

# DistilBERT
bert_val_auc = val_results['eval_auroc']
all_val_results['DistilBERT'] = bert_val_auc

for name, auc in all_val_results.items():
    print(f"  {name:25s}  Val AUROC = {auc:.4f}")

# Bar chart
fig, ax = plt.subplots(figsize=(12, 5))
names = list(all_val_results.keys())
aucs = list(all_val_results.values())
n_bow = len(bow_results)
n_tfidf = len(tfidf_results)
colors = (['#bdc3c7'] * n_bow + ['#95a5a6'] * n_tfidf +
          ['#3498db', '#e74c3c'])
bars = ax.barh(names, aucs, color=colors, edgecolor='white')
ax.set_xlabel('Validation AUROC')
ax.set_title('Model Comparison — Validation AUROC')
ax.set_xlim(0.5, 1.0)
for bar, v in zip(bars, aucs):
    ax.text(v + 0.005, bar.get_y() + bar.get_height()/2, f'{v:.4f}',
            va='center', fontsize=10)
plt.tight_layout()
plt.show()

# ── Final test evaluation ─────────────────────────────────────────────────
print("=== FINAL TEST SET RESULTS ===")
print("(All models selected on validation set; test set never seen before)\n")

test_preds_dict = {}

# Best TF-IDF model
best_tfidf_name = max(tfidf_results, key=lambda k: tfidf_results[k]['val_auc'])
best_tfidf_model = tfidf_results[best_tfidf_name]['model']
tfidf_test_proba = best_tfidf_model.predict_proba(X_test_tfidf)[:, 1]
tfidf_test_auc = roc_auc_score(y_test, tfidf_test_proba)
test_preds_dict[f'Best TF-IDF ({best_tfidf_name.split("+")[0].strip()})'] = {
    'auc': tfidf_test_auc, 'preds': tfidf_test_proba, 'labels': y_test}

# Best BoW model
best_bow_name = max(bow_results, key=lambda k: bow_results[k]['val_auc'])
best_bow_model = bow_results[best_bow_name]['model']
bow_test_proba = best_bow_model.predict_proba(X_test_bow)[:, 1]
bow_test_auc = roc_auc_score(y_test, bow_test_proba)
test_preds_dict[f'Best BoW ({best_bow_name.split("+")[0].strip()})'] = {
    'auc': bow_test_auc, 'preds': bow_test_proba, 'labels': y_test}

# LSTM
lstm_test_auc, _ = evaluate_lstm(lstm_model, test_dl)
test_preds_dict['BiLSTM'] = {'auc': lstm_test_auc}

# DistilBERT
bert_test = trainer.predict(test_hf)
bert_test_proba = torch.softmax(torch.tensor(bert_test.predictions), dim=-1)[:, 1].numpy()
bert_test_auc = roc_auc_score(y_test, bert_test_proba)
test_preds_dict['DistilBERT'] = {
    'auc': bert_test_auc, 'preds': bert_test_proba, 'labels': y_test}

for name, d in test_preds_dict.items():
    print(f"  {name:40s}  Test AUROC = {d['auc']:.4f}")

# ROC curves (for models where we have probabilities on same labels)
fig, ax = plt.subplots(figsize=(8, 6))

fpr, tpr, _ = roc_curve(y_test, bow_test_proba)
ax.plot(fpr, tpr, label=f'BoW (AUC={bow_test_auc:.3f})', color='#bdc3c7', lw=2)

fpr, tpr, _ = roc_curve(y_test, tfidf_test_proba)
ax.plot(fpr, tpr, label=f'TF-IDF (AUC={tfidf_test_auc:.3f})', color='#95a5a6', lw=2)

# Get LSTM test predictions for ROC
lstm_model.eval()
lstm_all_preds = []
with torch.no_grad():
    for input_ids, lengths, labels in test_dl:
        input_ids, lengths = input_ids.to(device), lengths.to(device)
        out = lstm_model(input_ids, lengths)
        lstm_all_preds.append(out.cpu().numpy())
lstm_test_proba = np.concatenate(lstm_all_preds)
fpr, tpr, _ = roc_curve(y_test, lstm_test_proba)
ax.plot(fpr, tpr, label=f'BiLSTM (AUC={lstm_test_auc:.3f})', color='#3498db', lw=2)

fpr, tpr, _ = roc_curve(y_test, bert_test_proba)
ax.plot(fpr, tpr, label=f'DistilBERT (AUC={bert_test_auc:.3f})', color='#e74c3c', lw=2)

ax.plot([0, 1], [0, 1], 'k--', lw=1, alpha=0.5)
ax.set_xlabel('False Positive Rate')
ax.set_ylabel('True Positive Rate')
ax.set_title('ROC Curves — Test Set')
ax.legend(loc='lower right')
plt.tight_layout()
plt.show()

🤔 Reflection 7.1 — Comparing Approaches#

  1. Rank the four approaches (BoW, TF-IDF, LSTM, DistilBERT) by test AUROC. Does the ranking match what you expected? Where are the biggest jumps?

  2. TF-IDF + Logistic Regression is often competitive with deep learning on text classification. This parallels fingerprints vs. GNNs in Lab 12. What is the common lesson about well-designed feature engineering vs. learned features?

  3. The LSTM and DistilBERT both preserve word order, but DistilBERT typically outperforms LSTM. Why? (Hint: it’s not just the architecture — think about data.)

  4. If you had access to 100K labeled clinical notes instead of 8K movie reviews, which approach(es) might benefit most from the additional data? Which would benefit least?

# ══ SOLUTION — Reflection 7.1 ═══════════════════════════════════════════════
# 1. Expected ranking: BoW < TF-IDF < BiLSTM < DistilBERT. The biggest jump is
#    typically BoW -> TF-IDF (better feature weighting and bigrams) and TF-IDF ->
#    DistilBERT (pre-trained knowledge). The LSTM often performs between TF-IDF and
#    DistilBERT — its advantage over TF-IDF (order sensitivity) is partially offset
#    by its disadvantage (no pre-training, training from scratch on small data).
#
# 2. The common lesson is that domain-appropriate feature engineering can be
#    surprisingly competitive with end-to-end learning, especially on small datasets.
#    Morgan fingerprints encode decades of chemistry knowledge; TF-IDF with bigrams
#    captures most of the discriminative signal in text. Learned features (GNNs,
#    Transformers) primarily shine when: (a) the dataset is large, (b) transfer
#    learning is available, or (c) the task requires complex reasoning that
#    handcrafted features can't capture.
#
# 3. Two reasons DistilBERT outperforms LSTM: (a) PRE-TRAINING — DistilBERT has
#    "seen" billions of words during pre-training, giving it a huge head start.
#    The LSTM learns from only 8K labeled examples. (b) ARCHITECTURE — self-attention
#    directly captures long-range dependencies (any two tokens interact in one layer),
#    while LSTM must propagate information sequentially through all intermediate tokens,
#    losing information over long distances. Both factors contribute, but pre-training
#    is the dominant one.
#
# 4. With 100K labeled clinical notes: (a) LSTM would benefit the most — it has high
#    capacity but was severely data-limited at 8K. More data would dramatically reduce
#    overfitting. (b) DistilBERT would also improve, especially with domain-specific
#    pre-training. (c) TF-IDF baselines would improve but plateau earlier — they can't
#    learn new patterns from more data, only estimate existing feature weights better.
#    (d) BoW would benefit least — its representation is too lossy regardless of data size.
print("See comments above for solution.")

Part 8 — Extensions: What Can You Do From Here?#

If you have extra time or want to explore further, try any of these extensions. They are optional and not graded.

# ── Extension 1: Effect of training data size ─────────────────────────────
# Train TF-IDF + LR and LSTM on subsets of increasing size to see learning curves.

subset_sizes = [500, 1000, 2000, 4000, 8000]
tfidf_curve = []
lstm_curve = []

for size in subset_sizes:
    # TF-IDF + LR
    tfidf_sub = TfidfVectorizer(max_features=10000, min_df=2, max_df=0.95,
                                ngram_range=(1, 2))
    X_sub = tfidf_sub.fit_transform(train_texts[:size])
    X_val_sub = tfidf_sub.transform(val_texts)
    lr_sub = LogisticRegression(max_iter=1000, C=1.0, random_state=42)
    lr_sub.fit(X_sub, y_train[:size])
    val_proba = lr_sub.predict_proba(X_val_sub)[:, 1]
    tfidf_curve.append(roc_auc_score(y_val, val_proba))
    print(f"Size {size:5d} | TF-IDF + LR Val AUC: {tfidf_curve[-1]:.4f}")

plt.figure(figsize=(8, 4))
plt.plot(subset_sizes, tfidf_curve, 'o-', color='#95a5a6', lw=2, label='TF-IDF + LR')
plt.xlabel('Training Set Size')
plt.ylabel('Validation AUROC')
plt.title('Learning Curves: How Much Data Do You Need?')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

🧠 Final Reflection — Ordinal Sequences in the ML Landscape#

Now that you’ve worked through text data alongside the tabular (Labs 0-5) and graph (Lab 12) modalities, answer these synthesis questions:

  1. The tabularization pattern: In Lab 12 (graphs), we extracted hand-crafted features and fingerprints. Here, we used BoW and TF-IDF. Both convert structured data into flat vectors, losing structural information. When is this trade-off worthwhile? When is it not?

  2. Order vs. structure: Text has 1D sequential structure (linear chain). Graphs have arbitrary topology. Images (which we’ll see next) have 2D spatial structure. How does the dimensionality and regularity of the structure affect which models work best?

  3. Pre-training as the great equalizer: In Lab 12, molecular fingerprints (hand-crafted) were competitive with GNNs (learned). Here, DistilBERT (pre-trained) typically outperforms TF-IDF (hand-crafted). What’s different? Why does pre-training help more for text than for molecules?

  4. Clinical NLP challenges: Real clinical text has challenges we didn’t encounter with movie reviews: de-identified placeholders, abbreviations, copy-paste artifacts, multilingual content, and heavily imbalanced labels. Which of these challenges would most affect each of our four approaches (BoW, TF-IDF, LSTM, Transformer)?

  5. Looking ahead: The latest clinical NLP systems use large language models (LLMs) that go far beyond BERT-style classification. How might instruction-tuned LLMs (like GPT-4 or Claude) change the paradigm of clinical text classification from “train a classifier” to “write a prompt”? What new challenges does this introduce?

# ══ SOLUTION — Final Reflection ══════════════════════════════════════════════
# 1. Tabularization is worthwhile when: (a) the dataset is small and complex models
#    overfit, (b) interpretability is required (TF-IDF weights are inspectable),
#    (c) speed is critical (TF-IDF + LR is orders of magnitude faster to train and
#    deploy than Transformers), (d) the task is well-captured by the features (keyword-
#    driven classification). It's NOT worthwhile when: (a) word order or structure is
#    critical (negation detection, relationship extraction), (b) large pre-trained
#    models are available, (c) the dataset is large enough to train deep models.
#
# 2. Regular, low-dimensional structure (1D text, 2D images) enables weight sharing:
#    the same convolution kernel or attention pattern works everywhere. Irregular
#    structure (graphs) requires more flexible architectures (message passing) that
#    can't exploit regularity. This is why CNNs work better on images than on graphs,
#    and why Transformers (with position encodings) work better on text than on graphs.
#    More regular structure → more parameter sharing → better sample efficiency.
#
# 3. Pre-training helps more for text because: (a) language has a rich, universal
#    structure that transfers across tasks — knowing grammar and word meanings helps
#    with ANY text task. (b) Massive text corpora are freely available for pre-training.
#    For molecules: (a) chemical "grammar" is simpler and mostly captured by fingerprints,
#    (b) pre-training molecular models is harder because 3D structure matters and
#    molecular datasets are smaller. However, recent molecular foundation models
#    (like Uni-Mol) are beginning to close this gap.
#
# 4. De-identified placeholders: affect BoW/TF-IDF most (waste vocabulary slots) and
#    Transformers least (learned embeddings adapt). Abbreviations: hurt all models but
#    are most problematic for pre-trained Transformers not exposed to clinical text.
#    Copy-paste: inflate TF-IDF scores artificially; LSTMs may learn spurious patterns
#    from repeated text. Imbalanced labels: affect all models but require careful
#    handling in neural models (weighted loss, focal loss, oversampling).
#
# 5. LLMs enable zero-shot and few-shot classification via prompting: instead of
#    training a classifier, you write a prompt like "Classify this clinical note as
#    [condition present / condition absent]: {note}". This eliminates the need for
#    labeled training data (a huge advantage in clinical settings where labeling is
#    expensive). New challenges: (a) hallucination — LLMs may fabricate clinical
#    findings, (b) cost — each inference call is expensive, (c) privacy — sending
#    PHI to external APIs raises HIPAA concerns, (d) calibration — LLM confidence
#    may not match accuracy, (e) reproducibility — prompt sensitivity and model
#    updates make results hard to reproduce.
print("See comments above for solution.")