Lecture 4 — Conditioning, Bayes’ rule, and information theory

Lecture 4 — Conditioning, Bayes’ rule, and information theory#

⚠️ AI-synthesized; not fully reviewed by course staff. Treat as a study aid; released slides, notebooks, and lecture recordings are authoritative.

Thu Jan 29, 2026 · Part 1 — Foundations · §1.1 Mathematical preliminaries

What this lecture is about#

This lecture is about three closely-linked moves: how to get from a joint distribution to marginal and conditional distributions; what Bayes’ rule is really doing (it’s just a rearrangement, not magic); and how information theory gives us a way to measure uncertainty, surprise, and divergence between distributions.

We move from the law of large numbers and expectation to:

  • Marginalization. Sum (or integrate) a joint distribution over the variable you don’t want.

  • Conditioning. Take a slice of the joint and renormalize. Conditioning is not filtering; it changes the distribution itself.

  • Independence and dependence. Read off from whether the joint factorizes.

  • Bayes’ rule. A statement about a joint distribution rewritten two ways. Prior × likelihood ÷ evidence = posterior.

  • Information theory. Entropy, KL divergence, mutual information — the tools that turn “uncertainty” into something you can write a loss against.

The paradox flagged is St. Petersburg, which is a stress-test for whether expectation is the right summary of a distribution at all.

Why it matters#

Two reasons this material is more than mathematical hygiene:

Bayes’ rule is the engine of every diagnostic conversation in clinical medicine. When prevalence is 0.1% and a test has 95% sensitivity and 95% specificity, the positive predictive value is around 1.9%. That’s not a curiosity — it is the entire reason “alert fatigue” is a measurable clinical phenomenon (which we’ll return to in L8 and L10). If your audience for an ML system is clinicians, you need to feel Bayes-rule arithmetic in your bones, because the system’s outputs will be interpreted through it.

Information theory underwrites almost every loss function the course will use. Cross-entropy decomposes as H(p, q) = H(p) + KL(p‖q): it equals the entropy of the data distribution plus the KL divergence between data and model. Because H(p) does not depend on the model, minimizing cross-entropy is equivalent to minimizing KL — but they are not identical objects. Negative log-likelihood averaged over a sample is a sample-based estimator of cross-entropy (it equals entropy only in the degenerate case where the model matches the truth exactly). The “perplexity” you’ll see in L14 (LLMs) is exp(cross-entropy). Once you see this, the loss-function landscape of ML stops feeling like a zoo and starts feeling like one idea (KL) appearing in many costumes.

Things you should walk away believing#

  • Conditional probability is renormalization, not filtering. ℙ(A|B) tells you about a different distribution than ℙ(A).

  • Bayes’ rule is just ℙ(A|B) ℙ(B) = ℙ(A,B) = ℙ(B|A) ℙ(A) rewritten. The “magic” is the rearrangement plus the willingness to call one of the conditionals “prior.”

  • Independence ⇔ joint factorizes. This is the same fact dressed up in different vocabulary across the rest of the course.

  • Entropy H(X) measures expected surprise; KL(p‖q) measures how surprising q is when p is the truth; they are not symmetric and that asymmetry matters in ML losses.

  • A “low PPV at low prevalence” is not a failure of the test — it is Bayes’ rule working correctly.

How this connects to the rest of the course#

  • L6’s “expected loss over the data-generating distribution” is built on conditioning.

  • L7 (probabilistic modeling) frames supervised learning as maximum likelihood — i.e., minimizing a KL-shaped object.

  • L8 (binary-classification evaluation) reuses Bayes’ rule to turn a confusion matrix into clinical claims.

  • L14 (LLMs) is literally entropy minimization over next tokens.

  • L17-L18 (EHR data) and L24 (causal inference) lean on conditioning constantly: “given the patient was observed, what does the distribution of their lab values look like?”

  • L28 (recap) returns to PPV-vs.-prevalence as a deployment lens.

Study guide#

Key terms, self-check questions, and additional resources for active recall.

Key Terms#

Term

Definition

Joint Distribution

P(X, Y): the probability distribution over pairs of outcomes. Captures all dependence structure between X and Y.

Marginal Distribution

P(X) = ∫ P(X,Y) dY (cont.) or Σᵧ P(X,Y=y) (disc.). Obtained by integrating/summing out the other variable.

Conditional Distribution

P(Y | X) = P(X,Y) / P(X). The renormalized joint for a fixed value of X.

Statistical Independence

X ⊥ Y iff P(X,Y) = P(X) P(Y); equivalently P(Y | X) = P(Y).

Bayes’ Rule

P(Y | X) = P(X | Y) P(Y) / P(X). Posterior = Prior × Likelihood / Evidence.

Prior P(Y)

The marginal distribution of Y before observing X. Not a subjective “belief” — it is the marginal of the joint.

Likelihood P(X | Y)

Probability of observing X given Y is true. With the prior, determines the posterior.

Information Content

I(x) = −log P(x). Rare events carry more information. Units: bits (log₂) or nats (ln).

Entropy

H(X) = −Σ P(x) log P(x) = 𝔼[−log P(X)]. Average information content; measures uncertainty.

KL Divergence

D_KL(P‖Q) = Σ P(x) log (P(x)/Q(x)). Non-symmetric measure of how P differs from Q; always ≥ 0.

Mutual Information

I(X;Y) = D_KL(P(X,Y) ‖ P(X)P(Y)). How much knowing Y reduces uncertainty about X.

Key Insight: The Prior Is Not a “Belief”#

Caution on Bayesian interpretation

A common misconception is that the prior P(Y) reflects a subjective belief or guess. Formally, it is the marginal distribution of Y in the joint P(X,Y). Bayesian vs. frequentist debates are philosophically interesting but should not distract from the mathematics: Bayes’ rule is a theorem, not a philosophy.

Clinical Application: Bayes in Diagnosis#

Why base rates matter

Bayes’ rule reveals why a highly-accurate test can still yield mostly false positives when the disease is rare. If prevalence is 1%, a test with 95% sensitivity and 90% specificity has positive predictive value of only ≈ 8.7%. Model accuracy metrics must always be interpreted in the context of class prevalence.

Self-Check Questions#

  1. Derive the marginal P(X) from the joint P(X,Y) for a 2×2 discrete example.

  2. Derive Bayes’ rule directly from the definition of conditional probability.

  3. Compute the entropy of a fair coin. What distribution over a 2-outcome space maximizes entropy?

  4. Why is KL divergence not a metric? What metric property does it fail?

  5. A diagnostic test has sensitivity 0.95 and specificity 0.90; disease prevalence is 1%. Using Bayes’ rule, what is P(disease | positive test)?

Additional Resources#