Lecture 24 — Causality and fairness

Lecture 24 — Causality and fairness#

⚠️ AI-synthesized; not fully reviewed by course staff. Treat as a study aid; released slides, notebooks, and lecture recordings are authoritative.

Thu Apr 16, 2026 · Part 4 — Population, causality, fairness

What this lecture is about#

This is the lecture that finally formalizes the warning that has been threading through the whole course since L1: prediction and intervention are different questions, and conflating them is one of the most consequential mistakes in clinical AI. The lecture has two interlocking halves — causality and fairness — that turn out to be the same problem from two angles.

Half 1: Causality.

  • Causal DAGs. A directed acyclic graph encoding qualitative causal assumptions about a data-generating process. Nodes are variables; edges are direct causal effects.

  • The do-operator. Pearl’s notation for an intervention — what would happen if we forced X = x rather than observed X = x. p(Y | X = x) and p(Y | do(X = x)) are different distributions in general; in observational data we only get the former.

  • Three DAG patterns.

    • Chain (X → Z → Y). Z mediates X’s effect on Y. Conditioning on Z blocks the path.

    • Fork (X ← Z → Y). Z is a common cause; without conditioning on Z, X and Y appear correlated even with no direct causal link.

    • Collider (X → Z ← Y). Z is a common effect. Conditioning on Z creates spurious correlation between X and Y. (Berkson’s paradox from L17 is a collider-conditioning story.)

  • Bias gallery. Selection bias, confounding bias, measurement error, reverse causation, ascertainment bias, immortal-time bias. Each has a DAG signature.

  • Propensity scores. The probability of receiving treatment given covariates: e(x) = p(T = 1 | X = x). Used for matching, weighting (IPTW), and stratification. Lets you compare treatment groups in observational data as if they had been randomized — provided you’ve measured all the right confounders.

  • Causal inference challenges. No-unmeasured-confounders is an assumption, not a fact. Sensitivity analysis (E-values, etc.) tells you how robust your conclusion is to a hidden confounder.

Half 2: Fairness.

  • Fairness definitions. Statistical parity, equalized odds, predictive parity, calibration-within-group, individual fairness. They sound similar; they aren’t equivalent.

  • Sufficiency vs. separation. Two compact families of fairness criteria.

    • Sufficiency (calibration within groups): given the score, the outcome distribution is the same across groups.

    • Separation (equalized odds / FPR / FNR balance): given the outcome, the score distribution is the same across groups.

  • The Chouldechova / Kleinberg impossibility. If base rates differ across groups (which they almost always do in health), you cannot simultaneously satisfy sufficiency and separation, except in trivial cases. A “fair” model is forced to choose which kind of fairness it satisfies.

  • The Obermeyer case study. A widely-used commercial risk-stratification algorithm under-served Black patients because it predicted healthcare cost as a proxy for health need — and Black patients had systematically lower costs given the same need due to access disparities. The bias wasn’t in the algorithm; it was in the label. Read the paper. Twice.

  • Causal fairness. Use the DAG to ask: where in the data-generating process is the unfairness coming from? This is what unifies the two halves of the lecture — fairness analysis without causal reasoning is incomplete.

  • Subgroup evaluation. Compute every metric (AUROC, calibration, PPV, FNR) per subgroup. Differences are diagnostic.

  • Mitigation. Three loci: data (fix labels, fix sampling), model (constrain training to satisfy a fairness criterion), threshold (group-specific thresholds). Each has trade-offs.

Why it matters#

Three reasons this lecture sits where it does in the course:

Most “AI fairness” papers treat fairness as a pure technical problem — pick metric, optimize against it, done. That misses the entire mechanism story. The Obermeyer algorithm satisfied calibration; it failed because the label (cost) was a biased proxy for what we cared about (need). No technical fairness criterion would have flagged it. You need to look at the DAG.

Causality and fairness are the same problem. Both ask: given observed data, what’s actually going on in the world? Both require explicit assumptions you can’t get from the data alone. Both reward DAG-thinking and punish “just optimize the metric” thinking.

The impossibility result is real and matters for deployment. A system can be calibrated within groups or equalize FPR/FNR across groups, but not both. Which one is the right choice is a value judgment, not a technical one. ML practitioners need to be ready to have that conversation with stakeholders.

Things you should walk away believing#

  • Prediction and intervention are different questions. p(Y|X) ≠ p(Y|do(X)) in general.

  • DAGs are explicit assumptions. They are wrong sometimes; their explicitness lets you debug.

  • Conditioning on a collider creates spurious correlation. This is the most common DAG mistake.

  • Propensity scores are useful, not magic. They depend on having measured all confounders.

  • Sufficiency and separation are different fairness families. Neither is universally “right.”

  • The Chouldechova / Kleinberg impossibility is real; choose which fairness you want.

  • Label bias (Obermeyer) is invisible to fairness criteria that only look at scores. Always ask: what does the label measure?

  • Mitigation has trade-offs. Group-specific thresholds can be the right answer; they can also be illegal.

  • Subgroup evaluation is non-optional. Every metric, per subgroup, every time.

How this connects to the rest of the course#

  • L9’s flu-vaccine motivation finally gets formalized here.

  • L17-L18 (EHR data) and L23 (population health) provide the substrate where these issues bite.

  • L8’s calibration is the “sufficiency” axis; L8’s separation-into-FPR/FNR is the “separation” axis. The fairness impossibility applies to L8’s metrics.

  • L25 (PRS transferability) is a fairness-across-ancestry story.

  • L15 (foundation models) inherits all of these issues at scale.

Study guide#

Key terms, self-check questions, and additional resources for active recall.

Key Terms#

Term

Definition

Causal DAG

Directed Acyclic Graph encoding qualitative causal assumptions. Nodes are variables, edges are direct causal effects. Wrong sometimes; their explicitness lets you debug.

do-operator

Pearl’s notation for an intervention: p(Y | do(X = x)) ≠ p(Y | X = x) in general. Observational data give the latter; we usually want the former.

Chain (X → Z → Y)

Z mediates X’s effect on Y. Conditioning on Z blocks the path.

Fork (X ← Z → Y)

Z is a common cause; without conditioning on Z, X and Y appear correlated even with no direct link.

Collider (X → Z ← Y)

Z is a common effect. Conditioning on Z creates spurious correlation between X and Y. (Berkson’s paradox.)

Confounder

A common cause of treatment and outcome. Failure to adjust for confounders biases observed associations.

Selection bias

Bias from non-random sampling (who’s in the data depends on the variables you’re studying).

Propensity score

e(x) = P(T = 1 | X = x). Probability of receiving treatment given covariates. Used for matching, weighting (IPTW), or stratification.

No unmeasured confounders

The assumption that all confounders are measured. An assumption, not a fact. Sensitivity analysis (E-values) tells you how robust your conclusion is to a hidden confounder.

Statistical parity

P(Ŷ = 1 | A = a) equal across groups. (Demographic parity.)

Equalized odds (separation)

P(Ŷ = 1 | Y = y, A = a) = P(Ŷ = 1 | Y = y) for all y, a.

Predictive parity (sufficiency)

P(Y = 1 | Ŷ = p, A = a) = P(Y = 1 | Ŷ = p) for all p, a. (Calibration within group.)

Chouldechova / Kleinberg impossibility

Sufficiency and separation cannot both hold for non-degenerate classifiers when base rates differ across groups.

Obermeyer et al. (2019)

Canonical case study: a “fair-looking” risk algorithm under-served Black patients because the label (cost) was a biased proxy for the target (need).

Causal fairness

Use a DAG to ask where in the data-generating process the unfairness comes from. Unifies the technical fairness criteria with mechanism analysis.

Prediction ≠ Intervention#

The single most consequential confusion

p(Y | X) ≠ p(Y | do(X)) in general. Observational ML answers the prediction question. It does not, by itself, license intervention claims. The whole point of causal DAGs and propensity scores is to bridge this gap with explicit assumptions.

The Fairness Impossibility#

Sufficiency vs. separation are mathematically incompatible

When base rates differ across groups (which they almost always do in health) and prediction is imperfect, you cannot simultaneously satisfy sufficiency (calibration within groups) and separation (equalized FPR/FNR). One must be prioritized. Which one to prioritize is a value judgment, not a technical one.

Label Bias Is Invisible to Score-Only Fairness#

Obermeyer in one sentence

A widely-used commercial risk-stratification algorithm under-served Black patients because it predicted healthcare cost as a proxy for health need — and Black patients had systematically lower costs given the same need due to access disparities. The bias was in the label, not the algorithm. Always ask: what does the label measure?

Self-Check Questions#

  1. Walk through the three DAG patterns (chain, fork, collider). For each, when does conditioning on the middle variable help, and when does it hurt?

  2. A propensity score lets you compare treatment groups in observational data as if they had been randomized. What assumption does that “as if” require, and what tools (e.g., E-values) check robustness to it?

  3. Two physicians want a fair sepsis predictor. Physician A wants equal calibration across groups (sufficiency); Physician B wants equal FNR across groups (separation). Why can’t you give them both?

  4. Walk through the Obermeyer mechanism: what was the label, what was the target, why did the proxy gap produce racial harm, and how did the fix work?

  5. Statistical parity, equalized odds, predictive parity: which of these does the Obermeyer algorithm satisfy? Which does it violate?

Additional Resources#