26. Experiments, Observation, and Causality

Experiments, Observation, and Causality¶

Overview¶

Variables in neuroimaging studies come in two flavors: those we manipulate and those we observe. In task fMRI, the stimuli and instructions we present — faces vs. houses, high vs. low reward, painful vs. non-painful heat — are independent variables (IVs) under experimental control, usually manipulated within-person so that each participant experiences every level. Brain activity at each voxel, behavior, and physiology are dependent variables (DVs) that we observe but do not control. This distinction maps onto two goals that every study shares. Internal validity is the ability to make correct causal inferences — to be confident that the activation we see is caused by the manipulated task attribute and not something else. External validity is the ability of results to generalize to new contexts, measures, and populations. Frustratingly, the two trade off: tightly controlled laboratory experiments maximize internal validity but test a narrow slice of behavior in a selected sample, while large observational studies (resting-state correlations with clinical traits, naturalistic movie-watching) sample richer contexts and populations but mix together multiple causal influences. The strongest research programs combine both: experiments to establish that an effect is causal, and observational or large-scale studies to establish its size and reach in the real world.

The single most important tool for internal validity is randomization. Without it, associations are open to confounding — a spurious component of the X–Y association created by a third variable that influences both. The examples are close to home. If most trials of condition A appear early in the run and condition B late, the A − B contrast is confounded with arousal, fatigue, and practice. If the word “Ouch!” appears on the screen only during painful stimulation, visual and semantic processing are confounded with pain. And if participants choose which stimulus they receive on each trial, the trial type becomes entangled with novelty, decision processes, and the fluctuating internal states that drove the choice — the observed activation could be caused by any of them. Randomizing the order and timing of conditions breaks all of these links at once, including links to confounders nobody thought of.

The same logic underlies randomized controlled trials (RCTs) in medicine, and their history holds a cautionary tale. Large observational studies in the 1980s found that women taking hormone replacement therapy (HRT) had roughly three-fold lower risk of coronary heart disease, and HRT was widely recommended for prevention; some 15 million American women were taking it by 2001. When a large RCT (the Women’s Health Initiative) finally reported in 2002, women randomized to HRT showed a 29% increase in heart disease. The observational association had the wrong sign because of healthy-user bias: the kind of person who elected to take HRT also exercised, adhered to medications, smoked less, and was healthier in ways that lowered heart disease risk on their own. The causal structure is easiest to see as a diagram:

In the observational world, the raw HRT–disease association blends the true causal path $b$ (harmful) with a spurious path through the latent confounder — healthy people both take HRT (path $a$) and avoid heart disease (path $c$) — and the spurious component dominates, flipping the sign. Randomization severs every arrow into the treatment, leaving only $b$. Measured proxies for the confounder (education, medication adherence) can be entered as covariates, and they help exactly to the extent that they capture the latent “healthy user” construct — which they rarely do completely. (This diagram is redrawn after Figure 26.1 from the book.)

The diagram language generalizes, and it repays learning because covariate adjustment is not always benign. Consider the simple linear system $X = aC + u$ and $Y = bX + cC + e$, where $a$ is the effect of the confounder $C$ on the exposure $X$, $b$ the true causal effect of $X$ on the outcome $Y$, $c$ the effect of $C$ on $Y$, and $u$ and $e$ are independent noise terms. Regressing $Y$ on $X$ alone yields a biased estimate of the causal effect:

Adjusting for the confounder $C$ removes the second term and recovers $b$. But the same operation applied to a different causal structure backfires. A collider is a variable caused by both X and Y (for example, a composite “data quality” or inclusion score). Conditioning on a collider — whether by entering it as a covariate or by selecting only observations above some threshold — creates a spurious X–Y association where none existed. This is why post-hoc selection of trials or participants based on performance, missing data, or head motion deserves real scrutiny. A mediator lies on the causal path from X to Y; adjusting for it removes the very effect you may want to measure, converting a total effect into a direct effect. Same regression, three different structures, three different verdicts:

Clinical trialists have catalogued the ways studies go wrong, and every one of them applies to fMRI. Selection bias: the study group differs systematically from the intended population (the healthy-user bias is one form), limiting generalization and — when selection differs across treatment arms — internal validity too. Attrition bias: dropout that differs across groups and relates to the outcome, as when control participants who feel no benefit leave the study. Performance bias: groups differ in exposures other than the treatment, such as extra follow-up care for the drug arm. Detection bias: outcomes are measured differently across groups. Reporting bias: what gets analyzed, published, and cited depends on the results (“file-drawer” effects, outcome switching, spin). Well-designed RCTs counter these with strict randomization, blinding of participants and assessors, identical procedures across arms, intent-to-treat analysis, checks that randomization actually balanced the groups (chance failures are common when group sizes fall below ~50), pre-registration of a primary outcome, and public registries. Neuroimaging is increasingly adopting the same practices — and one striking demonstration of why: in a meta-analysis of antidepressant trials, drug–placebo differences shrank toward zero when side-effect-driven unblinding was controlled, showing how expectation alone can manufacture a “treatment effect.”

For fMRI specifically, three strategies eliminate or manage confounds. Matching or stratification: design the study so IVs cannot be explained by known alternatives — stratify trial types across time blocks so fatigue and habituation are orthogonal to task contrasts, and match groups on age, sex, and baseline scores. Independent manipulation of the suspected confound: if pain-evoked activation might really reflect general negative emotion, manipulate negative emotion without pain and show the region is insensitive to it. Statistical control: enter what you cannot manipulate — head motion regressors, spike artifacts, physiological noise — as covariates, with the DAG lessons above in mind. Finally, keep the big causal picture in view. In a typical task fMRI study, the task is randomized but the brain is only observed: the design supports the claim that the task caused the activation, but the path from brain activity to behavior remains correlational, because neither brain activity nor performance was assigned at random. This brain-as-mediator framing — randomized X, observed brain M, observed outcome Y — is the natural bridge from experimental design to mediation analysis, and ‘triangulating’ effects across methods (invasive recordings, EEG/MEG, brain stimulation) is what ultimately turns correlated pathways into causal claims.

Hands-on tutorial¶

Causal structure is one of the few things in statistics you can see directly in simulation, because you know the ground truth. Below we build the two central demonstrations in a few lines each; the full labs extend them with replications, selection effects, and a mediation preview.

Step 1 — A confounder: adjusting fixes the bias. We recreate the HRT story in miniature: $C$ (think “general health”) drives both the exposure $X$ and the outcome $Y$. The true causal effect of $X$ on $Y$ is +0.4, but the naive regression gets the sign wrong — and adjusting for $C$ recovers the truth.

% Adapted from CANlab FMRI_simulations (covariates_in_regression.m)
rng(26);                            % seed, for reproducible random numbers
n = 1000;                           % n = number of observations ("participants")
C = randn(n, 1);                    % confounder ("general health")
X = 0.7*C + randn(n, 1);            % exposure, influenced by C
Y = 0.4*X - 1.2*C + randn(n, 1);    % true causal effect of X = +0.40

naive    = fitlm(X, Y);             % omits the confounder
adjusted = fitlm([X C], Y);         % adjusts for the confounder

fprintf('True effect: 0.40 | naive: %.2f | adjusted: %.2f\n', ...
    naive.Coefficients.Estimate(2), adjusted.Coefficients.Estimate(2));

import numpy as np, statsmodels.api as sm

rng = np.random.default_rng(26)                 # seed, for reproducible random numbers
n = 1000                                        # n = number of observations ("participants")
C = rng.standard_normal(n)                      # confounder ("general health")
X = 0.7 * C + rng.standard_normal(n)            # exposure, influenced by C
Y = 0.4 * X - 1.2 * C + rng.standard_normal(n)  # true causal effect of X = +0.40

naive    = sm.OLS(Y, sm.add_constant(X)).fit()                        # omits C
adjusted = sm.OLS(Y, sm.add_constant(np.column_stack([X, C]))).fit()  # adjusts

print(f"True effect: 0.40 | naive: {naive.params[1]:.2f} | "
      f"adjusted: {adjusted.params[1]:.2f}")

Example output:

True effect: 0.40 | naive: -0.12 | adjusted: 0.41

The naive regression does not just shrink the effect — it flips its sign. One extra covariate restores the truth.

Step 2 — A collider: adjusting creates bias. Now $X$ and $Y$ are truly unrelated, but both feed into a downstream composite $S$ (think of an inclusion score built from performance and data quality). Left alone, the regression correctly finds nothing; “controlling for” $S$ manufactures a strong spurious effect.

X = randn(n, 1);                    % exposure, unrelated to Y by construction
Y = randn(n, 1);                    % true causal effect of X = 0
S = X + Y + randn(n, 1);            % collider: caused by BOTH X and Y

naive    = fitlm(X, Y);             % correct: no effect
adjusted = fitlm([X S], Y);         % adjusting for the collider = biased!

fprintf('True effect: 0.00 | naive: %.2f | adjusted: %.2f\n', ...
    naive.Coefficients.Estimate(2), adjusted.Coefficients.Estimate(2));

X = rng.standard_normal(n)              # exposure, unrelated to Y by construction
Y = rng.standard_normal(n)              # true causal effect of X = 0
S = X + Y + rng.standard_normal(n)      # collider: caused by BOTH X and Y

naive    = sm.OLS(Y, sm.add_constant(X)).fit()                        # correct
adjusted = sm.OLS(Y, sm.add_constant(np.column_stack([X, S]))).fit()  # biased!

print(f"True effect: 0.00 | naive: {naive.params[1]:.2f} | "
      f"adjusted: {adjusted.params[1]:.2f}")

Example output:

True effect: 0.00 | naive: -0.01 | adjusted: -0.51

The naive estimate in Step 2 hovers near zero while the “adjusted” one lands around −0.5 with an impressive t-statistic — a completely spurious finding produced by the adjustment itself. The full labs go on to (1) contrast an observational study with an RCT on the same simulated population, (2) repeat both demonstrations across hundreds of replications, (3) show that selecting observations on a collider (as in post-hoc trial exclusion) biases results just like covarying for one, and (4) preview mediation analysis on a simulated task → brain → behavior chain.

Thought questions¶

1. Naturalistic movie-watching designs expose participants to rich, engaging stimuli, but close-ups of faces tend to occur at emotionally charged moments in the plot. Analyze this design in terms of internal and external validity, and propose a hybrid design that adds an experimental manipulation to recover some internal validity without destroying the naturalism.

2. In a pain study, participants move their heads more during high-intensity stimulation. Consider three causal readings of head motion — confounder, mediator (motion-induced signal change caused by the painful event), and part of a collider (trials excluded when motion exceeds a threshold). What does including motion regressors, or excluding high-motion trials, do to your pain contrast under each reading?

3. A study compares brain connectivity in medicated depressed patients vs. healthy controls and finds group differences. Drawing on the HRT example, list at least three variables that could play the role of “general health” here, say which ones can be measured as proxies, and explain what covariate adjustment can and cannot fix when the true confounder is latent.

4. A colleague argues, “Our task was fully randomized, so our finding that amygdala activity drives self-reported fear is causally established.” Which link in the task → brain → behavior chain does randomization actually secure, which does it not, and what converging evidence would strengthen the brain → behavior claim?

5. Pre-registration and registries were developed to combat reporting bias in clinical trials. Which specific analytic decisions in a typical fMRI study (first-level modeling, trial exclusion, ROI selection, contrast choice) create analogous opportunities for bias, and which would you commit to in advance?

Quiz yourself¶

⌂ Back to home · Table of contents · How to use this site