Python Banner

8.4 Instrumental Variables (IV)

Instrumental Variables (IV) is a quasi-experimental method for estimating causal effects when treatment is confounded by unobservables, but a variable (the instrument) shifts treatment exposure without otherwise affecting the outcome.

Overview

The Core Problem: Unobserved Confounding

In many business settings, treatment is self-selected and confounded by factors we cannot measure. For example:

  • High-ability employees are more likely to participate in training programs and have better performance
  • Engaged customers are more likely to adopt new features and spend more
  • Well-managed stores are more likely to try new initiatives and have higher revenue

Even if we control for many observable covariates, unobserved factors (ability, engagement, management quality) can bias our estimates. IV provides a solution when we can find an appropriate instrument.

The Core Idea

An instrument (Z) is a variable that:

  1. Affects treatment (D) — it changes the probability or intensity of treatment
  2. Does not directly affect the outcome (Y) — it only affects Y through D (exclusion restriction)
  3. Is as good as random — it’s not correlated with unobserved confounders (independence)

When these conditions hold, IV isolates the variation in treatment that is “as good as random” and uses it to estimate the causal effect.

Business Setting

IV is ideal when:

  • Treatment is self-selected and confounded by unobservables
  • An instrument exists that shifts treatment without directly affecting the outcome
  • The instrument is credible — stakeholders agree it only affects the outcome through treatment

Common business instruments:

  • Encouragement designs: Randomly sent emails that nudge feature adoption (randomized encouragement, self-selected adoption)
  • Supply shocks: Regional shipping delays as an instrument for delivery speed’s effect on repeat purchase
  • Distance/availability: Distance to the nearest store as an instrument for in-store program exposure
  • Quasi-random agent assignment: Customers routed to support reps with different propensities to offer discounts
  • Policy changes: Regulatory changes that affect some firms but not others

The Estimand: LATE

IV identifies the Local Average Treatment Effect (LATE) — the effect among compliers, defined as units whose treatment status is moved by the instrument.

This is different from ATE (average effect for everyone) or ATT (effect for the treated). LATE is specific to the subgroup that responds to the instrument.

Important: Be explicit with stakeholders about this limitation. “This is the effect for customers whose adoption the email tipped,” not “the effect for all customers.”

Implementation in Python

InstrumentalVariable is two-stage least squares. The first stage regresses treatment on the instrument. The second stage regresses the outcome on the fitted treatment. The interval is a Bayesian bootstrap of both stages.

The setup matches the other PyCausalML tutorials: find the repository, import the package, and install the editable checkout only if the import fails.

Set Up

Code 
import subprocess
import sys
from pathlib import Path

def _repo_root() -> Path:
    here = Path.cwd().resolve()
    for cand in [here, *here.parents]:
        if (cand / "pyproject.toml").is_file() and (cand / "src" / "pycausalml").is_dir():
            return cand
    raise FileNotFoundError(
        "PyCausalML repository was not found. Run this notebook from the repository."
    )

root = _repo_root()

try:
    import pycausalml  # noqa: F401
    print("PyCausalML ready.")
except ImportError:
    subprocess.check_call(
        [sys.executable, "-m", "pip", "install", "-q", "-e", f"{root}[plot]"],
    )
    print("PyCausalML installed.")
print("Repository:", root)
PyCausalML ready.
Repository: /home/zia207/Github/PyCausalML
Code 
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

from pycausalml import (
    InstrumentalVariable,

)

rng = np.random.default_rng(42)
plt.rcParams["figure.dpi"] = 110
pd.set_option("display.float_format", lambda x: f"{x:,.3f}")
print("Imported PyCausalML")
Imported PyCausalML

Simulated Example: Email Encouragement for Feature Adoption

We’ll simulate an encouragement design: randomly send emails encouraging customers to adopt a new feature. The email is randomized (valid instrument), but adoption is self-selected (confounded by unobserved ability).

Code 
n = 10000

# Unobserved confounder: customer ability
ability = rng.normal(0, 1, n)

# Instrument: randomized encouragement email
Z = rng.binomial(1, 0.5, n)

# Treatment: feature adoption depends on ability AND the email
# Higher ability customers are more likely to adopt
# The email increases adoption probability
logit_adopt = -0.8 + 1.2*ability + 1.1*Z
p_adopt = 1 / (1 + np.exp(-logit_adopt))
D = rng.binomial(1, p_adopt)

# True LATE: effect for compliers (those moved by the email)
TRUE_LATE = 15.0

# Outcome: spending depends on ability AND treatment
# Higher ability customers spend more
# Treatment adds to spending
y = 50 + TRUE_LATE*D + 12*ability + rng.normal(0, 10, n)

iv = pd.DataFrame(dict(Z=Z, D=D, y=y))

print(iv.head())
print(f"\nTrue LATE: ${TRUE_LATE}")
   Z  D      y
0  1  1 78.698
1  0  0 13.287
2  1  1 56.424
3  0  0 74.214
4  0  0 19.392

True LATE: $15.0

Naive OLS is Biased

Let’s see what happens if we naively compare treated and untreated customers using OLS:

Code 

design = np.column_stack([np.ones(len(iv)), iv["D"].to_numpy(float)])
ols_effect = float(np.linalg.lstsq(design, iv["y"].to_numpy(float), rcond=None)[0][1])
print(f"Naive OLS effect: ${ols_effect:.2f}")
print(f"True LATE: ${TRUE_LATE:.2f}")
print("OLS is larger because higher-ability customers adopt more and spend more.")
Naive OLS effect: $25.57
True LATE: $15.00
OLS is larger because higher-ability customers adopt more and spend more.

The OLS estimate is much higher than the true LATE because it confounds the treatment effect with the effect of unobserved ability.

Two-Stage Least Squares (2SLS)

The standard IV estimator is Two-Stage Least Squares (2SLS):

First stage: Regress treatment D on instrument Z (and any controls) to get predicted treatment \(\hat{D}\)

Second stage: Regress outcome Y on predicted treatment \(\hat{D}\) (and any controls)

The coefficient on \(\hat{D}\) in the second stage is the IV estimate.

We’ll use InstrumentalVariable from PyCausalML.

Code 

iv_fit = InstrumentalVariable(n_boot=200, random_state=0)
iv_fit.fit(iv["y"].to_numpy(), iv["D"].to_numpy(), iv["Z"].to_numpy())
print("IV (2SLS):")
print(pd.Series(iv_fit.summary()).round(3))
print(f"True LATE: ${TRUE_LATE:.2f}")
IV (2SLS):
estimate         14.297
ci_lower         11.522
ci_upper         16.754
posterior_mean   14.364
dtype: float64
True LATE: $15.00

The IV estimate should be close to the true LATE of $15. The 2SLS method successfully removes the bias from unobserved ability by using only the variation in treatment induced by the randomized instrument.

First-Stage Diagnostics

A valid instrument must be relevant — it must strongly affect treatment. We check this with the first-stage F-statistic.

Code 

z = iv["Z"].to_numpy(float)
d = iv["D"].to_numpy(float)
design = np.column_stack([np.ones(len(z)), z])
beta = np.linalg.lstsq(design, d, rcond=None)[0]
fitted = design @ beta
ss_full = float(np.sum((d - fitted) ** 2))
ss_restricted = float(np.sum((d - d.mean()) ** 2))
f_stat = ((ss_restricted - ss_full) / 1) / (ss_full / (len(d) - 2))
print(f"First-stage coefficient on the email: {beta[1]:.3f}")
print(f"First-stage F-statistic: {f_stat:.1f}")
print("A first stage above 10 is the usual rule of thumb for a relevant instrument.")
First-stage coefficient on the email: 0.213
First-stage F-statistic: 480.9
A first stage above 10 is the usual rule of thumb for a relevant instrument.

Weak Instruments

A weak instrument (first-stage F < ~10, or the stricter threshold of ~104 for standard 2SLS inference) produces:

  • Biased estimates (bias toward OLS)
  • Wildly uncertain estimates (large standard errors)
  • Invalid standard errors and confidence intervals

Solutions: - Report the first-stage F prominently - With borderline strength, use Anderson–Rubin confidence intervals - Find a stronger instrument if possible

Intuition: Wald Estimator

For a binary instrument and binary treatment, the IV estimate has a simple intuitive form:

\[ \hat{\tau}_{LATE} = \frac{\text{ITT on } Y}{\text{ITT on } D} = \frac{E[Y|Z=1] - E[Y|Z=0]}{E[D|Z=1] - E[D|Z=0]} \]

This is the intent-to-treat (ITT) effect on the outcome divided by the ITT effect on treatment. It asks: “for every additional person the email encouraged to adopt, how much did their spending increase?”

Code 
# Wald estimator (intuitive form)
itt_y = iv.loc[iv.Z == 1, "y"].mean() - iv.loc[iv.Z == 0, "y"].mean()
itt_d = iv.loc[iv.Z == 1, "D"].mean() - iv.loc[iv.Z == 0, "D"].mean()

wald_estimate = itt_y / itt_d

print("Wald estimator (intuitive form):")
print(f"ITT on Y (outcome): ${itt_y:.2f}")
print(f"ITT on D (treatment): {itt_d:.3f}")
print(f"Wald estimate (ITT_Y / ITT_D): ${wald_estimate:.2f}")
print(f"True LATE: ${TRUE_LATE}")
Wald estimator (intuitive form):
ITT on Y (outcome): $3.05
ITT on D (treatment): 0.213
Wald estimate (ITT_Y / ITT_D): $14.30
True LATE: $15.0

The Wald estimate matches the 2SLS estimate and is close to the true LATE. This shows the intuition: we’re scaling the outcome effect by how much the instrument moved treatment.

Adding Controls

IV can include exogenous control variables to improve precision. These controls should be pre-treatment and not affected by the instrument.

Code 

iv["tenure"] = 3 + 0.5 * ability + rng.normal(0, 1, n)
iv_controls = InstrumentalVariable(n_boot=200, random_state=1)
iv_controls.fit(
    iv["y"].to_numpy(),
    iv["D"].to_numpy(),
    iv["Z"].to_numpy(),
    X=iv[["tenure"]].to_numpy(),
)
print("IV with tenure as a covariate:")
print(pd.Series(iv_controls.summary()).round(3))
IV with tenure as a covariate:
estimate         13.851
ci_lower         11.111
ci_upper         16.644
posterior_mean   13.789
dtype: float64

Adding controls can improve precision but doesn’t fix a weak instrument or an invalid exclusion restriction.

Validating the Instrument

IV’s credibility rests entirely on the instrument’s validity. The three conditions are:

1. Relevance (Testable)

The instrument must affect treatment. Test with the first-stage F-statistic.

Rule of thumb: F > 10 (Stock & Yogo weak instrument test), or F > 104 for standard 2SLS inference.

2. Exclusion Restriction (Untestable)

The instrument must affect the outcome only through treatment. This is the most critical and untestable assumption.

Examples of violations: - An encouragement email itself (not the feature) reminds customers the product exists → increases spending directly - Distance to store affects both program participation and spending through convenience, not just the program

Defense: Argue from institutional knowledge and design. The cleanest instruments are encouragement designs where the only plausible channel is treatment.

3. Independence (Untestable)

The instrument must be as good as random — not correlated with unobserved confounders.

Best defense: Randomized instruments (encouragement designs). For natural instruments, argue based on the assignment mechanism.

Real-World Data

The series below is a real dataset shipped with CausalPy (PyMC Labs), under the Apache License 2.0. The files live in causalpy/data.

@software{causalpy,
  author = {{PyMC Labs}},
  title = {CausalPy: Causal inference for quasi-experiments in Python},
  url = {https://github.com/pymc-labs/CausalPy},
  year = {2026}
}

PyCausalML fits the estimator. CausalPy is the data source. When CausalPy is installed, causalpy.load_data reads the packaged file. Otherwise the notebook reads the same CSV from that GitHub directory.

Two classical instruments are included.

  1. Returns to schooling (schoolReturns in CausalPy; file schoolingReturns.csv). Card (1995) uses growing up near a college as an instrument for years of education. The outcome here is log wage.
  2. Colonial origins (risk in CausalPy; file AJR2001.csv). Acemoglu, Johnson, and Robinson (2001) use log settler mortality as an instrument for protection against expropriation, with log GDP per capita as the outcome. The exclusion restriction for this instrument is disputed, so the estimate is reported with that assumption in view.
Code 
CAUSALPY_FILES = {
    "banks": "banks.csv",
    "covid": "deaths_and_temps_england_wales.csv",
    "drinking": "drinking.csv",
    "risk": "AJR2001.csv",
    "schoolReturns": "schoolingReturns.csv",
    "california_prop99": "california_prop99.csv",
}

def load_causalpy(name: str) -> pd.DataFrame:
    """Load a CausalPy example dataset.

    Uses causalpy.load_data when that package is installed. Otherwise reads
    the same CSV from the CausalPy GitHub data directory.
    """
    try:
        import causalpy as cp
        frame = cp.load_data(name)
        print(f"Loaded {name!r} with causalpy.load_data. Shape {frame.shape}.")
        return frame
    except ImportError:
        url = (
            "https://raw.githubusercontent.com/pymc-labs/CausalPy/main/causalpy/data/"
            + CAUSALPY_FILES[name]
        )
        frame = pd.read_csv(url)
        print(f"CausalPy is not installed. Loaded {name!r} from")
        print(url)
        print("Install the package with: pip install 'pycausalml[causalpy]'")
        print(f"Shape {frame.shape}.")
        return frame

school = load_causalpy("schoolReturns")
school = school.dropna(subset=["wage", "education", "nearcollege", "experience"]).copy()
school = school[school["wage"] > 0]
y_school = np.log(school["wage"].to_numpy(float))
educ = school["education"].to_numpy(float)
near = (school["nearcollege"].astype(str) == "yes").astype(float).to_numpy()
experience = school[["experience"]].to_numpy(float)

school_iv = InstrumentalVariable(n_boot=200, random_state=0)
school_iv.fit(y_school, educ, near, X=experience)
print("Return to a year of schooling (log wage), instrumented by living near a college:")
print(pd.Series(school_iv.summary()).round(3))

z = near
design = np.column_stack([np.ones(len(z)), z, experience])
beta = np.linalg.lstsq(design, educ, rcond=None)[0]
ss_full = float(np.sum((educ - design @ beta) ** 2))
design_r = np.column_stack([np.ones(len(z)), experience])
beta_r = np.linalg.lstsq(design_r, educ, rcond=None)[0]
ss_r = float(np.sum((educ - design_r @ beta_r) ** 2))
f_school = ((ss_r - ss_full) / 1) / (ss_full / (len(educ) - design.shape[1]))
print(f"First-stage coefficient on near-college: {beta[1]:.3f}")
print(f"Partial first-stage F: {f_school:.1f}")

ajr = load_causalpy("risk").dropna(subset=["loggdp", "risk", "logmort0"])
ajr_iv = InstrumentalVariable(n_boot=300, random_state=0)
ajr_iv.fit(ajr["loggdp"].to_numpy(), ajr["risk"].to_numpy(), ajr["logmort0"].to_numpy())
print("\nAJR 2001, log GDP on expropriation protection, instrumented by log settler mortality:")
print(pd.Series(ajr_iv.summary()).round(3))
CausalPy is not installed. Loaded 'schoolReturns' from
https://raw.githubusercontent.com/pymc-labs/CausalPy/main/causalpy/data/schoolingReturns.csv
Install the package with: pip install 'pycausalml[causalpy]'
Shape (3010, 23).
Return to a year of schooling (log wage), instrumented by living near a college:
estimate         0.262
ci_lower         0.207
ci_upper         0.328
posterior_mean   0.261
dtype: float64
First-stage coefficient on near-college: 0.600
Partial first-stage F: 58.0
CausalPy is not installed. Loaded 'risk' from
https://raw.githubusercontent.com/pymc-labs/CausalPy/main/causalpy/data/AJR2001.csv
Install the package with: pip install 'pycausalml[causalpy]'
Shape (64, 31).

AJR 2001, log GDP on expropriation protection, instrumented by log settler mortality:
estimate         0.929
ci_lower         0.709
ci_upper         1.441
posterior_mean   0.951
dtype: float64

Common Pitfalls

1. Weak Instruments

A weak instrument produces biased, uncertain estimates.

Solution: Report first-stage F prominently. If weak, find a stronger instrument or use Anderson–Rubin CIs.

2. Exclusion Violations

If the instrument affects the outcome through channels other than treatment, IV is biased.

Example: An email encouraging feature adoption might also remind customers about the brand, increasing spending directly.

Solution: Design instruments where the only plausible channel is treatment. Be honest about uncertainty.

3. Misinterpreting LATE as ATE

IV estimates LATE (effect on compliers), not ATE (effect on everyone).

Example: Finding that an email-induced adoption is worth $15 does not mean adoption is worth $15 for all customers.

Solution: Be explicit with stakeholders: “This is the effect for customers whose adoption the email tipped.”

4. Invalid Monotonicity

IV assumes no “defiers” — units that would adopt because they were not encouraged. This is usually reasonable but worth noting.

Example: A contrarian customer who adopts a feature specifically because they didn’t get the email.

Solution: Usually not a major concern in business settings, but worth mentioning in limitations.

5. Including Bad Controls

Controls that are affected by the instrument or post-treatment variables will bias IV estimates.

Solution: Only include pre-treatment, exogenous controls.

When to Use IV

IV is the right choice when:

  • Treatment is confounded by unobservables (self-selection, unmeasured ability)
  • A credible instrument exists that shifts treatment without directly affecting the outcome
  • The instrument is strong (first-stage F > 10)
  • Stakeholders accept the exclusion restriction (instrument only affects outcome through treatment)
  • You’re comfortable with LATE (effect on compliers) as the estimand

Consider alternatives when:

  • No credible instrument → Propensity methods, DiD, RDD
  • Weak instrument → Find stronger instrument or accept limitations
  • Need ATE (not LATE) → RCT or other methods
  • Can justify unconfoundedness → Propensity methods, DML

Design Better Instruments

The highest-leverage move is often upstream: design better rollouts that create natural instruments.

Encouragement Designs

Randomize encouragement even when treatment cannot be randomized:

  • Randomly send emails encouraging feature adoption
  • Randomly offer training programs to some employees
  • Randomly provide information about a product

These create clean instruments because the randomization ensures independence.

Geographic or Temporal Variation

Leverage natural variation:

  • Policy changes that affect some regions but not others
  • Supply shocks that create temporary constraints
  • Seasonal patterns that affect treatment intensity

Defend the exclusion restriction by arguing the only channel is treatment.

Summary

Instrumental Variables is a powerful method for estimating causal effects when treatment is confounded by unobservables. Its strength lies in leveraging exogenous variation in treatment induced by a credible instrument.

Key takeaways:

  • IV solves unobserved confounding by using an instrument that shifts treatment
  • The instrument must be relevant (strong first stage), independent (as good as random), and satisfy exclusion (only affects outcome through treatment)
  • IV estimates LATE (effect on compliers), not ATE — be explicit with stakeholders
  • Check first-stage F-statistic; weak instruments produce biased estimates
  • Encouragement designs (randomized emails, offers) are the cleanest business instruments
  • Exclusion restrictions are untestable — defend them with institutional knowledge
  • Design better rollouts to create natural instruments

Scientific Terminology

Plain-language definitions for the terms in this tutorial are in the scientific terminology glossary.

Back to top