What works best and when: Accounting for multiple sources of pureselection bias in program evaluations

Haeil Jung, Maureen A. Pirog

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

Most evaluations are still quasi-experimental and most recent quasi-experimental methodological research has focused on various types of propensity score matching to minimize conventional selection bias on observables. Although these methods create better-matched treatment and comparison groups on observables, the issue of selection on unobservables still looms large. Thus, in the absence of being able to run randomized controlled trials (RCTs) or natural experiments, it is important to understand how well different regression-based estimators perform in terms of minimizing pure selection bias, that is, selection on unobservables. We examine the relative magnitudes of three sources of pure selection bias: heterogeneous response bias, time-invariant individual heterogeneity (fixed effects [FEs]), and intertemporal dependence (autoregressive process of order one [AR(1)]). Because the relative magnitude of each source of pure selection bias may vary in different policy contexts, it is important to understand how well different regression-based estimators handle each source of selection bias. Expanding simulations that have their origins in the work of Heckman, LaLonde, and Smith (), we find that difference-in-differences (DID) using equidistant pre- and postperiods and FEs estimators are less biased and have smaller standard errors in estimating the Treatment on the Treated (TT) than other regression-based estimators. Our data analysis using the Job Training Partnership Act (JTPA) program replicates our simulation findings in estimating the TT.

Original languageEnglish
Pages (from-to)752-777
Number of pages26
JournalJournal of Policy Analysis and Management
Volume33
Issue number3
DOIs
Publication statusPublished - 2014
Externally publishedYes

ASJC Scopus subject areas

  • Business, Management and Accounting(all)
  • Sociology and Political Science
  • Public Administration

Fingerprint Dive into the research topics of 'What works best and when: Accounting for multiple sources of pureselection bias in program evaluations'. Together they form a unique fingerprint.

  • Cite this