The World Bank Human Development Network Spanish Impact Evaluation Fund www.worldbank.org/hdchiefeconomist INSTRUMENTAL VARIABLES Technical Track Session IV This material constitutes supporting material for the "Impact Evaluation in Practice" book. This additional material is made freely but please acknowledge its use as follows: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: Ancillary Material, The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and not necessarily those of the World Bank.

Instrumental Variables and IE Instrumental variables have many uses IV can be generated ex ante: o Randomized promotion (or encouragement design) IV can be used ex post to correct for o Randomized offering of a program non-compliance or conduct retrospective IE: o Correction for non-compliance to recover TOT from ITT o E.g. Randomized Assignment with noncompliers

o E.g. Fuzzy Regression Discontinuity o Look for exogenous variation to evaluate the Here: impact of a program in absence of a prospective o General Principles behind IVs design. o Ex ante focus on randomized promotion o IV, non-compliance and randomized offering An example to start off with Say we wish to evaluate a voluntary job training program o Any unemployed person is eligible (Universal eligibility) o Some people choose to register (Participants) o Other people choose not to register (Nonparticipants)

Some simple (but not-so-good) ways to evaluate the program: o Compare before and after situation in the participant group o Compare situation of participants and nonparticipants after the intervention o Compare situation of participants and nonparticipants before and after (DD). Voluntary job training Say we decide to compare outcomes for those who program participate to the outcomes of those who do not participate: A simple model to do this: y = + 11 P + 22 x + P= 1

0 If person participates in training If person does not participate in training x = Control variables (exogenous & observed) Why is this not 2 working? problems: o Variables that we omit (for various reasons) but that are important o Decision to participate in training is endogenous. Problem #1: Omitted Even if we try to control for everything, Variables well miss: (1) Characteristics that we didnt know they mattered, and (2) Characteristics that are too complicated to o Talent, motivation

measure (not observables or not observed): o Level of information and access to services o Opportunity cost of participation Full model would be: y = 00 + 11 x + 22 P + 33 M11 + But we cannot observe M1 , the missing and unobserved variables. Omitted variable bias y = 00 + 11 x + 22 P + 33 M11 True model is: + But we estimate: y = 00 + 11 x + 22 P + If there is a correlation between M1 and P, then the OLS estimator of 2 will not be a consistent estimator of 2, the true impact of P. Why? When M1 is missing from the regression, the coefficient of P will pick up some of the effect of

M1 Problem #2: Endogenous Decision to Participate y = 00 + 11 x + 22 P + True model is: P = 00 + 11 x + 22 M22 with M2 = Vector of unobserved /+ missing characteristics (i.e. we dont fully know why people decide to participate) Since we dont observe M2 , we can only estimate a simplified model: y = 00 + 11 x + 22 P + Is 2, OLS an unbiased estimator of 2?

Problem #2: Endogenous Decision to Participate We estimate: y = 00 + 11 x + 22 P + But true model is:y = 00 + 11 x + 22 P + P = 00 + 11 x + 22 M22 with + Is 2, OLS an unbiased estimator of 2? Corr (, P) , P) = corr (, P) , 0 + 1 x + 2 M2 +) = 1 corr (, P) , x)+ 2 corr (, P) , M2) = 2 corr (, P) , M2) If there is a correlation between the missing variables that determine participation (e.g. Talent) and outcomes not explained by observed characteristics, then the OLS estimator will be biased. What can we do to solve this problem? We

estimate: y = 00 + 11 x + 22 P + So the problem is the correlation between P and How about we replace P with something else, call it Z: o Z needs to be similar to P o But is not correlated with Back to the job training program P= participation = that part of outcomes that is not explained by program participation or by observed characteristics Im looking for a variable Z that is: (1) (2)

Closely related to participation P but doesnt directly affect peoples outcomes Y, other than through its effect on participation. So this variable must be coming from outside. Generating an outside variable for the job Say that a social worker visits unemployed training program persons to encourage them to participate. o She only visits 50% of persons on her roster, and If effective, many people visits will

o she She is randomly chooses whom she she will visit enroll. There will be a correlation between receiving a visit and enrolling But visit does not have direct effect on outcomes (e.g. income) apart from its effect through enrollment in the training program. Randomized encouragement or promotion visits are an Instrumental

Characteristics of an instrumental variable Define a new variable Z Z= 1 If person was randomly chosen to receive the encouragement visit from the social worker 0 If person was randomly chosen not to receive the encouragement visit from the social worker Corr (, P) Z , P ) > 0 People who receive the encouragement visit are more likely to participate than those who dont Corr (, P) Z , ) = 0 No correlation between receiving a visit and benefit to the program apart from the effect of the visit on participation. Z is called an instrumental variable

Two-stage least squares Remember the original model with (2SLS) y P: = 00 + 11 x + 22 P + endogenous Step 1 Regress the endogenous variable P on the instrumental variable(s) Z and other exogenous variables P = 00 + 11 x + 22 Z + Calculate the predicted value of P for each observation: P Since Z and x are not correlated with , neither will be P. You will need one instrumental variable for each

potentially endogenous regressor. Two-stage least squares (2SLS) Step 2 Regress y on the predicted variable P and the other exogenous variables y = 00 + 11 x + 22 P + Note: The standard errors of the second stage OLS need to be corrected because P is not a fixed regressor. In Practice: Use STATA ivreg command, which does the two steps at once and reports correct standard errors. Intuition: By using Z for P, we cleaned P of its correlation with It can be shown that (under certain conditions) 2,IV yields a consistent estimator of 2 (large sample theory) Where do we find

instrumental variables? Searching for an IV ex Hard and risky post Generating an IV with information campaign designed ex ante o If everyone is eligible to participate in treatment o But some have more information than others (Who has more information will be more likely to participate) o Provision of additional information on a random basis Example 1: voluntary job training program Population Population eligible eligible for

for job job training training program program Random Random Sample Sample Randomized assignment Standard Standard Information Information Package Package only only Monthly Monthly income income 1 1 year

year later later = = 700 700 Standard Standard Information Information Package Package + + Additional Additional visit visit Monthly Monthly income income 1 1 year year later later = = 850 850

30% 30% take-up take-up 90% 90% take-up take-up Question: what is the impact of the job training program? Standard Standard Information Information Package Package only only Monthly Monthly income income 1 1 year year later

later = = 700 700 Standard Standard + + Additional Additional Information Information Package Package Monthly Monthly income income 1 1 year year later later = = 850 850 30% 30% take-up

take-up 90% 90% take-up take-up Question: what is the impact of the job training program? Difference between the well informed and not well informed group: .. Corrected for the differential take-up rate: . Practically: Impact = Link back to the estimation formula Stage 1 o Regress the participation on training on a dummy for whether person received

additional visit (linear model) o Compute predicted value of participation Stage 2 Regress wages on the predicted value of participation Example 2: School autonomy in Nepal Goal To Evaluate: A. Autonomous school management by communities B. School report cards Data o o o o You can include 1000 schools in the evaluation Each community freely chooses to participate or not

School report cards done by NGOs Each community has exactly one school Task Design the implementation of the program so it can be evaluated propose method of evaluation. School autonomy in Nepal Intervention B: School report card intervention by NGO. Instrumental variable for Intervention A: NGO visits community to inform on procedures for transfer of the school to community management. Yes

No Total Yes 300 300 600 No 200 200 400 Tota l

500 500 1000 Reminder and a word of caution corr (, P) Z,) =0 o If corr (, P) Z , ) 0, Bad instrument o Finding a good instrument is hard! o But you can build one yourself with a randomized encouragement design corr (, P) Z,P) 0 o Weak instruments: the correlation between Z and P needs to be sufficiently strong. o If not, the bias stays large even for large sample sizes. Recovering TOT from ATE in case of non-compliance Sometimes eligible units are selected randomly into the treatment group, are offered treatment, but not all of them

accept it. Computing the Average Treatment Effect (ATE) Straight difference in average outcomes between the group to whom you offered treatment, and the group to whom you did not offer treatment Computing the Effect of Treatment on the Treated (TOT) Use the randomized offering as an instrumental variable (, P) Z) for whether people accepted the treatment (, P) P) Note: IV is a local effect IV methods identify the average gains to persons induced to change their choice by a change of the instrument (referred to as compliers) however we cannot identify who these people are (local average treatment effect

or LATE) different instruments will identify different parameters and answer different questions Caution in extrapolating to the whole population References Angrist, J. D. and A. Krueger (2001). Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments, Journal of Economic Perspectives, 15(, P) 4). Angrist, J. D., G. W. Imbens and D. B. Rubin (1996). Identification of Causal Effects Using Instrumental Variables, Journal of the American Statistical Association, Vol. 91, 434. Angrist, J., Bettinger, E., Bloom, E., King, E. and M. Kremer (2002). Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment, American Economic Review, 92, Bradlow, E., 5. (1998). Encouragement Designs: An Approach to Self-Selected Samples in an Experimental Design, Marketing Letters, Imbens, 9(4) G. W. and J. D. Angrist, (1994). Identification and

Estimation of Local Average Treatment Effects. Econometrica, 62(2). Newman, J., M. Pradhan, L. B. Rawlings, G. Ridder, R. Coa, J. L. Evia, (2002). An Impact Evaluation of Education, Health, and Water Supply Investments by the Bolivian Social Investment Fund., World Bank Economic Review, vol. 16(2). Thank You ? Q&A