Catalogue no. 81-595-MIE No. 008 ISSN: 1704-8885 ISBN: 0-662-34802-8 Research Paper Educaton, sklls and learnng Research papers The labour market mpacts of adult educaton and tranng n Canada by Shek-wa Hu and Jeffrey Smth Culture, Toursm and the Centre for Educaton Statstcs Dvson 2001 Man Buldng, Ottawa, K1A 0T6 Telephone: 1 800 307-3382 Fax: 1 613 951-9040 Ths paper represents the vews of the authors and does not necessarly reflect the opnons of Statstcs Canada. Statstcs Canada Human Resources Development Canada Statstque Canada Développement des ressources humanes Canada
Educaton, sklls and learnng Research papers The labour market mpacts of adult educaton and tranng n Canada Shek-wa Hu Department of Economcs, Unversty of Wnnpeg Jeffrey Smth Department of Economcs, Unversty of Maryland Publshed by authorty of the Mnster responsble for Statstcs Canada Mnster of Industry, 2003 All rghts reserved. No part of ths publcaton may be reproduced, stored n a retreval system or transmtted n any form or by any means, electronc, mechancal, photocopyng, recordng or otherwse wthout pror wrtten permsson from Lcence Servces, Marketng Dvson, Statstcs Canada, Ottawa, Ontaro, Canada K1A 0T6. October 2003 Frequency: Irregular ISSN 1704-8885 ISBN 0-662-34802-8 Ottawa La verson françase de cette publcaton est dsponble sur demande (n o 81-595-MIF2003008 au catalogue). Statstcs Canada Human Resources Development Canada Ths paper represents the vews of the authors and does not necessarly reflect the opnons of Statstcs Canada or Human Resources Development Canada.
How to obtan more nformaton Specfc nqures about ths product and related statstcs or servces should be drected to: Clent Servces, Culture, Toursm and the Centre for Educaton Statstcs, Statstcs Canada, Ottawa, Ontaro, K1A 0T6 (telephone: (613) 951-7608; toll free at 1 800 307-3382; by fax at (613) 951-9040; or e-mal: educatonstats@statcan.ca). For nformaton on the wde range of data avalable from Statstcs Canada, you can contact us by callng one of our toll-free numbers. You can also contact us by e-mal or by vstng our Web ste. Natonal nqures lne 1 800 263-1136 Natonal telecommuncatons devce for the hearng mpared 1 800 363-7629 E-mal nqures nfostats@statcan.ca Web ste www.statcan.ca Orderng nformaton Ths product, Catalogue No. 81-595-MIE2003008, s avalable on the Internet for free. Users can obtan sngle ssues at: http://www.statcan.ca/cg-bn/downpub/studesfree.cg. Standards of servce to the publc Statstcs Canada s commtted to servng ts clents n a prompt, relable and courteous manner and n the offcal language of ther choce. To ths end, the Agency has developed standards of servce whch ts employees observe n servng ts clents. To obtan a copy of these servce standards, please contact Statstcs Canada toll free at 1 800 263-1136.
Acknowledgements We thank Human Resources Development Canada for fnancal support and Statstcs Canada for support n data access. We thank Luce Glbert for her support, encouragement and patence throughout the completon of ths report. We also thank the partcpants to the Adult Educaton and Tranng Survey Workshop held n Ottawa n May 2001 for helpful comments. Note of apprecaton Canada owes the success of ts statstcal system to the long-standng co-operaton of Statstcs Canada, the ctzens of Canada, ts busnesses, governments and other nsttutons. Accurate and tmely statstcal nformaton could not be produced wthout ther contnued co-operaton and goodwll. 3
Table of Contents Abstract 5 1. Introducton 6 2. Data 9 2.1 The AETS data 9 2.2 Defnng our measures of tranng recept 10 2.3 Defnng our outcome varables 11 3. Evaluatng the labour market effects of educaton and tranng 13 3.1 A model of labour market outcomes and partcpaton n tranng 13 3.2 Parameters of nterest 16 3.3 Cross-sectonal evaluaton methods that assume selecton on observables 17 3.4 Cross-sectonal evaluaton methods that assume selecton on unobservables 21 3.5 Assumptons and data 22 4. Estmates of the labour market mpact of tranng 24 4.1 Impacts of tranng on employment 24 4.2 Usual weekly earnngs 27 4.3 Plausblty of the mpact estmates 27 4.4 Summary of the mpact estmates 28 5. Problems wth the AETS for mpact evaluaton 30 6. Conclusons 31 Tables 32 References 39 Endnotes 42 Appendx tables 43 Cumulatve Index 49 Educaton, sklls and learnng Research papers 50 4
Abstract In ths report, we use the data from the 1998 Adult Educaton and Tranng Survey (AETS) to estmate the mpact of partcpatng n adult educaton and tranng on the employment and earnngs of Canadans. We apply methods that assume selecton on observables, ncludng both standard regresson-based methods and propensty score matchng methods. We also apply methods based on nstruments or excluson restrctons, ncludng standard nstrumental varables estmaton and the well-known Heckman bvarate normal selecton estmator. These methods am to deal wth selecton on unobservables. We fnd that none of the methods we examne produce plausble estmates of the mpact of adult educaton and tranng, although the methods that assume selecton on observables produce more reasonable estmates than those that assume an nstrument or excluson restrcton. Based on the results of our analyss, we suggest mprovements to the AETS that would make t a better tool for estmatng the labour market mpacts of adult educaton and tranng. 5
1. Introducton Evaluatng the mpacts of adult educaton and tranng s of great value for a number of reasons. To the polcy-maker, nformaton on the labour market effects of publcly fnanced adult educaton and tranng has obvous mplcatons once placed nsde a coherent cost-beneft framework. Smlarly, nformaton on the labour market effects of self-fnanced and employer-provded adult educaton and tranng provdes nsght on the extent to whch exstng polces to subsdze or tax such tranng (or the earnngs ncrements t leads to, f any) could nfluence decsons, potentally away from the socally optmal level of partcpaton n these actvtes. For scholars, nformaton on the mpacts of adult educaton and tranng provdes nsght nto how ndvduals and frms accumulate human captal and sheds lght on questons of poltcal economy, credt constrants on ndvduals that may prevent them from makng ndvdually and socally optmal nvestments n human captal, and theores of under-provson of tranng n the labour market. The lterature dstngushes publcly fnanced or provded tranng, especally that for the unemployed or for workers re-enterng the labour force, from that provded by frms to ther employees. There are several reasons for dong so, ncludng the fact that the populatons recevng the two types of adult educaton and tranng have qute dfferent characterstcs, as well as the fact that the content and duraton of the tranng tend to dffer substantally. We follow that dstncton n our emprcal work below and n our bref lterature revew here. An extensve lterature exsts on the labour market effects of government employment and tranng programs. Table 25 of Heckman, LaLonde and Smth (1999) lsts lterally dozens of such studes from numerous countres around the world. In the Unted States, numerous studes have made use of random assgnment methods to produce hgh qualty, credble estmates of the mpacts of programs focused on job search assstance, classroom tranng and wage subsdes. Table 22 of Heckman, LaLonde and Smth (1999) provdes a partal lst of such studes. The publshed evaluaton record for such programs n Canada s much more lmted. Two notable sources are Park, et al. (1996) and the wdely cted Self-Suffcency Project, summarzed n Mchalopoulos, et al. (2000). Rddell (1991), Smth and Sweetman (2001) and Warburton and Warburton (2002) analyze and crtque the evaluaton of publc adult educaton and tranng programs n Canada. Fortunately, the types of programs and populatons served n the Unted States are smlar enough that ther evaluatons provde a useful benchmark to compare our fndngs n ths study. The exstng lterature on the effects of employer-provded tranng s much thnner. There are several reasons for ths. Frst, governments are, qute reasonably, wllng to spend a lot more money evaluatng ther own programs than evaluatng those of prvate frms. Second, good data on the recept of employer-provded tranng s hard to come by. Even when a large survey contans questons relatng to employer- 6
provded tranng, there are strong ssues of measurement error. 1 Thrd, whle ndvduals typcally partcpate n government-fnanced tranng only once, or only at rare ntervals when they are unemployed, employer-provded tranng often contnues throughout the lfecycle. Tranng epsodes are often short, on the order of days or weeks, and there are often multple spells wthn a year. As documented below, these patterns characterze employer-provded tranng n Canada. These features of employer-fnanced tranng mply the need for longtudnal rather than cross-sectonal data and also make t dffcult to know how to code partcpaton whether n terms of ncdence, hours, epsodes and so on. Brtsh studes by Arulampalam, Booth and Elas (1997) and by Blundell, Dearden and Meghr (1996) attempted to use panel data to study the labour market effects of such tranng and wrestled wth these ssues n depth. Heckman, Lochner, Smth and Taber (1997) and Carnero, Heckman and Manol (2002) summarze the evdence from the lterature that attempts to evaluate the labour market effects of employer-fnanced adult educaton and tranng. One common fndng s qute hgh estmated effects, whch are generally attrbuted to a falure of the avalable data to completely control for the assumed selecton of more able and more motvated employees nto tranng wthn frms. We dd not fnd any studes along these lnes usng data from Canada. In ths paper, we estmate the mpacts of partcpaton n adult educaton and tranng usng the data from the 1998 Adult Educaton and Tranng Survey (AETS). The AETS s a supplement to the Canadan Labour Force Survey (LFS), and as a result ncludes all of the nformaton on labour market actvty and demographc characterstcs ncluded n the LFS. Our analyss has two prmary goals. The frst s to begn to fll the vod n the lterature n regard to the labour market effects of adult educaton and tranng n Canada. The second s to determne the value of the AETS as a data source for use n evaluaton. The second goal s not trval because the prmary focus of the AETS s on documentng the types and extent of partcpaton n adult educaton and tranng, as well as provdng a vehcle for studes of the determnants of partcpaton n adult educaton and tranng. The actve lterature on econometrc methods for evaluatng the mpacts of treatments such as adult educaton and tranng ncludes a varety of alternatve estmaton strateges. 2 The basc problem addressed by all of these estmators s the general absence of data on random assgnment treatments. In the absence of random assgnment, we have observatonal data, whch have the fault that the observed varaton n treatment, n our context the observed varaton n partcpaton n adult educaton and tranng, comes from the (assumed) optmzng choces of ndvduals. Indvduals have nformaton that the analyst does not, and have characterstcs that the analyst does not observe. As a result, smple comparsons of the labour market outcomes of partcpants and non-partcpants combne the effects of partcpaton wth dfferences due to non-random partcpaton. These dfferences lead to selecton bas. The lterature offers two wde classes of estmators to deal wth ths problem: those that assume suffcent nformaton n the data to mostly correct for systematc dfferences between partcpants and non-partcpants, and those that assume the absence of such nformaton but nstead assume the presence of a varable (an nstrument or excluson restrcton) that affects partcpaton but not outcomes n the 7
absence of partcpaton. We utlze two evaluaton strateges drawn from each of these broad classes. In the frst class, we use standard regresson-based methods (the so-called Barnow, Can and Goldberger (1980) estmator) as well as recently developed propensty score matchng methods. In the second class, we use nstrumental varables methods as well as the wdely known Heckman (1979) bvarate normal selecton estmator. In each case, our specfcatons buld on what we learned from our analyss of the determnants of partcpaton n adult educaton and tranng. As dscussed, for example, n Heckman, LaLonde and Smth (1999) and Smth (2000), each of these estmators makes dfferent assumptons about the processes that generate partcpaton n adult educaton and tranng as well as employment and earnngs, the two labour market outcomes we examne. At most, the assumptons of one of the estmators we consder match the data and nsttutonal context we examne here. Our purpose n examnng all of them s to allow the data to nform us, n part, regardng whch estmaton strateges seem most plausble n the AETS context, and also to allow the fndngs to suggest ways n whch the AETS data mght be mproved for the purposes of mpact estmaton. The mpact estmates we obtan from all of the econometrc methods we apply prove dsappontng. The mpact estmates for tranng fnanced by employers are much too large, whle those for tranng fnanced by the government are often negatve. Theoretcal arguments based on expected rates of return, as well as comparsons wth alternatve estmates n the lterature that use better data (n the case of government-fnanced tranng, often expermental data), cast strong doubt on the estmates obtaned here. These poor results hold for all of the estmators we examne, but the results from the nstrumental varable and Heckman bvarate normal estmators prove the least credble. Senstvty analyses ndcate that these poor results are robust to modest changes n the specfcaton we estmate, leadng us to conclude that the prmary problem wth the estmates les n the data rather than n the methods. Put smply, the AETS data lack crtcal elements necessary to produce credble estmates of the mpact of adult educaton and tranng on labour market outcomes. The remander of our study proceeds as follows. Secton 2 descrbes the 1998 AETS data that we use, and defnes our measures of tranng partcpaton and labour market outcomes. Secton 3 descrbes the non-expermental estmaton methods we employ to estmate the effects of adult educaton and tranng on partcpants earnngs and employment. Secton 4 presents our mpact estmates, and ndcates why they are problematc n lght of the exstng theoretcal and emprcal lterature. Secton 5 brefly summarzes our suggestons for ways to make the AETS a better tool for estmatng mpacts; these suggestons are elaborated on n our companon paper Hu and Smth (2003). We conclude n Secton 6 wth a summary and some conclusons regardng adult educaton and tranng n Canada. 8
2. Data 2.1 The AETS data The data we use come from the 1998 Adult Educaton and Tranng Survey (AETS) master fle. The 1998 AETS was the thrd n a seres of comparable surveys desgned to measure partcpaton n adult educaton and tranng, defned as educaton and tranng that occur after the concluson of formal schoolng. The objectves of the survey are to measure partcpaton rates, determne the role of employers n adult educaton and tranng partcpaton and provson, and to dentfy barrers to adult educaton and tranng. Statstcs Canada collected the AETS data on behalf of Human Resources Development Canada. The AETS s a supplement to the Labour Force Survey (LFS). The LFS has an overlappng panel desgn. Each month a new random sample of the LFS populaton cvlans ages 15 and over s drawn. Each such sample s called a rotaton group. Each rotaton group s of roughly equal sze, and each one remans n the LFS for sx consecutve months, at whch pont t s no longer followed but nstead replaced by a new rotaton group. The AETS was admnstered to fve of the sx rotaton groups n the January 1998 and March 1998 Labour Force Surveys. 3 The 1998 AETS (hereafter just AETS) conssts of fve modules, desgnated A to E. The questons n Module A collect background nformaton on the respondent. The module also asks whether the respondent receved any tranng or educaton wthn the prevous year. Respondents who ndcate that they dd not receve any educaton or tranng skp the followng three modules, B, C and D, and proceed drectly to module E. The questons n Module B ask about the detals of any tranng or educaton leadng to formal certfcaton of some sort. The AETS calls such educaton and tranng tranng programs. The questons n Module C ask about the detals of any educaton or tranng not leadng to formal certfcaton. The AETS calls such educaton and tranng tranng courses. The questons n Module D concern courses taken for hobby, recreatonal or personal development reasons. They also cover resdual tranng actvtes not reported n Module B or C. We omt the courses reported n Module D from our analyss due to our focus on tranng related to labour market outcomes. In each of Modules B, C, and D, the survey collects nformaton on up to fve dfferent courses or programs. The nformaton collected on each course or program ncludes the feld of study, the locaton, the provder, the teachng medum, the duraton, whether or not the tranng was completed, who pad for the tranng, and what employer support was provded (f any). The survey also collects nformaton on respondents reasons for takng the tranng, expectatons regardng the tranng, and opnons of the tranng s usefulness. All of the questons n Modules B, C and D refer to educaton and tranng actvtes undertaken n 1997. 9
All respondents complete Module E. Ths module collects nformaton on labour market outcomes n 1997 for whch data are not collected on the LFS. Ths ncludes nformaton on the man job n 1997 f t dffers from that at the tme of the LFS completon n 1998. Module E also collects a varety of demographc nformaton ncludng characterstcs of the respondents parents and the respondents mmgraton and dsablty statuses. To supplement the nformaton collected n Module E, the labour force nformaton collected on the LFS s attached to the record of each AETS respondent. In addton, Module E ncludes a seres of questons that seek to determne why respondents were not able to partcpate n the tranng they wanted or needed to take durng the reference year. The household response rate to the LFS was 95.3% 4, whle 85.2% of LFS respondents responded to the 1998 AETS. Ths mples a respectable overall AETS response rate of 81.2%. The 1998 AETS has a total of 33,410 respondents. In order to restrct our attenton to those who have completed ther formal schoolng, we further restrct the sample to persons 25 to 64 years of age who are not full-tme students at the tme they complete the LFS. Table 1 shows how these restrctons result n basc analyss samples of 10,748 males and 12,418 females. The samples actually used n some analyses are smaller due to tem non-response on partcular covarates. 10 2.2 Defnng our measures of tranng recept As noted n the ntroducton, the lterature on adult educaton and tranng (hereafter we often just call t tranng ) dstngushes publc and prvate tranng for a number of reasons. In Canada, the populatons that receve publc and prvate tranng dffer substantally. Publc and prvate tranng also tend to dffer n ther content and n the nature of ther provders. Thus, we dstngush between publc and prvate tranng n ths study. The AETS dstngushes between employer-supported tranng and nonemployer-supported tranng. The AETS nterprets employer support very broadly, to nclude such thngs as unpad tme off for tranng. In contrast, we feel that someone recevng unpad tme off from hs or her employer to partcpate n a governmentsponsored tranng program should be desgnated as recevng government tranng, rather than prvate tranng. Thus, we adopt an alternatve defnton that (necessarly) reles on the nformaton avalable n the AETS but nstead focuses on who pad for the tranng. In partcular, we defne three mutually exclusve categores: employer or unon fnanced tranng, self-fnanced tranng, and government or other fnanced tranng. The frst category conssts of any tranng pad for, n whole or n part, by an employer or a unon. Ths category domnates the others n the sense that tranng pad for by both an employer and the government, or by the employer and the ndvdual, s counted only n ths category. The second category, self-fnanced tranng, ncludes any tranng pad for solely by the respondent, along wth tranng provded free of charge. Ths category may nclude some tranng where the government subsdzes (n whole or n part) the tranng provder, dependng on whether or not respondents recognze ths subsdy n ther survey response. It may also nclude tranng that receves ndrect subsdes n the form of tax credts, transportaton assstance, chldcare allowances or exempton for job search requrements. The fnal category
s a resdual category that ncludes tranng the respondent reports as exclusvely funded by the government or other sources. The vast majorty of the tranng n the thrd category s reported as funded exclusvely by the government. As already mentoned, the AETS dstngushes between programs and courses based on whether or not the tranng leads, or s ntended to lead, to formal certfcaton. Partcpaton n programs and partcpaton n courses are not mutually exclusve n the AETS data, although the vast majorty of ndvduals partcpate n only one or the other. In addton, they provde some evdence of dfferences n the determnants of partcpaton for programs and courses. However, gven our relatvely small sample szes, and gven that the dstncton between programs and courses reles on judgements by the respondents, we combne the two types of tranng n defnng our measures of treatment. Whle some AETS respondents report recevng multple tranng spells, whether programs or courses, n 1997, the vast majorty of partcpants report only a sngle spell. Thus, n our analyses the treatment varables consst of ndcator (dummy) varables for recept of employer/unon tranng (courses, programs or both), self-fnanced tranng (courses, programs or both) or government/other tranng (courses, programs or both). Table 2 provdes descrptve statstcs for the three treatment measures, as well as the underlyng dstrbutons n terms of courses and programs for each of the three fundng sources. To get a sense of how much tranng the treatment represents, Table 3 presents descrptve statstcs on the dstrbuton of hours wthn reported tranng spells. The top panel ndcates the mean duraton, as well as the 1st, 2nd (medan) and 3rd quartles of the dstrbuton for the combned sample of tranng programs and courses. The nformaton s presented separately for men and women and, wthn these groups, both overall and separately by the fnancng source for the tranng. Four man fndngs emerge from Table 3. Frst, employer-fnanced tranng generally has much shorter duratons than government and self-fnanced tranng. Ths holds for both men and women and for both courses and programs. Second, as expected gven ther defntons, tranng programs tend to have much longer duraton than tranng courses, although the two dstrbutons do have non-trval overlap. Thrd, the mean duratons show a remarkable smlarty between men and women. Fourth, the data reveal a huge amount of heterogenety n the ntensty of the treatment whose mpacts we seek to measure. To take just one example, government-fnanced tranng for men has a mean duraton of 591.5 hours, but the 1st quartle duraton s 90 hours, whle the 3rd quartle s over 1,000 hours. A number of the tranng spells n the data were stll n progress at the tme of the AETS ntervew n 1998. Persons wth a spell of tranng n progress at that tme, whch s when our outcomes are measured, are ncluded n the descrptve statstcs but omtted from the mpact analyses. 2.3 Defnng our outcome varables Studes of the mpact of educaton and tranng typcally focus on ther effect on employment and on earnngs. We care about employment because the employed generally support themselves, rather than relyng on employment nsurance or socal assstance. Thus, gettng people employed represents a goal of many government 11
tranng and assstance programs. At the same tme, for conventonal cost-beneft analyss, earnngs provde a more natural outcome measure. In addton, earnngs reflect dfferences n hours of work and rates of pay between jobs. All else equal, the government (and the tranee!) would prefer that government-fnanced or subsdzed tranng result n full-tme jobs wth hgher rates of pay rather than parttme jobs wth lower rates of pay. We follow the lterature by defnng two outcome measures, one related to employment and one related to earnngs. The frst outcome measure s employment at the tme of the respondents LFS ntervew n 1998. The second s respondents usual weekly earnngs at ther man job as of the LFS ntervew n 1998. Table 4 reports the means (and, for earnngs, the standard errors) of these varables for the full samples of men and women, and condtonal on partcpaton n each type of tranng (employer/unon-fnanced, self-fnanced, and government/other-fnanced) or on recevng no tranng n 1997. As dscussed n detal n Hu and Smth (2003), these outcome varables have the very mportant drawback that they are measured no more than 12 (or 15 n the case of the March 1998 respondents) months after the completon of the tranng whose effect we seek to measure. In some cases, the lag may be only a month or two, or the tranng may not yet even be complete. As a result, the outcomes may not fully pck up the earnngs and employment effects of tranng, partcularly f t takes some tme to fnd a job followng completon of the tranng. Recent evdence from the Calforna GAIN program presented n Hotz, Imbens and Klerman (2000) suggests that government-fnanced human captal acquston may have a payoff that does not fully appear for a couple of years after the completon of tranng. 12
3. Evaluatng the labour market effects of educaton and tranng In ths secton, we lay out a model of labour market outcomes and partcpaton n tranng. We then descrbe the assumptons requred under dfferent econometrc methods of estmatng the mpact of tranng on outcomes. In consderng alternatve evaluaton methods, we are lmted by the fact that the AETS s essentally a cross-sectonal survey. The AETS collects nformaton on each respondent only once. Whle the AETS data contan nformaton on total annual earnngs for the year 1997, whch s the perod durng whch the tranng t measures takes place, ths earnngs measure s not comparable to the weekly labour earnngs measure obtaned n the Labour Force Survey n 1998. Moreover, for most longtudnal estmaton strateges, we want data on the outcome pror to, rather than at the same tme as, the tranng whose mpact we seek to estmate. The lack of precse nformaton on the tmng of tranng durng 1997 further lmts any attempt at usng longtudnal methods. Thus, the data compel us to restrct our analyss to cross-sectonal evaluaton methods. We consder two pars of related methods. The frst par of methods reles on the assumpton that the data contan nformaton on all of the mportant factors affectng both labour market outcomes and partcpaton n tranng. The lterature refers to ths assumpton as selecton on observables. We consder both parametrc regresson and sem-parametrc matchng estmators that buld on ths assumpton. The second par of methods allows for selecton on unobservables. Both methods requre the presence n the data of an nstrument (or excluson restrcton). An nstrument s a varable that affects partcpaton but not outcomes, other than through ts effect on partcpaton. Credble examples of such varables are dffcult to come by n ths context; we examne the performance of multple canddate nstruments n the emprcal work presented n Secton 4. As methods are not our prmary purpose, our dscusson s short and focuses on the man ponts. Further detal on all of the methods we consder appears n Angrst and Krueger (1999) and Heckman, LaLonde and Smth (1999). 3.1 A model of labour market outcomes and partcpaton n tranng The standard human captal earnngs functon (see, for example, Becker, 1964, or Mncer, 1974) forms the bass of the outcome models we use to estmate the mpact of tranng on earnngs and employment usng the AETS. Assumng a lnear functonal form, we have the outcome equaton, Y = β + β X +... + β X + δ T +... + δ T + ε, t 0 1 1t K Kt 1 1t J Jt t 13
wherey t denotes the outcome of nterest for person n perod t (earnngs or employment n our case), X kt, k = 1,..., K denote factors such as years of schoolng and experence, and Tjt, j = 1,.., J are ndcators for recept of dfferent types of tranng. Henceforth, gven that we have only cross-sectonal data, we drop the t subscrpt. For smplcty later on, we defne Y ( 1 = Y1 X, T = 1, ε ) to be the observed outcome wth tranng and Y0 = Y0 ( X, T = 0, ε ) to be the observed outcome wthout tranng. Now consder the partcpaton equaton. For smplcty, assume for the moment only a sngle type of tranng, so that the partcpaton choce conssts of takng tranng or not, and that the tranng s avalable only n a sngle perod. LetY * ( X, T = 1) denote the expected, dscounted present value of earnngs assocated wth tranng. Smlarly, let Y * ( X, T = 0) denote the expected, dscounted present value of earnngs assocated wth not takng tranng. Now let C( W ) denote the expected, dscounted present value of the costs assocated wth takng tranng, where W, whch may nclude elements of X, denotes factors that vary the cost of tranng among persons. Such factors may nclude age, exstng human captal, famly characterstcs, ndustry, occupaton, job tenure, frm sze, regon, and so on. Assumng lnearty, ths gves the tranng cost functon: C = + W + + W + u, γ 0 γ1 1... γ L L wherew 1,..., W L are the ndvdual elements of W. A rsk-neutral ndvdual wll take the tranng f and only f, Y ( X,T = 1)-C(W) > Y(X,T = 0). * * The expected, dscounted gan (or loss) from tranng s gven by H ( X, W) = Y ( X, T = 1) Y ( X, T = 0) C( W). * * * We do not observe H * ( X, W ), because we do not observe the counterfactual expected earnngs wthout tranng for persons who take tranng or the counterfactual expected earnngs wth tranng for persons who do not take tranng. What we do observe n the data s the decson of whether or not to take tranng. We can wrte ths decson n the form of a bnary choce model, T = 1 for persons who take tranng and T = 0 for those who do not: T * 1 f H ( X, W ) > 0; = 0 otherwse. Assumng thaty * ( X, T ) s a lnear functon of X andt (as above), that C( W ) s a lnear functon of W, and that the unobservable components of both have normal dstrbutons centered at zero, yelds a reduced-form probt model of partcpaton. 14
The generalzaton to ndvduals who are not rsk-neutral s straghtforward. Smply change the dscounted earnngs streams above to dscounted utlty streams. Equally straghtforward s the generalzaton to multple types of tranng, so long as we contnue to assume that the tranng takes place n only one perod an assumpton consstent wth our cross-sectonal data (but not, of course, wth realty). In ths case, there are multple possble earnngs, or utlty, streams, wth one for no tranng and one assocated wth each avalable type of tranng. Each ndvdual chooses the tranng acton assocated wth the maxmum of these dscounted values. Now consder some mplcatons of ths smple model of partcpaton and outcomes for the mpact estmaton undertaken n ths model. Ths s a model of ratonal tranng partcpaton. Indvduals partcpate n tranng when they expect, ex ante, that the benefts wll exceed the costs. Ths feature of the model has several mplcatons. Frst, t suggests a strong pror belef that the mpacts of tranng, partcularly of prvate tranng (publcly fnanced tranng s sometmes taken for other reasons), wll have a postve mpact on labour market outcomes. Negatve mpact estmates wll rase suspcon and can be consdered an nformal specfcaton test of sorts. The second mplcaton of ratonal behavour n the context of ths model relates to nstrument selecton. Indvduals decdng whether or not to take tranng are weghtng the costs and benefts of dong so. Good nstruments wll be varables that affect the costs and benefts of takng tranng wthout affectng outcomes n the absence of tranng. Examples of possble nstruments suggested by ths lne of reasonng nclude varables specfcally related to costs, such as dstance to the local tranng centre, and varables related to varaton n the mpacts of tranng. The thrd mplcaton of ratonal behavour relates to heterogeneous mpacts. If some ndvduals gan more (or gan at all) from tranng and others gan less, we would expect that f ndvduals can predct ther gans to some extent, those we observe takng tranng wll have larger mpacts from t than those we do not. Ths has mportant mplcatons for polces that seek to ncrease partcpaton n tranng, as t suggests care n generalzng estmated mpacts of tranng to populatons not presently observed to take t. There may be a reason they are not dong so. Next, consder the ssue of selecton bas n the context of the smple model. Some varables affect both partcpaton n tranng and outcomes n the absence of tranng. If we fal to condton approprately on these varables when estmatng the mpact of tranng, our estmates of the mpact of tranng wll be based as the tranng ndcator wll proxy for the mssng varables that affect both tranng and outcomes. Two standard examples of such varables are ablty and motvaton. Both ablty and motvaton lkely have a postve effect on both earnngs and partcpaton n tranng, whch mples a postve bas n the estmated mpact of tranng f they matter and we fal to condton on them. If we observe the relevant varables that affect both partcpaton n tranng and outcomes n the absence of tranng n our data, then we have what Heckman and Robb (1985) refer to as selecton on observables. In ths case, ncludng these varables approprately usng the methods dscussed n Secton 3.3 wll suffce to solve the selecton problem. If we do not observe the relevant varables, then n terms of the model these unobserved factors result n a correlaton between the error terms n the outcome and partcpaton equatons, so that corr( ε, u ) 0. In that 15
case, we requre methods for selecton on unobservables, whch we dscuss n Secton 3.4. These methods typcally requre an nstrument or an excluson restrcton, whch, n terms of our model, s a varable that belongs n W but not n X. Fnally, consder the relatonshp between ths smple, statc model and the underlyng dynamc process of tranng partcpaton over the lfecycle. As shown n Becker (1964), t makes sense to take tranng when young rather than old, as young people have a longer perod over whch to realze the labour market benefts of ther tranng. Ths dynamc aspect of the tranng partcpaton decson can be captured n the statc model by ncludng age as a determnant of tranng. Another lfecycle ssue concerns repeated partcpaton n tranng. Emprcally, we observe ndvduals takng both publcly fnanced tranng (see, for example, Trott and Baj, 1993, for the U.S.) and prvate tranng (see, e.g., Blundell, Dearden and Meghr, 1996 for the U.K.), more than once. Suppose these repeated nstances of tranng are not ndependent, but nstead are postvely correlated, perhaps due to unobserved dfferences n tastes for tranng. In ths case, the mpact of current tranng we estmate may also nclude the mpact of past tranng epsodes we do not observe. To the extent that tranng has the postve effect suggested by theory, ths would bas our mpact estmates up, f we nterpret them strctly as mpacts of the tranng we observe n the AETS. 3.2 Parameters of nterest In a world n whch ndvduals have heterogeneous mpacts from tranng, t s mportant to consder precsely what the parameter of nterest s n evaluatng the mpact of tranng. 5 To keep the dscusson smple, we agan assume for the moment only a sngle tranng type, wth mpactδ ( = Y1 Y0) for person. The extenson to multple tranng types s straghtforward. We consder three possble parameters of nterest and brefly dscuss the relatonshps among them. The average treatment effect (ATE) s, smultaneously, the effect of tranng on a randomly selected person n the populaton of nterest and the mean effect on all persons n the populaton of nterest. It s defned as ATE = E( δ ). Ths parameter s of nterest n cases where a populaton wll be requred (or nduced) to partcpate n tranng. The most common treatment effect parameter n the lterature s the so-called treatment on the treated ( TT) parameter. Ths parameter measures the mean mpact of tranng on those observed to receve t n the data. In term of our notaton, t s gven by TT = E( δ T = 1) Ths parameter s of nterest f we want to perform a cost-beneft analyss on tranng currently beng receved, whether prvately or publcly funded. 16
The fnal type of parameter of nterest conssts of varous treatment effects measured at the margn. If we have a bnary nstrument, then we can defne local average treatment effects (LATEs), as n Imbens and Angrst (1994). The LATE s the mean effect on those persons who change tranng partcpaton status when the nstrument changes value. It assumes a monotonc response, so that persons do not, for example, become more lkely to partcpate when they move farther away from a tranng centre. Each dfferent nstrument mples ts own LATE, and the LATEs for two dfferent nstruments may dffer substantally dependng on the mpacts realzed by the persons each nstrument nduces to partcpate. If the nstrument s a polcy varable, such as the tuton for the tranng, then the LATE may be of great polcy nterest. If we have a contnuous nstrument, we can defne margnal treatment effects (MTEs) as n Heckman and Vytlacl (2001). The margnal treatment effect they defne s the effect on the person just ndfferent to partcpatng at ther current value of the nstrument. That s, the margnal person s one for whom H * ( X, W ) = 0. Heckman and Vytlacl (2001) show that all of the other common treatment effect parameters can be wrtten as partcular ntegrals of such MTEs. Fnally, we can defne other margnal effects not necessarly related to nstruments. If we buld a new tranng centre n a depressed town, then we can defne the mpact on the persons who choose to partcpate n the presence of the new tranng centre who dd not partcpate before when they had to travel to the next town. Ths treatment effect we refer to as a margnal average treatment effect (MATE). It s not a LATE, because the varaton (the new tranng centre) s not an nstrument, due to the placement of t n a depressed town where, presumably, outcomes n the absence of tranng are lower. From the dscusson, the polcy nterest n partcular MATEs s clear. In ths paper, we look only at the mpact of treatment on the treated for dfferent types of tranng. We do so for several reasons. Frst, although t s the parameter most often examned n the lterature, estmates for both publcly fnanced and prvately fnanced tranng reman somewhat controversal, especally the latter. Second, none of the nstruments we examne arse from varaton n polcy, whch s typcally necessary for LATEs to be of nterest. Fnally, as no one s proposng makng ether publc or prvate tranng mandatory, potental nterest n the ATE parameter n ths context s small. 3.3 Cross-sectonal evaluaton methods that assume selecton on observables In ths secton, we consder two methods based on selecton on observables. That s, both methods assume that we observe n the data all the man factors that affect both partcpaton n tranng and outcomes n the absence of tranng. The most common (and, whch s not unrelated, the smplest) method for evaluatng the mpact of tranng reles on standard regresson methods. For smplcty, we frst consder the case of one tranng type and a common effect of tranng. The extenson to multple tranng types s straghtforward; we dscuss the extenson to heterogeneous tranng mpacts below. Now suppose that X ncludes the standard varables one ncludes n the human captal model, such as prevous schoolng and experence (or ts proxy, age). But suppose that there reman other factors, not ncluded n the standard human captal 17
model but avalable n the data, whch affect both outcomes n the absence of tranng and partcpaton n tranng. Geographc locaton s a potental example here. These latter varables represent a subset of W. We let Z denote the unon of ths subset ofw wth X. Under these assumptons, we have that but E( ε X, T) 0, E( ε Z, T) = 0. (1) Barnow, Can and Goldberger (1980) (hereafter BCG ) frst derved ths motvaton for estmatng the mpact of tranng usng standard regresson methods but wth a rch set of covarates rch enough to make the outcome equaton error term condtonally mean ndependent of tranng. As dscussed n Heckman and Robb (1985) and Heckman and Smth (1996), n a world wth heterogeneous mpacts, the error term n the outcome equaton now mplctly ncludes the person-specfc component of the mpact for persons who receve tranng. That s, the error term ncludes the dfference between the mean mpact of treatment on the treated and the ndvdual tranee s mpact from tranng as well as the unobserved component of the outcome n the absence of tranng. In the BCG set-up, the only major change s n nterpretaton. The coeffcent on the tranng ndcator now just estmates the mean mpact of treatment on the treated; under the common effect world t was also an estmate of the average treatment effect. Lke the standard regresson estmator, matchng assumes selecton on observables. However, rather than assumng a functonal form for the outcome equaton, matchng drectly compares the outcomes of traned and untraned persons wth the same (or smlar) values of those varables thought to nfluence both partcpaton n tranng and outcomes n the absence of tranng. Matchng has two advantages relatve to the regresson estmator just dscussed, and one dsadvantage. The prmary advantage s that t s sem-parametrc. No functonal form assumpton from the outcome equaton s requred to mplement the estmator. In standard regresson analyss, even f you have the correct covarates, you can stll get based estmates f you assume the ncorrect functonal form say by falng to nclude needed hgher order or nteracton terms. The second advantage s that you can match on varables that are correlated wth the error term n the outcome equaton. Ths s the case because matchng only requres that the mean of the error term be the same for tranees and non-tranees wth gven values of the condtonng varables, not that t be zero. In notaton, t requres that E ( ε Z, T 1) = E ( ε Z, T = 0), = but t does not requre, as regresson does, that both terms equal zero. Ths s a weaker assumpton than assumpton (1) above. 6 The dsadvantage of matchng s that, f the lnear functonal form restrcton, mplct n regresson-based analyss, n fact holds n the data, then falng to mpose t reduces the effcency of the estmates. Put dfferently, f the outcome equaton really s lnear, mposng lnearty wll lead to smaller standard errors on the mpact estmates. 18
The condtonal ndependence assumpton (CIA) that justfes matchng s gven by: Y 0 T Z. Ths assumpton mples the balancng condton mentoned n the precedng paragraph. The CIA states that, condtonal on Z, the varables affectng both partcpaton n tranng and outcomes n the absence of tranng, partcpaton n tranng s unrelated to outcomes n the absence of tranng. Put dfferently, whatever selecton nto tranng takes place, wthn groups defned by the same values on all the varables n Z, partcpaton s unrelated to what would happen f the person dd not take tranng. Thus, overall, tranees may have better or worse labour market prospects than non-tranees, but condtonal on Z, ther expected labour market outcomes are equvalent. Two techncal detals deserve note. Frst, ths s the verson of the CIA that justfes usng matchng to estmate the mean mpact of treatment on the treated; a stronger verson, whch also apples toy 1, s requred for estmatng the average treatment effect. Second, the varables n Z may not be factors that can be altered by the tranee n antcpaton of takng tranng (or by a non-tranee n antcpaton of not takng tranng). See Lechner and Mquel (2002) for more on ths latter pont and Heckman, LaLonde and Smth (1999) for more on the former pont. The problem wth matchng drectly on Z s that any set of covarates that plausbly satsfes the CIA s gong to be of relatvely hgh dmenson. Even f all the elements of Z are dscrete, the number of dstnct combnatons becomes large very rapdly, leadng to the problem of empty cells values of Z for whch we observe partcpants but no correspondng non-partcpants to provde the estmated counterfactual. Smply omttng tranees n cells wth no non-tranees s not a very satsfyng soluton. Equally unsatsfyng are the varous ad hoc cell combnaton algorthms used n some of the evaluatons of the Comprehensve Employment and Tranng Admnstraton (CETA) program n the U.S. These evaluatons are surveyed and referenced n Barnow (1987). Ths problem of the potental absence of non-tranees to provde the estmated counterfactual for tranees wth certan values of the condtonng varables s called the support problem. The support s a statstcal term meanng the set of values for whch a densty functon s non-zero; that s, t s the set of values of a varable that you mght observe wth postve probablty. Along wth the CIA, the second man assumpton underlyng matchng s the support condton, gven by Pr ( T = 1 Z ) < 1 for all possble values of Z. Ths condton states that for all values of the condtonng varables, some persons wll not partcpate. Even f ths condton holds n the populaton, t may sometmes fal n fnte samples. A more general verson of the support condton s requred to estmate the average treatment effect. 7 Regresson-based methods, such as the BCG estmator, mplctly solve the support problem through the lnear functonal form assumpton. The functonal form assumpton flls n where the data are absent. Ths fact reveals another advantage of matchng; t hghlghts the support condton and makes t clear whether the results obtaned were generated by the data, or whether the counterfactuals nstead depend heavly on the lnearty assumpton. 19
The lterature has converged on an alternatve soluton to the curse of dmensonalty and the related support problem. Rosenbaum and Rubn (1983) showed that f you can match on Z, that s, f Z satsfy the CIA, then you can also match on P ( Z ) = Pr( T = 1 Z ). Ths quantty s the probablty of partcpaton or propensty score. Ths helps solve the problem because PZ ( ) s a scalar just a real number between zero and one, rather than a vector. The lterature contans a number of dfferent methods for actually mplementng propensty score matches. These nclude nearest neghbour matchng (wth and wthout replacement), cell matchng, kernel matchng and local lnear matchng. These methods are all consstent n the sense that, as the sample sze becomes arbtrarly large, they all gve the same answer because n an arbtrarly large sample, all of them rely only on comparsons of tranees and non-tranees wth equvalent values of PZ ( ). Detaled dscussons of the varous methods can be found n Heckman, Ichmura and Todd (1997), Heckman, LaLonde and Smth (1999) and Smth and Todd (2003). In ths paper, where we are concerned wth substance rather than methods, we confne ourselves to nearest neghbour matchng wth replacement, but vary the number of nearest neghbours. Consder frst just one nearest neghbour. Nearest neghbour matchng wthout replacement goes through the treated (tranee) observatons one by one and, for each one, fnds the non-tranee wth the nearest (n absolute value) estmated propensty score. That non-tranee becomes the nearest neghbour match for the current tranee and may not be matched to any other tranees. Nearest neghbour matchng wth replacement proceeds n the same fashon, but allows a gven non-tranee to be used as the match for more than one tranee. Matchng wth replacement reduces the average dstance (n propensty scores) between each tranee and hs or her matched non-tranee. Ths should reduce bas. The cost s that f some non-tranees are re-used, ths wll ncrease the varance of the resultng estmate. Deheja and Wahba (1999) clearly llustrate the problem wth matchng wthout replacement when the number of comparson non-tranee observatons wth hgh probabltes of tranng s less than the number of tranees wth hgh probabltes of tranng (as t usually s for obvous reasons). In ths case, you get a lot of bad matches. To avod ths, we match wth replacement. The formula for the (sngle) nearest neghbour estmator s gven by n n{ T= 1} n{ T= 0} 1 Y 1 wjy0 j { T 1} = 1 j= 1 = ( ), where the sum s over tranees, the j sum s over the non-tranees, n { T = 1} s the number of tranees, n { T = 0} s the number of non-tranees, and where w j f j = arg mn = 0 otherwse. { P( Z ) P( Z ) }; 1 j The generalzaton to the case of multple nearest neghbours, each recevng equal weght, s straghtforward. Varyng the number of nearest neghbours n the estmaton allows us to trade off between the bas and varance n our estmator. Consder swtchng from usng 20
one nearest neghbour to construct the counterfactual for each observaton to usng two nearest neghbours. The average dstance (n terms of propensty scores) between each tranee and the non-tranees used to construct hs or her estmated counterfactual mean necessarly ncreases. At the same tme, the number of observatons used to construct the counterfactual ncreases, whch reduces the varance of the estmator. The optmal number depends on the densty of non-partcpants. For example, f there are not many more non-partcpants than partcpants, there s lttle gan to usng addtonal neghbours. We experment wth one, two, and fve nearest neghbours n our emprcal work. The norm n the economc lterature that employs matchng s to use bootstrappng methods to estmate the standard errors, for reasons lad out, for example, n Heckman, Ichmura and Todd (1997, 1998). For reasons of tme and of computng convenence, we nstead report here estmates that do not take account of the varance components resultng from the estmaton of the propensty score or from the matchng tself. Thus, our estmated standard errors are lkely downward based. 3.4 Cross-sectonal evaluaton methods that assume selecton on unobservables We consder two (related) methods that attempt to deal wth selecton on unobservables. In ths settng, we beleve that we do not have all of the varables that affect both partcpaton and outcomes n the absence of partcpaton n our data. But, we beleve that we have a varable, an nstrument or excluson restrcton, whch affects partcpaton but does not affect outcomes other than through ts effect on partcpaton. The frst of these estmators s the Heckman (1979) bvarate normal selecton estmator. Ths estmator assumes that the error terms n the partcpaton and outcome equatons have a jont normal dstrbuton, and that the selecton bas results from a non-zero correlaton between the two error terms. When the outcome varable s bnary, ths model corresponds to a bvarate probt model. Techncally, the Heckman (1979) model s dentfed solely based on the jont normalty assumpton, and no excluson restrcton s requred. Extensve experence n the lterature n the form of both tral and error and Monte Carlo studes ndcates that, n practce, an excluson restrcton s requred to ensure the stablty of the model. The lterature also reveals that the performance of the bvarate normal estmator depends crtcally on the valdty of the normalty assumpton. Smulaton results n Heckman, LaLonde and Smth (1999) show that t also depends on havng a strong excluson restrcton that s, that the varable ncluded n the partcpaton equaton but not n the outcome equaton has a substantvely mportant effect on partcpaton. Puhan (2000) summarzes much of the methodologcal lterature on the performance of the bvarate normal estmator. The bvarate normal model s often referred to as the Heckman two-step model, because of the smple two-step estmaton procedure outlned n Heckman (1979). However, estmaton n two-steps s neffcent. As many common software packages (for example, Stata) now nclude routnes to jontly estmate the partcpaton and outcome equatons, t makes sense to do so, and to drop ths name for the estmator. There are both common effect and heterogeneous effect versons of the 21