AN EVALUATION OF TRAINING

Transcription

1 Offce of Evaluaton and Oversght AN EVALUATION OF TRAINING FOR THE UNEMPLOYED IN MEXICO Marcelo Delajara, Samuel Freje, Isdro Soloaga * Workng Paper: OVE/WP-09/06 September, 2006

2

3 Electronc verson: Inter-Amercan Development Bank Washngton, D.C. Offce of Evaluaton and Oversght, OVE An Evaluaton of Tranng for the Unemployed n Mexco Marcelo Delajara, Samuel Freje, Isdro Soloaga * * Ths study was prepared by Marcelo Delajara, Samuel Freje, and Isdro Soloaga from Centro de Investgacón en Economía y Polítcas Públcas (CIEPP). Unversdad de las Amercas, Puebla. Ths workng paper s one of several background reports to OVE s Labour Tranng Thematc Evaluaton carred out durng the Ex-Post Evaluaton Cycle. The team of ths Thematc Study was comprsed of Pablo Ibarraran, Davd Rosas- Shady, and coordnated by Inder J. Ruprah. The fndngs and nterpretatons of the authors do not necessarly represent the vews of the Inter-Amercan Development Bank. The usual dsclamer apples. Correspondence to: Pablo Ibarraran, e-mal: pbarraran@adb.org, Offce of Evaluaton and Oversght, Inter-Amercan Development Bank, Stop B-750, 300 New York Avenue, NW, Washngton, D.C

4

5 TABLE OF CONTENTS INTRODUCTION I. INSTITUTIONAL ANALYSIS... A. Economc and Poltcal Background.... GDP growth, Unemployment n Informalty Reformng the Mexcan Labor Market...4 B. Insttutonal Capacty...7 C. Descrpton of the Program...8 D. Operatve Capacty...0 II. A REVIEW OF PREVIOUS EVALUATIONS...3 III. IV. IMPACT EVALUATION...6 A. Methods of Impact Evaluaton...20 B. Avalable Data...2 C. Results Accordng to non-parametrc method assumng selecton on observables Accordng to parametrc method assumng selecton on unobservables Comparson of results between methods Results by program modalty and populaton group...30 A COST BENEFIT ANALYSIS...32 V. CONCLUSIONS...34 REFERENCES

6 ANNEXES ANNEX A: ANNEX B: ANNEX C: ANNEX : ANNEX 2: ANNEX 3 ANNEX 4 ANNEX 5 ANNEX 6 TABLES GRAPHS METHODOLOGICAL ANNEX PROBABILITY MODEL FOR TAKING A TRAINING COURSE PROBIT MODEL FOR TAKING THE PROGRAM, SELECTED OBSERVATIONS, YEAR 2002

7 INTRODUCTION Ths paper summarzes the fndngs of an mpact evaluaton of the Mexcan tranng programs PROBECAT_SICAT for the perod It s a study commssoned by the Offce of Evaluaton and Oversght of the Inter-Amercan Development Bank n accordance to the Bank s polcy of ex-post evaluaton of operatons. The paper has fve addtonal sectons. Secton 2 provdes an account of the Mexcan publc polcy towards the unemployed and a descrpton of the nsttutonal and operatve capacty of ths polcy for the last two decades. In ths secton we explan the evoluton of pro-cyclcal behavor of unemployment and nformal employment wth respect to economc growth n Mexco. Moreover, we descrbe the orgns and organzaton of labor publc polcy n Mexco and descrbe the evoluton of the tranng for the unemployed programs PROBECAT/SICAT. Secton three revews the prevous mpact evaluatons of the tranng programs for the unemployed. Secton four ncludes a dscusson of the methods we use for program evaluaton and our man results. Secton fve, presents a succnt cost-beneft analyss of the program, makng use of the results of the mpact evaluaton. Fnally, secton sx concludes wth a summary of results.

8

9 I. INSTITUTIONAL ANALYSIS A. Economc and Poltcal Background For a few years, before 982, the Mexcan economy and federal government benefted from hgh ol prces and the dscovery of new ol reserves. The mprovement n the economy was consdered permanent by the government and thus run large defcts fnanced by foregn savngs. Defcts were further encouraged by the low nterest rates and abundance of captal n nternatonal fnancal markets. By there s a strkng, and for the government, unexpected reversal of the nternatonal prces: there s a sharp fall n ol prces and a sharp ncrease n nternatonal nterest rate. The mpact on the Mexcan publc defct s large. Mexco has at the tme a fxed exchange rate system, whch s hghly nconsstent wth large publc defcts. Intally the government tres to sort the crss out by natonalzng the bankng system; ths, however, dd not avod a terrble devaluaton of the Mexcan peso, and the captal flghts typcally assocated wth devaluaton. Durng the 80s Mexco was gong to have ts poorest performance of the 20 th century n terms of economc growth. But the crss dd somethng good to the Mexcan publc polcy: t changes t permanently for the better. Mexcan economc polcy would rely from that tme untl the present on four basc tenets - less government nterventon n the economy, openness to nternatonal trade, a flexble exchange regme, and macroeconomc stablty, as the mean to recover nternatonal credblty and economc growth. The process by whch ths polcy stance was adopted was gradual and panful. The frst years were not easy at all for the populaton. The Consttuton had to be modfed n order to be able to adopt the new polcy stance. A way to generate consensus was to acheve poltcal deals wth the opposton poltcal and socal forces, the so-called pactos. The Pacto de Soldardad Económca of December 987 and the Pacto para la Establdad y el Crecmento Económco, of January 989 are two good examples. These pactos were ratfed many tmes durng the followng decade. Despte the deep reforms of the prevous decade, Mexco s macroeconomc performance could not be spared the shocks that ht many emergng markets economes n the second part of the 990s. Actually the Mexcan crss of December 994 was the frst n a seres that shocked the world economy for more Artcles 25 and 26, whch stablsh the degree of government nterventon n the economy and the role of the federal government n Mexco s development strategy.

10 than seven years. Perhaps the Mexcan crss was the least predctable of all them, because the economy s fundamentals seemed sound. A deep recesson followed n 995 from whch the country recovered however very fast n 996. GDP growth rates went back to the normal level of 4-6% between 996 and From 200 to 2005 economc growth slowed down consderably, however. The GDP growth rate halved. Inflaton has been declnng snce then, as the monetary polcy has become more predctable, and t s fully under control now. By the end of the perod the bggest problem conssts of reganng the GDP growth rate of 4-6 %.. GDP growth, Unemployment n Informalty Despte de nflexblty of labor contractual arrangements n Mexco, the unemployment rate s very senstve to changes n the level of economc actvty, as the followng graphs show. Graph shows the more recent evoluton of the unemployment rate and the (Log) GDP of Mexco between 996 and Between 996 and 2000 the unemployment rate and GDP moved n opposte drectons, as expected. After 2000, growth slowed down consderably and unemployment rate started to grow agan. Therefore, t seems crucal for the level of employment that the economy grows steadly. Graph 2 shows the annual rate of growth of GDP together wth the absolute annual change n the unemployment rate for the perod 980 and The correlaton coeffcent between both seres s -0.7, very sgnfcant. Although unemployment remans low on average, t changed qute dramatcally n response to changes n the level of economc actvty. Graph 3 shows the schoolng attanment of the Mexcan labor force. In 200, one quarter of workng-age populaton was llterate or had dropped-out from elementary school; half of the workng-age populaton had elementary or junor hgh educaton; the other quarter had senor hgh school or college level educaton. In partcular about 40 % of the workng-age populaton had only elementary educaton or had dropped-out from elementary school. IN contrast, an ncreasng share of the unemployed has attaned senor hgh school and unversty degrees. Only about a quarter of the unemployed have elementary or lower educaton, whle these groups represent 45 % of the workng-age populaton. So the groups wth senor hgh school and unversty degrees are the ones more represented among the unemployed (see Graph 4). Regardng the age dstrbuton of the unemployed for the perod , the age groups n the ranges of 2-9, 20-24, and 45 years-old or older have ncreased ther partcpaton n about, 4 and 2 percentage ponts, respectvely. 2

11 These groups represent 60% of the unemployed n The groups n age groups and have reduced ther partcpaton n about 3.5 percentage ponts each. So, most of job seekers are ether very young or already mature workers. An dea of the dynamcs of the labor market can be ganed from the study of the reasons for whch workers lose ther jobs. We fnd that durng the economc expanson of about 40% of the unemployed left ther jobs voluntarly n order to fnd a better one elsewhere; about 30 % were dsmssed and another 20 % had ther temporary jobs termnated. As economc growth slowed down n the early 2000s, the share of unemployed who had voluntarly qutted halved, but the share of those who had been dsmssed or had ther temporary job termnated remaned stable. The duraton of unemployment has remaned stable for the perod About 60 % of job seekers reman unemployed for less than 4 weeks; 25 % have to wat between and two months to fnd a job; the rest have to wat more than two months. The prevous analyss suggests that unemployment n tself does not seem to be a bg problem n Mexco, at least no more than t s a problem n the rest of modern economes. The Mexcan economy can grow and create employment opportuntes; a hgh share of workers change jobs voluntarly durng an expanson of economc actvty; most do not remaned unemployed for more than one or two months. Mexcan labor markets suffer, however, from nformalty and the lack of employment opportuntes for the youth. A large share of the labor force works n the nformal sector; the swngs n the unemployment rate are assocated wth changes n the sze of nformal employment; most of job seekers are 25 years old or younger. Ths suggests that the man problem n the Mexcan labor market s the problem to create formal jobs for the youth. Informalty n labor relatons and arrangements affects between 40 % and 60% of the labor force employed n Mexco, dependng on the way we measure nformalty. Economsts tend to thnk that nformalty has to do wth llconceved frm and labor legslaton regardng the regulaton and taxaton of economc actvty that ultmately hurts small and medum sze frms and ther workers welfare. 2 Graph 5 shows the recent evoluton of nformalty as measured by the share of the employed labor force not regstered at the IMSS (Insttuto Mexcano de Seguro Socal). The graph reveals a small declne n 2 Others thnk the nformal sector s the consequence of lack of growth and supportve socal polces. For more on nformal labor markets n Latn Amerca see, just two among the myrad of references, Loayza (997) and IBERGOP (2005) 3

12 nformalty from 65 % to 60% of employment between 996 and 2003 for all economc sectors. Informalty seems to declne wth expanson n economc actvty. Graph 6 shows the relaton between the evoluton of the GDP and the evoluton of nformalty. The Mexcan economy grew about 22 % between 996 and 2000 and nformalty fell to 6% from 65 % n 996. So there s a postve effect of economc growth on nformalty but ths effect s small. Graph 7 graph shows that nformalty and unemployment dynamcs are closely related to each other as well. The dynamcs of nformalty are smlar to those of the unemployment rate. Informalty declnes wth employment. Snce the unemployment rate s small, however, there s an obvous lmt to use employment polces to curb the huge nformalty we observe n Mexcan labor market. In short, nformalty may nvolve between 40% and 60% of the Mexcan labor force. When the economy s boomng and the unemployment rate decreases nformalty also decreases, but the change n nformalty s very small. PROBECAT/SICAT by-laws requre that frms nvolved n the mxed modalty hre at last 70%-80% of workers at the end of the tranng perod. Snce SNE montors and enforces ths requrement, partcpatng frms are most lkely to belong to the formal sector of the economy. Therefore, frms n the nformal sector are very unlkely to partcpate n the program. 2. Reformng the Mexcan Labor Market Reformng the Mexcan labor market nsttutons and laws has been probably the most dffcult part of the pro-market reform process started by the Mexcan federal government n the md 980s. The current labor legslaton has been ncapable of fosterng labor productvty and the creaton of enough formal employment opportuntes. Among the obstacles to acheve these objectves are: the hgh costs of takng n and frng employees; a pro-worker paternalst legal framework; lack of alternatve wage settng mechansms, n partcular mechansms that take nto account productvty gans; and excessve nterventon of labor unons n wage settng mechansm, labor contracts, and frms decsons regardng the role of human resources n producton. The Mexcan labor unons, employers, and government representatves have ndeed been dscussng the need for a reform snce the begnnng of the 990s; the dscussons and negotatons have acheved, however, a small progress so far. 4

13 In 995 a formal negotaton started between CTM (Confederacón de Trabajadores Mexcanos) and COPARMEX (Confederacón Patronal de la Repúblca Mexcana) under the arbtrage of STPS (Secretaría de Trabajo y Prevsón Socal). The parts nvolved recognzed the need to reform labor markets and labor relatons n order to acheve full employment and promote workers productvty and frms compettveness, and thus establshed a Comsón Central and a Comté Técnco that would nsttutonalzed these negotatons and make sure that they yeld tangble outcomes. After a decade, however, nothng lke a plan for a labor reform had emerged from these negotatons. Not only the Mexcan labor legslaton had become ncreasngly outdated relatve to those of other Latn Amercan countres, but also t had contrbuted to the growth of nformal employment, manly through of ts narrow nterpretaton of job protecton and stablty. Wth the am of producng a breakthrough n the negotatons the Secretary of Labor establshed a Roundtable on the Labor Reform (Mesa Central de Decsón sobre la Reforma Laboral) wth representatves from workers and employers n 200. The Secretary of Labor establshed the Roundtable wth the objectve of fosterng the dscusson of keys aspects of the reform: ) new contractual arrangements, that would allow for temporary postons and tranng; 2) hours worked, the number of whch would result from a negotaton between the frm and each worker ndvdually 3) ncentves, that allow frms to promote labor productvty gans. Well before the frst year of talks had ended the workers representatves abandoned the negotatons, and unlaterally announced a proposal, whch they sought to get approved at the Congress wth the support of some poltcans. Ths ntatve faled and workers representatves eventually resumed ther dscusson actvtes at the table. By the end of year 2002 the groups represented reached an agreement that the poltcal partes PRI, PAN and PVEM took to Congress for ts dscusson. The agreement, whch contaned proposals for a modfcaton of the Federal Labor Law (Ley Federal de Trabajo) but left all consttutonal provsons (n partcular, artcle 23) unchanged, ncluded provsons for temporary labor contracts, workers choce of labor unon, and specal consderatons to labor relatons n Medum and Small Frms. Unfortunately the agreement dd not reach the stage of a reform proposal due to the opposton of other poltcal partes wth seats at the Congress, and efforts to acheve a reform have been abandoned snce then. Meanwhle, the Natonal Labor Polcy Program for (Programa Naconal de Polítca Laboral ) has establshed fve basc prncples 5

14 that would gude labor polces makng: All nterest groups should be ncluded n the dscusson of labor polces; reform should be gradual so that workers and frms can adjust to the changes; reform or nnovaton should be acheved through a consensus; actons by nterest groups should be taken wthn the lmts of the law; and labor relatons should be framed wthn a peaceful envronment. Gradualsm, consensus, and ncluson of all nterest groups may ndeed produce a peaceful and cooperatve envronment n whch reforms and nnovatons can be dscussed wthout jeopardzng labor relatons; but t must be recognzed that these labor polcy gudelnes mght also delay the reform ndefntely. The Natonal Labor Polcy Program for has also set fve objectves for ther labor polces and fve strateges through whch these objectves would be acheved. The fve declared objectves are:. To establsh a labor culture n socety; 2. To reform labor and other regulatory laws; 3. To modernze labor market nsttutons and government s role wthn them; 4. To modernze and democratze labor unons; 5. To help workers fnd a response to the challenges of globalzaton. The fve declared strateges to acheve them are: I. Promoton of formal employment; II. Tranng and retranng of the labor force; III. Growth of frms and workers productvty; IV. Increase n the compettveness of the Mexcan economy; V.Increase n the welfare of workers and ther famles. From our pont of vew, ths Natonal Labor Polcy Program retans the paralyzng confuson between objectves and strateges that characterzed prevous natonal programs. From our perspectve, of the fve objectves lsted above, tems 2, 3 and 4 should be vewed as strateges rather than objectves; and of the fve strateges lsted above, perhaps only tem II should be vewed as a strategy, the rest should clearly be vewed as objectves. Thus, the establshment of a labor culture; the promoton of formal employment; the growth of frms productvty and compettveness; and the mprovement n the lvng standard of workers and ther famles should be the objectves of the labor polcy. The strateges to acheve them should be the reform of labor and other regulatory laws; the modernzaton and lberalzaton of labor and other market nsttutons; and the modernzaton and democratzaton of labor unons; and probably the desgn of new taxaton schemes for busness. It s not clear at all whether the tranng and retranng of workers should be an objectve or a strategy. In any case we beleve t could be an strategy whose scope, tmng, and fnancng should be entrely left to frms themselves, rather than a strategy orchestrated at large scale by the government as a surrogate for a better educaton system and labor market reform. 6

15 B. Insttutonal Capacty Snce at least the md 970s Mexco s federal government has followed actve labor market polces and has consstently bult nsttutonal capacty to mplement those polces. The Servco Naconal de Empleo, Capactacón y Adestramento (SNE) s establshed n 978 as part of a reform to the Federal Labor Law (Ley Federal del Trabajo). Its man objectves were to mprove the matchng between job seekers and potental employers, to mprove the chances of the unemployed of fndng a job, and to study the labor market n order to mprove labor market polces. Durng the years, whch followed the soveregn debt, crss of 982 workers saw ther real wages declned sharply due to the hgher nflaton rate and the fall n the demand for labor. Informal labor started to grow fast. In order to curb nformalty and mprove the matchng between job seekers and vacances the government adopted an even more actve labor market polcy stance, whch conssted of a strengthenng of the SNE and the SNE polces and resources. In 984 Probecat, the tranng program for the unemployed s started and the SNE s the nsttuton chosen to mplement t. Ths was a logcal consequence of the 978 Federal Labor Law, whch establshed workers tranng as an oblgaton for frms whle smultaneously establshed the SNE. In 988 ths polcy s further strengthen wth the launchng of the Programa de Caldad Integral para la Modernzacón (CIMO) whch provded tranng to employed workers n ther own -small and medum szed- frms. Further nnovaton to the polcy were ntroduced n 993 wth the launchng of the Sstema Normalzado y de Certfcacón de Comptencas Laborales whch sought to clearly establsh the competences of workers so that the tranng programs Probecat and CIMO could focus more effcently on the abltes and knowledge that frms demanded from the workers. The SNE s n charge of CIMO and t plays an mportant role n defnng workers competences. The SNE decdes the way these programs are gong to be mplemented, the federal government sets the normatve framework and provdes the resources; the programs are then mplemented n each Mexcan state by the Servcos Estatales de Empleo (SEE), wth the help of addtonal state funds. The scope of actvtes and processes that the SNE must mplement and montor has thus grown consderably; SNE s nfrastructure and resources have grown as well. The SNE started wth a headquarters n Mexco Cty and only 5 branches across Mexco n 978, by 2002 the number of offces has reached 39. Ths 7

16 admnstratve organzaton s addtonally supported by 77 unts run by the SEE. About 200 employees run the whole system, 920 at the Federal level and about 80 at the state level. Its budget has been growng as well. In 2002 the SNE spent 0 mllon pesos n programs assocated wth the matchng of job seekers and potental employers and other actvtes of ntermedaton between workers and frms. It also spent more than 700 mllon pesos n the mplementaton of Probecat. The SNE runs all the dfferent types of tranng avalable for the unemployed through Probecat: the one offered through regular courses at techncal and other schools (school tranng, now dscontnued); the tranng offered at the frm (frm tranng, also called mx tranng); the tranng amng at helpng those selfemployed (self-employed tranng); the tranng for those nvolved n local ntatves of employment (local employment tranng) We conclude that the nsttutonal capacty to mplement Probecat has, at least formally, been consstently bult and sustaned over the years. The queston remans to whether a publc nsttuton lke SNE, wth a country- and economywde scale of operatons s effcent at all. In partcular, takng nto account the mandatory nature of the tranng at frms and the need to regulate t and montor t, t s dffcult to determne whether the growth of SNE s nsttutonal capacty s just nertal and a by-product of the mandate to tran workers or t s the result of a carefully planned strategy. C. Descrpton of the Program Probecat, an abbrevaton for Programa de Becas de Capactacón para Trabajadores Desempleados, was launched n 984 wth the objectve of provdng assstance and tranng to the unemployed. In 200 ts name was changed to SICAT (Sstema de Capactacón para el Trabajo) and more recently changed agan to Bécate (Becas a la Capactacón para el Trabajo). The benefcares of the program receve a scholarshp equvalent to a mnmum salary whle they take part n a three-month tranng course; about 4.75 mllon workers have been traned between 984 and Graph 8 shows the evoluton of the number of partcpants or tranees. In the frst 0 years of the program, 7 thousand workers were traned on average every year. The scale of operatons ncreased dramatcally after 994; from 995 to 2000, 530 thousand workers were traned on average every year. Durng the years 999 and 2000, nearly 20% of unemployed workers receved tranng n 8

17 ths program. The numbers of tranees has decreased steadly snce then and the fgures for 2005 are smlar to those of the pre-994 perod. The SNE (Sstema Naconal de Empleo) s the nsttuton n charge of organzng and mplementng the program wth the ad of the regonal offces of SEE (Servcos Estatales de Empleo). Whle the SEE decde the type of tranng actvtes to be offered as well as the capabltes and abltes that the tranees should developed durng ther tranng, the SNE s n charge of provdng the fundng for these actvtes. Fundng channeled by the SNE covers the workers scholarshps and all the costs assocated wth the tranng actvtes. The total amount of resources allocated to the program s shown n Graph 9. The evoluton of resources allocated has had an evoluton smlar to that of the number of tranees, but the real expendture per tranee has a negatve trend, as Graph 0 shows. In the begnnng Probecat offered just one type of tranng program called escolarzada ; that s school-based tranng. The tranng conssted bascally on spendng the three months of tranng attendng classes at a publc school sometmes the SEE would also hre ONGs to provde ths type of tranng. When the tranng was over, workers would look for a job usng the placement servces avalable at the SNE and the SEE. A few years later an on-the-job tranng modalty was ntroduced. Ths type of tranng the so-called mxta (that s mxed ) conssted on tranng done at the frm plant or workshop. The SNE pad for the workers scholarshps; the SEE pad for the operatve costs and the frm fnanced the tranng tself. After the tranng, 70 % or so of the tranees would be hred by the frm, the rest would try to fnd a job through the SNE placement offces. There s a large dfference between both types of tranng actvtes; whle the escolarzada offered a general type of educaton, the mxta offered a specfc type of tranng. It s not clear whether unemployed workers could choose between one of these two actvtes or f they were just assgned to them by SEE clerks. There s some evdence, however, that the SEE dstngushed between workers wth and wthout prevous experence, between qualfed and unqualfed workers, and between temporary unemployed workers and self-employed nformal workers. The escolarzada type of tranng was domnant untl 998 when the mxta started to receve a larger share of the tranees. In 994, other types of tranng were also establshed; the most mportant of them beng the so-called tranng for 9

18 the self-employed ( autoempleo ). Therefore, after 994 the share of tranees allocated to the escolarzada type of tranng started to declne; the program was termnated n 200. Snce 2002 the mxta and autoempleo types domnate the tranng actvtes accountng for about 60 % and 30 % of the tranees, respectvely (see Graph ). For the perod , 45 % of tranees n the mxta type receved tranng at medum and large frms and 55 % n small frms. As t was explaned n the former secton, the SNE msson s twofold: to manage Probecat and to serve as placement offce for the unemployed. The tranng effort done n 999 and 2000 s mpressve but ths was acheved at the expense of placement effcacy. Placement effcacy, measured by the rato of vacances flled by the SNE wth unemployed workers to the number of unemployed workers traned by the SNE through Probecat, declnes from 997 to 999 when t reaches ts lowest value. As tranng effort decreases after 2000 placement effcacy starts to ncrease agan. After 2002, both SNE s placement effcacy and tranng effort show a negatve trend, however. D. Operatve Capacty In several offcal documents we fnd that the purpose of Probecat, Mexco s tranng program for the unemployed, was to mprove the matchng between the supplers of labor and ther potental employers, to ncrease the employment probabltes and future wages of the unemployed and to mprove the productvty and compettveness of frms. Thus neffcent matchng, hgh unemployment, nformalty, low wages and low productvty were consdered a consequence of the low level of human captal n the Mexcan labor force. From these offcal documents t s clear that the program s targeted populatons were those characterzed by low levels of schoolng, low wages, hgh unemployment, low share of qualfed labor, hgh level of nformalty n the labor markets, and that Mexcan states where the labor market ndcators looked worse would requred, all else constant, relatvely more resources. We thus conjecture that n order for Probecat to acheve ts objectves the resources allocated to Servcos Estatales de Empleo (SEE) n each state should be n prncple hgher the worse the stuaton of the labor markets there. By operatve capacty of the SEE we mean the avalablty of enough resources to acheve the objectves of Probecat. That s, to reverse a partcularly troublesome combnaton of low educaton, hgh unemployment and low ncome among the labor force -n partcular among the unemployed. The operatve capacty of the 0

19 SEE should be hgher, that s, more resources should be allocated to t, the worse the stuaton n the state n terms of the ndcators mentoned above. In ths secton we explore whether the states SEE were granted the operatve capacty that they needed by analyzng the correlaton between the resources they receved per tranee and ndcators of the labor force and labor market. As we shall see from ths analyss, the budgets allocated to the SEE were unrelated or even worse, negatvely related to the degree of deteroraton of the labor force and labor markets n the state. For nstance, was the operatve capacty assocated wth the level of unemployment across states? The budget allocated per tranee to each SSE seems at best weakly assocated wth the unemployment rate. Graph 3 shows that spendng per tranee ncreased wth the rate of unemployment across states n every year, but the correlaton was neglgble or very weak. The correlaton mproved, however, n the years Regardng the average years of educaton of the workng-age populaton, Graph 4 shows that they are not assocated wth the budget per tranee allocated to each state n years , but are postvely though weakly assocated wth those resources n the perod That s, contrary to the objectves of the program, states wth hgher average years of educaton receved more resources per tranee than states wth lower educated labor force. Summarzng, operatve capacty allocated to the SEE has been ether unrelated or negatvely related wth the sze of ther needs. 3 If the degree of backwardness n the labor markets facng the SEE was not the crteron followed to allocate the resources, what does explan ther dstrbuton across states? Graph 5 shows for the years that spendng per tranee has been drven bascally by spendng per tranng actvty, that s, per tranng course; from whch t follows that the average number of tranees takng each course has vared wdely across states. The average spendng per tranng actvty ranged from 30 thousand pesos to 65 thousand n 998 and 2000, and from 40 thousand to more than hundred thousand n 200. Ths means that spendng per tranee reflects dfferences n the qualty of tranng courses offered across states and, as mentoned above, ths qualty n unrelated to the degree of backwardness of labor markets facng the SEE. 3 Smlar conclusons were found when relatng spendng to wages, nformal employment and share of sklled worker per state.

20 The operatve capacty could also be measured n terms of the total number of courses offered. As Graph 6 shows, ths number ncreases wth the total budget allocated to each SEE across states. The large heterogenety across states n the total, per tranee and per course budgets and the way they related, or rather unrelated, to the degree of backwardness observed n the states labor markets allow us to reconstruct or hypothesze the way n whch the operatve capacty of the SEE was determned. Of course, a man reason behnd the change n the sgn of the relaton between resources spent per tranee and the labor market condtons across Mexcan states after 200 s that some types of tranng have been dscontnued. In 200 the tranng through formal schoolng was termnated and the so-called mxed was encouraged. The mxed tranng conssts on tranng done at frms wth the objectve of tranng the unemployed n the specfc abltes and knowledge useful to the frm. As the resources for Probecat were channeled to that purpose the relaton across states between the operatve capacty of the SEE and the degree of backwardness n the labor markets and labor force weakened even more, as states wth better labor market ndcators (that s, where most of the frms are located) receved a larger budget per tranee and per course. 2

21 II. A REVIEW OF PREVIOUS EVALUATIONS There has been a seres of mpact evaluatons of the Mexcan tranng program for the unemployed. Each study adopts dfferent methods, databases and evaluates dfferent outcomes. In ths revew, we emphasze only those ssues that are comparable to our study. The frst analyss s by Revenga, Rboud and Tan (994) who make use a retrospectve database for benefcares of the 992 cohort. They estmate a probt model n whch the probablty of employment three months after tranng depends on age, educaton, experence, unemployment duraton, seasonal dummes and an ndcator varable of program partcpaton. 4 The authors fnd that partcpants have an 8-percentage ponts hgher probablty of fndng a job than non-partcpants. Besdes, they estmate an earnngs equaton corrected for selectvty and fnd that monthly earnngs of male tranees are around 7% hgher than male non-tranees, but not sgnfcantly dfferent for females. 5 The Mexcan Mnstry of Labor fnshes a smlar study shortly after Revenga s. STPS (995) makes use of a smlar database but for the 993 cohort. They also estmate earnngs equatons corrected for selectvty and fnd postve effects of around 200 pesos a month for males, but no effect for females, wth large benefts for those wth experence and takng on-the-job tranng. They also fnd a postve mpact on the probablty of fndng employment of around 20 percentage ponts, both for males and females, for those takng on-the-job tranng. 6 Fve years later, Wodon and Mnowa (999) crtcze the prevous studes on several grounds. They notce that usng as controls a sample from ENEU wth hgh probablty of partcpatng n the program nduces contamnaton bas: that s, there may be observatons n the control group that actually took the tranng. Also, the earnngs equatons correct for selectvty n takng the program but not for selectvty n partcpatng n the labor market. Wodon and Mnowa address these two ssues by estmatng a probablty model of partcpatng on one of the two modaltes of tranng (.e., on-the-job and school-based) usng the ENEU 4 Ths probt model has a selectvty correcton not fully explaned n the text. See Revenga, Rboud and Tan (994), pages The earnngs equaton n ths case has experence, educaton and ts nteractons as explanatory varables. The program partcpaton equaton, not shown n the paper, controls for martal status, number of chldren, educaton and duraton of unemployment. 6 See STPS tables V.7 and V.0bs. The employment effects, n ths case, were derved from a Cox hazard duraton model. 3

22 and the ENCOPE surveys for the 993 cohort. 7 Then they use the ftted ndex (not the ftted probablty) as an nstrument for program partcpaton n a duraton model and an earnngs equaton corrected by labor market partcpaton. Ther program partcpaton models have an explct excluson restrcton: number of program partcpants as a proporton of state populaton. They fnd a negatve effect on wages for men who had school-based tranng, and no effect on women or other modalty. They also fnd a postve effect on employment for women who had school based tranng. 8 More recently, Calderón and Trejo (200) also make use of the data for the 993 cohort for a study that, for the frst tme, adopts propensty score matchng for the evaluaton. 9 The authors compute dfference-n-dfference for wages before and after tranng between controls and treatments selected accordng to a sort of nearest neghbor matchng. They fnd that the program had a negatve effect on hourly wages for men under every modalty (around 35 cents/hour, that s less than 0%) and a postve effect for women under some modaltes (smlar sze). They are also the frst to estmate a model that assumes selecton on unobservables, followng the procedure proposed by Heckman, Tobas and Vytlacl (2003). Wth ths methodology they fnd a larger negatve effect on wages of 24%. Fnally, Navarro-Lozano (2002) uses the same 993 cohort data and explores the Heckman et al. (2003, 200) methods further. Ths author s the only one that contrasts dfferent methods and parameters of nterest. He compares the estmates of the treatment effect on the treated (ATT) from a non-parametrc estmaton usng propensty score matchng to a parametrc estmaton usng selecton correcton methods. However, only wage effects for males are gauged n ths study. He fnds a postve wage effect of 0% when usng the selecton correcton methods but a negatve effect of 5% when usng matchng. In addton, Navarro-Lozano estmates the margnal treatment effect (MTE) and fnds t ndcates a postve selecton (that s, those who beneft the most from the program are more lkely to partcpate n t). The most recent study s by Calderon-Madrd (2005) and s the only one that makes use of a more recent database. He computes the mpact of the program on the probablty of employment transtons (from unemployment to formal and 7 We also make use of these samples, but for more recent years. A thorough explanaton of these samples s n secton 0 8 No actual sze of the effects n pesos or percentage ponts was provded n ths paper. 9 There s the study by Aportela (999) but t only estmates the mpact on unemployment duraton. Snce we are nterested n comparng results n terms of probablty of employment and wages, we do not comment ths report. 4

23 nformal employment) as well as on wages makng use of data for year He fnds that the benefcares of the SICAT program have hgher probabltes of fndng formal employment but lower probabltes of fndng an nformal job than comparable control ndvduals. On the other hand, he fnds no robust evdence of a postve mpact on wages. Makng use of several matchng procedures as well as panel and cross sectonal data, he fnds ether no sgnfcant effect or sgnfcant effects that dffer by method of estmaton. Ths lterature revew has two common strands. Frst, all the studes (wth the excepton of Calderón-Madrd, 2005) make use of a database more than ten years old and make use of a sngle year database. Second, results depend crtcally on methods used. Thrd, most studes, wth the excepton of Navarro-Lozano (2002) only measure the effect wth the parameter known as ATT, that s average treatment effect on the treated. Our study ams at releasng the evaluaton of PROBECAT-SICAT from these constrants. We make use of several databases spannng a fve-year perod ( ), so a story of the evoluton of the program mpact can be obtaned. Besdes, we adopt two dfferent methods of mpact evaluaton and compute several parameters of nterest, whch allows the study to report not only the robustness of average effects by dfferent methods, but also to descrbe the selecton mechansms that underle the program. As wll be explaned later, our methods allow us to dscuss the exstence of hdden bas n the estmates. Fnally, we wll report both the average treatment effect (ATE) and the average treatment effect on the treated (ATT), whch allow us to dscuss the selecton mechansm of the program and nfer whether the program attracts ndvduals that beneft the most from t. Gven the debate sparked n the lterature on the methods for program evaluaton, and the complant of several authors on the methodologcal ambguty of some studes, we proceed wth a detaled explanaton of the methods used n ths study, and ts justfcaton. Furthermore, we add a METHODOLOGICAL ANNEX, where a detaled explanaton of the procedures adopted s provded. 5

24 III. IMPACT EVALUATION The outcome of a program or polcy nterventon upon ndvduals from a gven populaton of sze n, can be denoted by the varables: j Y j j = 0, =,2,..., n Here, Y stands for the outcome of a varable of nterest for ndvdual under treatment (j=) and not under treatment (j=0). However, ndvduals are not observed wth and wthout treatment. Actually, ndvduals are ether under treatment or not. The selecton nto the treatment can follow many dverse mechansms (e.g. randomzaton, selecton desgn, self-selecton, etc). Hence, there s an ndcator varable (D) that sgnals whether the observaton corresponds to an ndvdual under treatment or not: 0 D = f f Y = Y Y = Y The evaluaton of the mpact of a program or polcy nterventon can be measured n dfferent ways. We could be nterested n knowng whether the dfference between the outcome under treatment and the outcome wthout 0 treatment s postve for every ndvdual (formally, Y > 0 ). Alternatvely, we could be nterested n comparng the dstrbuton of these two varables or some statstcal moment of ther dfference. The most common measure n program evaluatons s the average dfference n the varable of nterest for the ndvduals wth and wthout the program. Ths s known as the average treatment effect (ATE): ATE = E Y 0 0 [ Y ] Another popular measure s the average treatment effect restrctng the populaton to only those who were subject to the polcy nterventon under analyss. Ths s called the average treatment effect on the treated (ATT) 0 : 0 [ Y = ] ATT = E Y D Y 0 There are other parameters of nterest for program evaluaton such as the margnal treatment effect and the local treatment effect. See Heckman, Tobas and Vytlacl (200) or Wooldrdge (2002). 6

25 The comparson between the ATE and the ATT provdes nterestng nformaton on the selecton mechansm of the program. Notce that the ATE can be also defned as the weghted average of the effect on the treated and the effect on the untreated: [ Y Y D] = P( D = ) E[ Y Y D = ] + P( D = 0) E[ Y Y = 0] ATE = ED D The dfference between ATT and ATE s: ATT ATE = 0 0 [ P( D = ) ] E[ Y Y D = ] P( D = 0) E[ Y Y D = 0] = P Hence, snce P D = 0, 0 0 ( D = 0) { E[ Y Y D = ] E[ Y Y D = 0] } ( ) [ Y D = ] > E[ Y Y = 0] ATT > ATE E Y D In other words, f the ATT s hgher than the ATE that means that the mpact among the treated s hgher than the mpact among the untreated, whch can be called postve selecton because the program s attractng those who beneft more from t. In the opposte case,.e. ATT lower than ATE, we have negatve selecton because the program would have had a larger mpact among the untreated than among the treated. Intutvely, the dfference between the average effect on the populaton and the average effect on the treated s a smple comparson between means. If the mean of a group s hgher (lower) than the mean of the whole populaton, then such a group has a hgher (lower) mean than the rest of the populaton. However, snce we observe nether the varable wthout the treatment for those partcpatng nor the varable wth treatment for those who dd not partcpate (.e., the counterfactuals), none of these parameters can be computed drectly. Alternatvely, what can be computed wth the avalable data s: E 0 [ Y D = ] E[ Y D = 0] that s, the dfference between the average outcome among the treated mnus the average outcome among the non-treated. The natural queston to ask s whether ths dfference comes close to the ATE. We concentrate n what follows on the ATE, but a smlar argument can be made for the other parameters of nterest. 7

26 If, gven the jont dstrbuton of Y, Y 0, and D, we assume that: f j j ( Y D) = f ( Y ) j = 0, then the expectaton of the outcomes condtonal on partcpaton to the program equals the uncondtonal expectaton. Ths s known as the assumpton of gnorablty of treatment and t s a vald assumpton when the program nterventon s randomly assgned among the populaton. In ths stuaton we have that: [ D= ] EY [ D= 0] = EY [ ] EY [ ] + { EY [ D= ] EY [ ]} { EY [ D= 0] EY [ ]} 0 0 = EY [ ] EY [ ] = EY [ Y ] = ATE EY Namely, the dfference n the observed averages equals the average treatment effect. However, n most cases, treatment s not randomly assgned but there exst a selecton mechansm. Let us further assume that there are other observable varables (vector X) and an unobservable varable (ε ) that explan the selecton mechansm and are jontly dstrbuted wth the varables Y, Y 0 and D. We may then assume that ether there s selecton on observables: f f j j ( Y D) f ( Y ) but or selecton on un-observables: f f j = 0, j j ( Y D) = f ( Y X ) j = 0, j j ( Y D) f ( Y ) but j = 0, j j ( Y D) = f ( Y X, ε ) j = 0, Then, t can be shown that the dfference n the observable averages s based. Formally: 0 0 [ D = ] E[ Y D = 0] = E[ Y ] E[ Y ] {( E[ Y X ] E[ Y ]) ( E[ Y X ] E[ Y ])} {( E[ Y X, ε ] E[ Y X ]) ( E[ Y X, ε ] E[ Y X ])} E Y 8

27 where the second term to the rght s called the overt bas and the thrd term s the hdden bas. The former s due to the exstence of a selecton mechansm that can be explaned wth observable varables whereas the second needs unobservable varables to fully explan the partcpaton n the program. If we assume that there s selecton only on observables (so there s no hdden bas ) then the average treatment effect can be obtaned from the average of condtonal treatment effects. Formally: ATE = E 0 0 [ E[ Y X ] E[ Y X ] = E [ E[ Y Y X ] E [ ATE( X )] X X = On the other hand, f we assume that there s selecton on unobservables then the ATE has to be obtaned from average treatment effects condtoned both on observable and un-observable varables. Formally: ATE = E 0 0 [ E[ Y X ε ] E[ Y X, ε ] = E [ E[ Y Y X, ε ] E [ ATE( X, )] X, ε, X, ε = X, ε ε X Fnally, the comparson between the estmates assumng selecton on observables and the estmates assumng selecton on unobservables provdes nformaton on the exstence of hdden bas. Formally: and hence 0 0 [ Y X, ε ] E[ Y Y X ] ATE( X, ε ) ATE( X ) = E Y 0 0 ( ε ) E[ Y Y X, ε ] E[ Y Y X ] ATE( X, ε ) ATE In other words, f the average effects dffer between methods there s evdence of hdden bas (that s, unobservable varables are not gnorable). Ths s mportant because the sze and sgn of the hdden bas may alter the results of the mpact evaluaton as well as the assessment of the drecton of the selecton mechansm. Intutvely, the presence of hdden bas warns about the exstence of unobservable varables that nfluence the mpact of the program. For nstance, f partcpaton n the program s affected by age, whch s observable, and by work ethcs, whch s not observable, then computng the dfferental mpact by age group s not enough for controllng the systematc dfferences between partcpants and non-partcpants. In ths case, the evaluaton mght mstakenly report a weak mpact of the program that s not because the program s neffectve but s the consequence of partcpants havng a weaker work drve than non-partcpants. 9

28 A. Methods of Impact Evaluaton The exposton of the foregong secton makes t clear that a correct mpact evaluaton has to take nto consderaton the exstence of selecton bas and ts components: overt and hdden bas. Methods of mpact evaluaton clng to assumng ether one or both bases. Hence, methods can be dvded nto two categores: methods assumng selecton-on-observables and methods assumng selecton-on-unobservables. Furthermore, snce the parameters of nterest are condtonal expected values, two approaches can be adopted for estmaton. Frst, a non-parametrc approach that computes sample averages of the form: 0 Y w(, j) Y j j N where w(,j) s a functon that assgns weghts to each control observaton j wth respect to the treatment observaton, and N s the relevant number of observatons. Second, a parametrc approach that assumes that condtonal expectatons can be modeled as functons (lnear or non-lnear) of the form: so Y ( X,, u ) = 0, = f j j β [ Y X ] = f ( X, ) j = 0, E j β Therefore, the methods for mpact evaluaton can be classfed nto four categores, dependng on assumptons about hdden and overt bases, and on the method for computng expectatons. For ths study we have chosen two opposte methods: frst, propensty matchng score wth nearest neghbor controls, whch s a non-parametrc method assumng selecton on observables and, second, selecton correcton, whch s a parametrc procedure assumng selecton on unobservables. For the former we have adopted the methodology developed by Becker and Ichno (2002) based n the semnal work of Rosenbaum and Rubn (983). For the latter we follow the methodology proposed by Heckman, Tobas and Vytlacl (2003). 2 We have chosen these methods for the sake of robustness 2 The procedures mplemented are extensvely explaned n the Methodologcal Annex, page. 20

29 and, as wll be seen below, because comparng these two methods provdes addtonal nsghts on the performance of the program under evaluaton. 3 B. Avalable Data We make use of three dfferent surveys n ths study: the ENCOPE (Spansh acronym for Employment survey of PROBECAT/SICAT benefcares), the ENECE (Spansh acronym for Natonal Tranng and Educaton Survey) and the ENEU (Spansh acronym for Urban Employment Survey). All of them are produced, wth varyng perodcty by the Mexcan statstcs bureau (INEGI). The ENEU s a survey that provdes nformaton on human captal and labor force characterstcs for the populaton aged 2 and more n ctes wth no less than nhabtants. Ths survey s done every quarter snce 988. It has a rotaton mechansm that allows dentfyng ndvduals for fve consecutve quarters. It s mportant to clarfy that each ndvdual n the rotatng panel s ntervewed at a fxed span of 3 weeks. Ths s to say, for nstance, that for a gven year, f one ndvdual was ntervewed n the frst week of January t wll be re-ntervewed n the frst week of Aprl, agan n the frst week of July, agan n the frst week of October and then, for the last tme, n the frst week of January of the followng year. Every week of each quarter an approxmately fxed number of ndvduals s ntervewed untl completon of the sample sze for that quarter. Ths characterstc of the ENEU wll become mportant snce the data for the treatment group do not follow the same pattern. The ENECE s a specal module ntroduced n the ENEU every second year from 99 to 999, and every year snce 200. It provdes soco-demographc nformaton for ndvduals aged 2 and more as well as nformaton on formal schoolng and tranng. It provdes ndvdual data on number of courses, type of tranng, duraton, place and sponsorng of tranng. Snce the ENECE s just a module ENEU, nformaton of tranng can be matched wth all human captal and labor partcpaton characterstcs for sampled ndvduals. Fnally, ENCOPE s a survey that ntervews a sample of PROBECAT-SICAT benefcares between three and sx months after fnshng ther tranng. Although t has detaled nformaton on type of course taken, soco-demographc characterstcs and labor partcpaton at the moment of the ntervew, t has 3 For an extensve account of methods of program evaluaton see Lee (2005), Cameron and Trved (2005) and Wooldrdge (200). For a dscusson on evaluaton methods appled to ant-poverty program see Ravallon (2005) 2