Draft general guidance on sampling and surveys for SSC projects

Transcription

1 Twetiet meetig Page 1 Draft geeral guidace o samplig ad surves for SSC projects Itroductio 1. Te purpose of tis documet is to provide guidace o applig samplig metods we usig small scale CDM metodologies, as well as to provide directio o wat is expected i te proposed samplig pla. Several approved small scale CDM metodologies require estimates of ke iputs based o samples of affected equipmet or facilities. SSC projects tat use samplig sall iclude a descriptio of te scope ad approac to samplig ad a justificatio for te selectio of te cose approac i te project desig documet. Project participats are ecouraged to refer to stadard refereces o probabilit samplig teciques or cosult wit experts to resolve questios tat arise i te cotext of teir specific project requiremets. A list of refereces o probabilit samplig metods ad issues is provided i te aex to tis report.. Te purpose of samplig as it applies to SSC projects is to obtai 1) ubiased ad ) reliable estimates of te mea or total values of ke variables to use i te calculatios of greeouse gas emissio reductios. Recogizig tat measuremets take from a subset (sample) of subjects will tpicall differ from te meas for te etire populatio, te estimates must be ubiased i te sese tat measuremets take from repeated, idepedet samples would be, o average, equal to te populatio values. 3. Secod, te estimates must be reliable i terms of ow closel te are likel to fall aroud te true populatio value i repeated samples. Te statistical reliabilit or precisio of a estimate based o a sample is tpicall expressed i terms of te probabilit tat te sample value falls witi a specified iterval aroud te populatio value. For example, oe migt describe a samplebased estimate as avig a 90% probabilit of fallig i a rage of ± 10% of te true populatio value (ofte deoted as 90/10 precisio). All accepted probabilit samplig metods provide formulas for calculatig te precisio of a estimate, i.e., te probabilit tat it falls witi a give rage of te true populatio value, based o te variabilit of idividual measuremets i te sample. 4. Te precisio of a sample-based estimate icreases directl wit its size. Prior to drawig a sample, project participats ca calculate te size required to acieve a give precisio level based o a forecast or expectatio of te variabilit of te caracteristic i te populatio. I proposig ow large a sample to use to obtai a give estimate, project applicats are expected to justif te umber of measuremets eeded to acieve a target level of precisio establised b te Executive Board. Tat justificatio sould be based o previous studies or soud egieerig judgmets. If te actual sample fails to acieve te target miimum precisio level set b te Board, project implemeters ma be required to take a supplemetal sample to acieve it. 5. Subject to tese two requiremets (ubiased estimates ad miimum precisio levels), project participats ave broad discretio i te samplig approac te propose to use to obtai te estimates. Te coice of wic tpe of sample to propose depeds o several cosideratios, icludig te tpes of iformatio to be collected troug samplig, te kow caracteristics of te populatio, ad te cost of iformatio gaterig. Some of te most commol used samplig metods are summarized below, alog wit guidelies o circumstaces were eac is most applicable. More complete descriptios, togeter wit formulas, are preseted i a appedix to tis documet. Samplig Precisio Requiremets 6. Small scale project activities are ofte omogeous (i.e. sare a commo tecolog wit similar operatig caracteristics) but dispersed (i.e., te tecolog is implemeted at a large umber of sites, ouseolds or facilities). A example is a solar cookig project. Moitorig of

2 Twetiet meetig Page te efficiec of ever sigle solar cooker i a give project activit ma ot be practical or ecoomical. Aoter example could be a project activit tat istalls compact fluorescet lamps i a large umber of residetial dwelligs. A samplig approac could be used to collect data o retetio rates ad operatig ours. I situatios suc as tese, samplig is likel to be a appropriate approac for baselie determiatio ad moitorig provided te approved metodolog applied to te project does ot explicitl state tat samplig sould ot be used. 7. SSC metodologies specif miimum required levels of precisio ad cofidece for various categories of variables collected b wa of samplig. Te samples sould be cose so as to meet or exceed tese miimum levels. Project propoets ma request a revisio of tese requiremets i te metodolog or request a deviatio from te approved metodolog i accordace wit te procedures (see <ttp://cdm.ufccc.it/referece/procedures/metssc_proc0_v01.pdf> ad <ttp://cdm.ufccc.it/referece/procedures/reg_proc03_v0.pdf> providig sufficiet justificatios as to w a lower level is suitable for te plaed applicatio. If te estimates from te actual samples fail to acieve te target miimum levels of precisio, project participats sall perform additioal data collectio o a supplemetal sample. 8. Were tere is o specific guidace i te approved metodolog, project propoets sall coose 90/10 precisio as te miimum precisio targets for te most importat data collectio efforts o te most importat data variable affectig te emissio reductios of te project activit (For example, cosider a project activit tat as istalled ouseold biogas digesters i umerous distributed locatios to displace fossil fuel use for cookig. Te umber of auall operatig biogas digesters directl impacts te emissios reductios of te project activit; terefore te umber of ouseolds for te sample sould be cose so as to acieve a 90% cofidece iterval wit 10 per cet error margi for te collected data. O te oter ad a 90/30 precisio ma be adopted for parameters of outside impact, idirect impact ad verificatio aalsis (For example positive spill over effect of a biogas digester project activit i.e., te umber of ouseolds outside te boudar of te project activit, wo are ot project participats but everteless istalled biogas digesters o teir ow ma be assessed at 90/30 precisio). 9. Approved SSC metodologies defie variables wose values will specificall be obtaied toug samplig. Moreover, project implemeters ma propose to obtai estimates of oter variables usig samplig teciques if tat is te ol practical or cost effective meas to obtai tem. For example, several metodologies require tat estimates for ke variables be obtaied troug moitorig, witout specifig te extet of suc moitorig. 10. I broad terms, te metodologies require samplig i te followig tpes of applicatios: Obtaiig a poit estimate for a variable to be used i a defiitioal or egieerig formula. For example, te average aual ours of operatio of ligtig is used to estimate savigs, were savigs equal te cage i wattage (determied at istallatio) multiplied b te average ours of operatio (based o a sample estimate). ; Estimatig te baselie peetratio or caracteristics of a equipmet tecolog. For example, several metodologies require estimates of te average efficiec of replaced equipmet, suc as eatig or ligtig sstems; Estimatig weter te peetratio or operatig caracteristics of a efficiet tecolog or process ave caged over time. For example, te refrigerator replacemet program requires a aual surve to estimate te percet of uits still i operatio over time.

3 Twetiet meetig Page 3 I additio, future metodologies or amedmets ma call for estimatig weter a field value is sigificatl differet from a value based o laborator tests or previous studies. Table 1 summarizes te miimum precisio requiremets for differet applicatios. Table 1 SSC Precisio Requiremets Tpe of Samplig Estimate Miimum Cofidece Level Maximum Error Boud Miimum Sample Size Poit Estimate for Egieerig Calculatio Baselie Peetratio or Equipmet Caracteristic Cage i Tecolog Peetratio or Performace 90% ± 10% 50 90% ± 10% 50 80% ± 0% 50 Summar of Samplig Approaces ad Applicabilit 11. Te followig is a summar of some of te most commo tpes of samplig approaces ad situatios were eac is recommeded. Formulas for calculatig stadard errors of estimates from eac samplig tecique ad associated sample sizes are provided i te aex to tis report. Simple Radom Sample 1. A simple radom sample is a subset of idividuals (a sample) cose from a larger set (a populatio). Eac idividual is cose radoml ad etirel b cace, suc tat eac idividual as te same probabilit of beig cose at a stage durig te samplig process, ad eac subset of k idividuals as te same probabilit of beig cose for te sample as a oter subset of k idividuals (Yates, Daiel S.; David S. Moore, Dare S. Stares (008). Te Practice of Statistics, 3 rd Ed.. Freema. ISBN ). Simple radom samplig is te most straigtforward metod for desigig a sample. 13. Uder simple radom samplig, eac case i te sample frame (a exaustive list of all te cases i te populatio) as a equal probabilit of beig selected ito te sample. Te mea value of te measuremet from a radom sample is a ubiased estimate of te true populatio mea, wic meas tat repeated idepedet samples will provide estimates tat are, o average, equal to te populatio mea. 14. Simple radom samplig as te advatages tat it is te most straigtforward wa of obtaiig a represetative estimate based o a radom subset of te populatio. Oe simpl assigs a radom umber to eac case i te sample frame ad selects te cases wit te igest umbers correspodig to te target sample size. (For practical reasos discussed below, it is alwas advisable to oversample from te frame.) Usig radom samplig metods are recommeded we more efficiet samplig teciques are ifeasible, impractical, or were te populatio is relativel omogeeous wit respect to te object of te stud. For example, oter samplig metods tpicall require more iformatio from te sample frame, suc as a stratificatio variable. 15. Simple radom samplig is free of classificatio error, ad it requires miimum advace kowledge of te populatio. Its simplicit also makes it relativel eas to iterpret data collected.

4 Twetiet meetig Page 4 For tese reasos, simple radom samplig best suits situatios were tere is limited iformatio available about te populatio ad data collectio ca be efficietl coducted o radoml distributed items, or were te cost of samplig is small eoug to make efficiec less importat ta simplicit. If tese coditios are ot true, stratified samplig or cluster samplig ma be a better coice. Sstematic Samplig 16. Sstematic samplig is a statistical metod ivolvig te selectio of elemets from a ordered samplig frame. Te most commo form of sstematic samplig is a equal-probabilit metod, i wic ever k t elemet i te frame is selected, were k, te samplig iterval (sometimes kow as te skip ), is calculated as: k = populatio size (N) / sample size () 17. Usig tis procedure eac elemet i te populatio as a kow ad equal probabilit of selectio. Tis makes sstematic samplig fuctioall similar to simple radom samplig. It ma be muc more efficiet, owever, if variace of te caracteristic of iterest witi te sstematic sample is greater ta its variace i te populatio. 18. Te researcer must esure tat te cose samplig iterval does ot ide a patter. A patter would treate radomess. A radom startig poit must also be selected. Sstematic samplig is to be applied ol if te give populatio is logicall omogeeous, because sstematic sample uits are uiforml distributed over te populatio. 19. Sstematic samplig is applicable i a umber of situatios. If tere is a atural flow of subjects i te populatio, suc as output of bricks i a maufacturig process, te it is tpicall easier to sample ever kt uit to test for qualit as te are produced. If persoel are expected to take field measuremets from a sub-sample of subjects based o iformatio gatered i te course of eac surve, te sstematic samplig ma be easier to implemet. Tat would be te case, for example, if a surveor takes a ivetor of ligtig fixtures i a buildig ad te istalls meters o a subset of tem. I all cases, it is importat tat te list of subjects or te process is aturall radom, i te sese tat tere is o patter to its order. Stratified Radom Sample 0. Aoter metod is called stratified radom samplig. We sub-populatios var cosiderabl, it is advatageous to group cases ito relativel omogeeous subpopulatios ad sample eac subpopulatio, called a stratum, idepedetl. Te strata sould be mutuall exclusive: ever elemet i te populatio must be assiged to ol oe stratum. Te strata sould also be collectivel exaustive: o populatio elemet ca be excluded. For example, te populatio of participats i a commercial ligtig program migt be grouped accordig to buildig tpe. Te stratificatio requires tat iformatio o te stratificatio variable, e.g., buildig tpe, be cotaied i te sample frame. Te radom or sstematic samplig is applied witi eac stratum. 1. Stratificatio ca icrease te efficiec, i.e., produce a gai i precisio for a give sample size, if te cases witi eac stratum are more omogeeous ta across strata. For example, if ligtig usage witi buildig tpes (office buildigs, retail stores, etc.) varies less ta across buildig categories, te estimates of ours of operatio usig a stratified sample will produce a estimate wit lower variace for a give sample size.. If populatio desit varies greatl witi a regio, stratified samplig ca also esure tat estimates will be made wit equal accurac i differet parts of te regio, ad tat comparisos of sub-regios ca be made wit equal statistical power. For example, a surve take trougout a

5 Twetiet meetig Page 5 particular provice migt use a larger samplig fractio i te less populated ort, sice te disparit i populatio betwee ort ad sout is so great tat a samplig fractio based o te provicial sample as a wole migt result i te collectio of ol a adful of data from te ort. Radomized stratificatio ca terefore be used to improve populatio represetativeess i a stud. 3. Stratified radom samplig is most applicable to situatios were tere are atural groupigs of subjects wose caracteristics are more similar witi group tat across groups. It requires tat te groupig variable be kow for all subjects i te sample frame. For example, te samplig frame would require iformatio o te buildig tpe for eac case i te populatio to allow stratificatio b tat caracteristic. Cluster Samplig 4. Clustered samplig refers to a tecique were te populatio is divided ito sub-groups (clusters), ad te sub-groups are sampled, rater ta te idividual elemets to be studied. Cluster samplig is used we atural groupigs are evidet i a populatio. I tis tecique, te total populatio is divided ito sub-groups (clusters), ad a sample of te groups is selected. For example, suppose a project istalls ig efficiec motors i buildigs, wit several motors tpicall i eac buildig. If oe is iterested i estimatig te operatig ours of te motors, oe migt take a sample of te buildigs istead of te motors, ad te meter all of te motors i te selected buildigs. I cotrast to stratified samplig, were te equipmet of iterest is grouped ito a relativel small umber of omogeeous segmets, tere are ma clusters of motors (i.e., buildigs), ad tere is o expectatio tat te motors i eac buildig are more omogeeous ta te overall populatio of efficiet motors. 5. Oe versio of cluster samplig is area samplig or geograpical cluster samplig. Clusters cosist of geograpical areas. Because a geograpicall dispersed populatio ca be expesive to surve, greater ecoom ta simple radom samplig ca be acieved b treatig several respodets witi a local area as a cluster. It is usuall ecessar to icrease te total sample size to acieve equivalet precisio i te estimates, but cost savigs ma make tat feasible. 6. Tere are at least two reasos for usig a clustered samplig approac to collect data. Te first is cost. If a sigificat compoet of te cost of data collectio is travel time betwee sites, te it ma make sese to moitor all of te equipmet at idividual locatios to reduce tat cost compoet. Uder tat approac, it will tpicall be ecessar to meter more pieces of equipmet ta uder radom samplig to acieve a give level of precisio. But te reductio i cost ma more ta offset a egative effects o sample precisio, allowig oe to take a larger sample for a give budget, wit a icrease i precisio. 7. Te secod reaso is te ease of costructig te sample frame. I some cases were te program participat is collectig baselie iformatio, it ma be impossible to eumerate te populatio of pieces of equipmet from wic to draw te sample. But it is possible to eumerate te clusters, e.g., buildigs. I tat situatio, te program participat could sample te buildigs ad coduct a ivetor of te equipmet i te cose uits. 8. I most applicatios of cluster samplig to moitor efficiet equipmet, te sub-groupigs of uits occur aturall, wit a differet umber of elemets per cluster. For example, a buildig or plat locatio migt costitute a atural cluster, wit varig umbers of motors per locatio. Multi-Stage Samplig 9. Multistage samplig is a complex form of cluster samplig. Measurig all te sample elemets i all te selected clusters ma be proibitivel expesive or ot ecessar. Uder tose

6 Twetiet meetig Page 6 circumstaces, multistage cluster samplig becomes useful. I multi-stage samplig, te uits (referred to as primar uits) i te populatio are divided ito smaller sub-uits (referred to as secodar uits), similar to cluster samplig. I cotrast to cluster samplig were all of te secodar uits (elemets) are measured, data are collected for ol a sample of te sub-uits. For example, a stud of efficiet ligtig migt first draw a sample of buildigs, ad te take a sample of ligtig fixtures i eac selected buildig. If te caracteristics of te fixtures i a give buildig are ver similar ad te costs of measurig tem is relativel ig, te takig a sample of fixtures ma be sufficiet to acieve a target level of precisio at lower cost. O te oter ad, if te measuremets are iexpesive oce a tecicia is o-site, te it ma make sese to moitor all of te fixtures. Multi-stage samplig ca be exteded furter to tree or more stages. For example, oe migt group te populatio ito buildig complexes, te buildigs, ad fiall fixtures. 30. Tere are ma variatios i metods i applig multi-stage samplig. If te umber of secodar uits i eac primar uit is ot kow i te sample frame, te oe approac is to draw a sample of primar uits at radom, cout te umber of secodar uits i eac selected primar uit, ad te take detailed measuremets for a sample of secodar uits. If te umber of secodar uits is kow i te sample frame ad varies ol moderatel across uits, te oe ca stratif te primar uit populatio b size ad draw successive radom samples of primar ad secodar uits. Te stadard formulas for radom samplig appl to te secodar uit meas, ad te formulas for stratified samplig appl to te grad mea. Aoter optio is to sample te primar uits wit probabilit proportioal to size, ad to draw a radom sample of te secodar uits i te selected primar uits. Te relative performace of tese alteratives depeds o te populatio caracteristics, te costs of data collectio, ad te availabilit of iformatio o te primar ad secodar uits i te sample frame. Samplig Practices 31. I all of te approaces, care must be take to esure tat te samples are draw i a maer tat avoids a sstematic bias ad tat te data collectio miimizes o-samplig errors. I order to acieve tose goals, practitioers are expected to observe soud practices i desigig samples ad admiisterig surves ad field measuremets. 1 Tose practices iclude: Defiig precisel te samplig objectives, target populatio ad te sample measuremets. Te target populatio from wic te sample will be draw, te iformatio tat will be collected, ad te metods of measuremets sould be clearl specified. Developig te samplig frame. Te implemeter sould compile as complete list of te subjects i te target populatio as possible, alog wit a iformatio eeded to implemet te cose samplig tecique ad to cotact selected subjects. I cases were te plaed measuremets will be take from project participats, tat list would tpicall ave bee maitaied as part of te program or project trackig. I cases were te measuremets are aimed at determiig baselie peetratios or tecolog caracteristics, te implemeter ma eed to costruct te list from oter sources, suc as electric utilit accout records, veicle registratios, or busiess directories. Te implemeter sould idetif were te sample frame ma differ from te target populatio ad establis procedures o ow tat problem will be adled. For example, if te target populatio is residetial ouseolds or dwelligs wit electric service ad utilit 1 For a ver compreesive treatmet of issues surroudig sample/surve desig, see Houseold Sample Surves i Developig ad Trasitio Coutries, Uited Natios, 005, ISBN

7 Twetiet meetig Page 7 billig records are used to costruct te frame, te te samplig ma eed to address suc issues as o-residetial ad master metered accouts. I situatios were it is impossible to costruct a sample frame tat accuratel represets te target populatio, te implemeter ma eed to use area cluster samplig or aoter approac tat is feasible, wile applig te correspodig estimatio ad statistical cofidece formulas. Radomizig cases ad drawig sample. Te implemeter sould esure tat te sample is draw at radom from te sample frame. If a sstematic samplig is cose, te te orderig of subjects o te sample sould be radom ad free of a tred or cclical patter. Selectig te most effective iformatio gaterig metod. Te implemeter sould decide o wat would be te most reliable ad cost effective metod for collectig te data, depedig o te variables of iterest. Alterative metods iclude visual ispectios, psical measuremets, respodet self-reports, ad operatioal logs. For example, equipmet peetratios ad retetio rates ma be determied b ispectios or self-reports. Estimates of electric cosumptio could be based o differet meterig tecologies depedig o te caracteristics of te equipmet. Veicle travel miles or equipmet operatig scedules could be draw from odometers or operatio logs. Coductig surves/measuremets. Te project implemeter is expected to establis ad implemet procedures to esure tat te field data collectio is performed properl ad tat a potetial errors are miimized ad documeted. Suc procedures iclude developig field measuremet protocols, traiig persoel, establisig cotact procedures, documetig coverage problems, missig cases, ad o-respose, miimizig o-samplig measuremet errors, ad qualit cotrol for data codig errors. Requiremets for Samplig Pla i PDD 3. Project desig documets submitted to te Executive Board sould iclude plas for collectig iformatio from samples. Tose plas sould cover te topics summarized i te previous sectio: Field Measuremet Objectives ad Data to be collected. Te pla sould clearl describe te variables to be collected, te scope ad metod of te surve or field measuremets, teir frequec, ad ow te data will be used; Target Populatio. Te pla sould describe te target populatio for te surve or field measuremets ad summarize its kow caracteristics. Samplig Frame. Te pla sould summarize te samplig frame ad te iformatio it cotais tat will be required to implemet te proposed sample metod. If te frame is ot alread at least partiall costructed we te proposal is submitted, te pla sould describe ow it will be developed. Te pla sould also describe ow te caracteristics of te samplig frame ma differ from tose of te target populatio ad weter suc differeces ma seriousl affect ow represetative te estimates ma be of te desired variables. Metods for dealig wit samplig frame problems, icludig possibl supplemetig it for kow sources of o-coverage, sould be described. (See Kis, pp for a treatmet of frame problems.) Sample Metod. Te samplig metod sould be preseted. Tat metod sould be cosistet wit te iformatio cotaied i te frame.

8 Twetiet meetig Page 8 Desired Precisio/Expected Variace ad Sample Size. Te pla sould preset ad justif te target umber of completed surves or field measuremets (te sample size). Tat justificatio sould iclude a predictio of te variace of te variables of iterest ad basis for tat predictio. Procedures for Admiisterig Data Collectio ad Miimizig No-Samplig Errors. Te pla sould describe te procedures for coductig te data collectio, icludig traiig of field persoel, provisios for maximizig respose rates, documetig out-of-populatio cases, refusals ad oter sources of o-respose, ad related issues. Samplig Pla Evaluatio Criteria 33. Te proposed samplig plas will be evaluated based o weter te adequatel address all of te issues ad topics outlied above. Assessmet icludes weter te proposed approac to sample is practical give te available iformatio about te populatio ad te feasibilit of developig te sample frame. Te samplig approac will be evaluated for its adequac i dealig wit te rage of samplig ad o-samplig errors tat ma arise. Te basis for te forecasts of te variace will assessed, alog wit te sufficiec of te proposed sample size give te miimum precisio/cofidece levels. Te samplig pla submitted b project propoets will be reviewed ad assessed based o ow effectivel te address te followig issues ad questios: Does te samplig pla preset a reasoable approac for obtaiig ubiased, reliable estimates of te variables? Is te data collectio metod likel to provide reliable data give te ature of te variables ad project, or is it subject to measuremet errors? Is te populatio clearl defied ad ow well does te proposed approac to developig te samplig frame represet tat populatio? Does te frame cotai te iformatio ecessar to implemet te samplig approac? Is te samplig approac suitable, give te ature of te variables, te data collectio metod, ad te iformatio i te sample frame? Is te proposed sample size adequate to acieve te miimum cofidece/precisio requiremets? Is te ex ate estimate of te populatio variace eeded for te calculatio of te sample size adequatel justified? Example Are te procedures for te data measuremets well defied ad do te adequatel provide for miimizig o-samplig errors? Baselie Peetratio of Compact Fluorescet Lamps (CFLs), Average Aual Operatig ours, ad Survival Rates for Project Lamps 34. Project Descriptio. Te project provides CFLs to residetial ouseolds wit low electric use troug direct istallatios. Te project targets low use ouseolds because te utilit provides service to tose customers at a discout to its margial cost of electricit uder its iverted block tariff. Teams of istallers ispect dwelligs ad idetif fixtures suitable for CFLs. Te bulbs i eac of tose fixtures are replaced wit a CFL of comparable lumes. 35. Target Populatio, Measuremet Objectives ad Metods. Te target populatio for te project ad te field data collectio is residetial dwelligs wit electric service wose average

9 Twetiet meetig Page 9 motl cosumptio falls below a give level. Tree tpes of measuremets will be take. Te first is a baselie ivetor of ligtig fixtures i eac dwellig, icludig te percetage of screwi fixtures alread usig CFLs. Te iformatio from te baselie surve will be used for future program plaig, but ot for te curret project, because te requiremet uder te direct istallatio program tat 100% of te retrofitted fixtures use icadescet bulbs. Oce customers become more familiar wit CFLs, te project will trasitio to offerig icetives troug ormal retail caels. Te baselie surve will provide critical iformatio for desigig tat program. 36. Te secod measuremet is te ours of operatio of te CFLs. Tose will be measured usig ligt loggers tat record te time itervals we te fixtures are tured o. Te loggers will be placed i fixtures for a miimum of iet das ad moved periodicall to capture a seasoal variatios i ligtig use. Te primar objective of te measuremets is to gai a reliable estimate of te average aual ours of operatio of retrofitted fixtures for te purpose of calculatig electricit savigs. 37. Te tird set of measuremets is aimed at determiig te retetio rates or effective useful lives of te CFLs. Tat will be accomplised be ispectig a sample of retrofitted fixtures auall to determie if te CFL is still operatig or as bee replaced b a icadescet bulb or comparable CFL. 38. Sample Frame. Te sample frame for te baselie surve will be developed from te utilit s customer accout records. Te frame cosists of curretl active accouts wit a residetial service code. Te frame icludes iformatio o te customer s service ad billig address, as well as electricit cosumptio for te past twelve billig periods. 39. Te sample frames for te ours of operatio ad te CFL retetio rates will be developed as part of te project trackig sstem. Eac retrofitted fixture will be eumerated i te trackig sstem, alog wit iformatio o its caracteristics (e.g locatio, lumes). 40. Sample Metod. Te baselie surve will be performed o a cluster sample of dwelligs. Te dwelligs will be draw at radom from te sample frame wit eac case avig a equal probabilit of selectio. For eac cose sample tat participates i te surve, a complete ivetor of fixtures will be take. Iformatio o fixtures locatio, tpe, wattage, ad oter relevat caracteristics will be recorded. Te baselie peetratio of CFLs will be calculated as te total umber of istalled CFLs divided b te total umber of screw-i fixtures tat are CFL compatible. 41. Te fixtures for te ours of operatio measuremets will be selected usig two stage samplig. First, a sample of participatig dwelligs will be draw at radom from te project trackig sstem wit te probabilit of selectio proportioal to te umber of retrofitted fixtures i te dwellig. Te a sigle retrofitted fixture will be cose at radom from eac dwellig i te sample for meterig. Tis samplig procedure will be repeated ever quarter ad te meters will be moved to te ew sample. 4. Desired Precisio/Expected Variace ad Sample Size. Te target levels of precisio for te baselie peetratio ad te average aual CFL operatig ours are bot ±10% wit a 90% percet cofidece level (critical t value of 1.64). For te purpose of determiig te sample size for te CFL peetratio rate, te project plaers expect tat approximatel 0% of all residetial fixtures alread ave CFLs. Tat expectatio is based o a o-represetative pilot surve ad aecdotal iformatio from project plaers. If te fixtures for te baselie surve were selected totall at radom from te residetial populatio, te sample size eeded to estimate te 0% peetratio witi ± % wit a 90% cofidece level would be 1076 (equals..8 (1.64/.0) ). But because te sample is clustered, te expected variace is iger. Te plaig purposes, it is assumed tat te actual variace is.5 times tat for a radom sample. Tat ields a total sample

10 Twetiet meetig Page 10 size of fixtures equal to 690. Te plaers coservativel estimate tat tere is a average of at least six screw-i fixtures per dwellig, resultig i a total umber of surveed dwelligs equal to approximatel For te purpose of determiig te sample size for te meterig of ours of operatio, te variace of te estimate used i plaig is approximated b te formula for a simple radom sample. Tat is a good approximatio, give tat te dwelligs are selected wit probabilit proportioal to te umber of retrofitted fixtures. For plaig, it is assumed tat te average aual ligtig use is 150 ours, or sligtl less ta 3.5 ours per da. Based o tis, te target precisio boud is =/- 15 ours per ear. Previous studies of residetial ligtig usage ave foud tat te stadard deviatio of usage is o te order of 500 ours per ear, wic implies tat 95% of usage lies i te rage of 50 to 50 ours per ear. Te sample size eeded to estimate te ours of usage witi te target rage at a 90% cofidece level is less ta 50 (equals ( /15) = 43). To be coservative, as well as to capture seasoal variatios i ligtig usage, four groups of fift fixtures will be metered for successive iet da periods. Eac of te four sub-samples will be draw idepedetl, allowig seasoal comparisos of usage (altoug at a lower cofidece level ta average dail usage). Te total sample of 00 fixtures is ver coservative ad is iteded, i part, to compesate for a icrease i te samplig error due to te two stage samplig approac. 44. Te CFL attritio rate will be estimated b meas of a logitudial aual surve of participatig dwelligs. A radom sample of participats will be draw, ad teir dwellig fixtures will be ispected auall to verif cotiued CFL use. For plaig purposes, we expect tat te attritio rate will be approximatel 10% i te first ear, due to earl failure ad dissatisfactio wit CFL performace. Afterwards, te removal rate is expected to fall to aroud 4%. Usig te iger sample size eeded to estimate te 4% failure rate =/- 0% at te 80% cofidece level (critical t value of 1.8), a miimum of almost 986 fixture must be ispected auall. Usig a average umber of four retrofitted fixture per dwellig, te miimum umber of ousig uits to be ispected is 50. To be coservative, te project will ispect 300 dwelligs for auall. 45. Data Collectio Procedures. Te baselie surve, meterig, ad aual ispectios will be carried out b a professioal surve firm. Te firm will use experieced field ispectors wo are full traied i proper teciques. Te project will prepare te field ispectors wit iformatio about te project, CFL caracteristics, cotact procedures, treatmet of out-of-populatio cases, refusals, ad oter issues arisig i o-site ispectios of tis tpe. Te recipiets of te CFLs will be required to agree to beig surveed as a coditio of project participatio. All of te data collected uder eac surve compoet will be coded ito a electroic database ad cecked for accurac. Complete reports summarizig te results of eac surve compoet will be prepared.

11 Twetiet meetig Page 11 Aex 1 SAMPLING FORMULAS Defiitios N deotes te umber of projects or devices i te populatio ad assume tat te projects are labeled i = 1, K, N ; deotes a measurable variable of iterest, suc as ours of operatio, ad i deotes te value of for project i ; Y deotes te true total of for all N projects i te populatio, i.e., Y = Y 1 N Y deotes te populatio mea of, Y = = = = i 1 i ; N N 1 S deotes te (true) populatio variace of, S = ( Y ) 3 i ; ( N 1) deotes te sample size; s deotes te estimate of te populatio variace based o te sample 4. Simple Radom Samplig N i i= 1 ; 1. Uder simple radom samplig, eac case i te sample frame as a equal probabilit of beig selected ito te sample. Te estimate of te mea value from te sample is give b te formula: = i. Te sample mea is a ubiased estimate of te true populatio mea, wic meas tat repeated idepedet samples will provide estimates tat are, o average, equal to te populatio mea. 3. Te sample estimate for te populatio variace is give b te formula: Some texts use te otatio µ to deote te populatio mea. 3 Some texts use te otatio σ to deote te populatio variace rater ta (uppercase) S. 4 It is importat to ote te differeces amog te uses of te term variace i tis documet. Te first is te populatio variace, wic is te true variace of te variable of iterest i te populatio ad is ukow, uless a complete cesus is take. Te secod is te estimated variace of te variable of iterest from te sample. Te tird is variace of te mea, wic is te variace of te mea value. Its square root is te stadard error of te mea. Te last is te expected variace. Tis is te researcer s expectatio (or prior guess) of wat te sample variace will tur out to be prior to takig te measuremets from te sample.

12 Twetiet meetig Page 1 ( ) i 5 S = 1 Te stadard error of, deoted b te smbol s, is simpl its square root. 4. Te stadard error of te mea, a measure of te dispersio of te estimate of te mea value from te sample aroud te true populatio mea, is: s S = 1 / N 6 5. Uder te fairl mild assumptio tat is ormall distributed, wic is adequate i most practical situatios 7, oe ca state te probabilit tat falls witi a specified rage of te populatio mea. Tat formula is give b: ts ts Prob Y 1 / N Y + 1 / N = Prob( t) Were t is te ormal deviate correspodig to Prob (t), i.e., te percetage of te ormal probabilit desit tat falls witi t stadard deviatios of te mea. Suppose, for example, tat oe obtais a estimate of mea operatig ours of 800 per ear from a sample, wit a stadard error ( S ) of ± 40. Te oe ca state wit 90% cofidece tat te true populatio mea lies i te rage of ± 1.64*40 aroud 800, were 1.64 is te value of t correspodig to 90%. Oe ca state wit approximatel 95% cofidece tat average operatig ours falls i te rage of ours per ear (1.96 stadard deviatios aroud 800 i eac directio). 6. Te formula also provides te basis for specifig te size of a sample, give a target level of precisio for te estimate of te mea value ad a prior estimate of te variace i te variable of iterest i te populatio. Suppose, for example, tat oe wised to estimate te average operatig ours wit a precisio of ± 100 ours per ear, wit a cofidece of 90%. Furter suppose tat previous studies sowed tat te stadard deviatio of operatig ours i te populatio is o te order of 500 ours. (For te purpose of tis example, assume tat te populatio (N) is large, so tat oe ca igore te term 1 / N, kow as te fiite populatio correctio term or fpc.) Give tat te sample estimates of te mea are approximatel ormall distributed, 90% of its desit falls witi 1.64 stadard deviatios of its mea. To meet te precisio target of 100, oe must obtai a estimate of te mea wit a stadard error of 100/1.64, or approximatel 61. Solvig for te value of results i a sample size of Roudig up to te earest wole umber, a sample of 68 will produce a estimate of te operatio ours tat lies witi 100 of te true populatio average over 90% of te time. 7. Te formula for te required miimum sample size ( mi ) to estimate te mea value of a variable ( ) witi a iterval ( l t ) at a desigated probabilit P, were t crit, P is te t value correspodig to P, is give b: 5 Cocra William G., (1977). Samplig Teciques, 3 rd Ed.. Jo Wile ad Sos. ISBN X, p Cocra (1977), p.7 7 Cocra (1977), p. 39

13 Twetiet meetig Page 13 mi t crit, p S = + ( t crit, P l S ) t N 8. Te secod term i te deomiator, ( S tcrit, p ) / N, approaces zero for a large populatio, so tat te formula ca be approximated b: mi t = crit S, p l t 9. We te variable of iterest is biar (es/o), te variace s is simpl te frequec (percet es ) times oe mius te frequec (percet o ). Tat would be te case, for example, i estimatig te attritio rate of istalled equipmet, were es idicates te equipmet was removed ad o idicates te equipmet is still operatig. If oe uses a ex ate estimate of 10% attritio, te expected variace is.09 (equals.1.9). Suppose oe wats to calculate te miimum sample size ecessar to estimate te attritio rate, ± 0%, at te 80% cofidece level. Te t crit,80% = 1.8. Te precisio is.0 (0% of.1). Te formula te is sample size =.1.9 (1.8/.0). Tat is equal to Estimates for variables wit greater variabilit require larger samples to acieve comparable cofidece/precisio levels. Oe measure of te variabilit relative to te populatio mea value is called te coefficiet of variatio (cv). It is defied as te ratio of te stadard deviatio to te mea (te stadard deviatio is te square root of te variace). Usig stadard otatio, tis is σ/µ, were σ deotes te stadard deviatio ad µ deotes te populatio mea. Te cv is a useful measure of variabilit if te rage of te variable is alwas positive. Tat would be te case, for example, for ours of operatio of equipmet. I tat situatio, te cv less ta oe migt be reasoable. Average ligtig use migt be i te rage of 150 ours per ear, wit a stadard deviatio of 500 ours. Te 95% of all use lies rougl betwee 50 ad 50, two stadard deviatios of te mea. Te cv i tis example is 0.4. Te cv for a biar variable wose tat is betwee 0% ad 80% lies betwee.5 ad.0. But we te biar variable is close to zero, te cv grows muc larger ad is ver sesitive to small cages. 11. If te precisio rage is defied i terms of a percetage of te mea value (e.g., 10% of te mea), te formula for te miimum sample size ca be re-stated i terms of te cv ad te percet (pct). It is: Mi sample size = (σ/µ) (t crit /pct). For example, if oe wised to estimate te average ours of operatio for ligtig i a large populatio, wit a cv of 0.4, witi ±10% at te 90% cofidece level (t crit, 90% =1.645), te oe would eed a miimum sample size of 44 (43.3, rouded up). 1. Te followig tables provide miimum sample size values b cofidece precisio levels for represetative cases were te cv = 1 (Table A) ad were te cv = 0.5 (Table B). I bot examples, it is assumed tat te populatio is sufficietl large, so tat tere is o eed to use te fiite populatio correctio factor. Also, te miimum sample size of 50 is substituted i cases were te formula ields a umber below tat tresold. Tat avoids a adjustmet i te critical values due to small sample sizes.

14 Twetiet meetig Page 14 Table A: Miimum Samples Sizes for Coefficiet of Variatio = 1 Cofidece Level 80% 90% 95% 99% Precisio as Percet of Mea 1% % % % Table B: Miimum Samples Sizes for Coefficiet of Variatio = 0.5 Cofidece Level 80% 90% 95% 99% Precisio as Percet of Mea 1% Sstematic Samplig 5% % % Sstematic samplig is a statistical metod ivolvig te selectio of elemets from a ordered samplig frame. Te most commo form of sstematic samplig is a equal-probabilit metod, i wic ever k t elemet i te frame is selected, were k, te samplig iterval (sometimes kow as te skip ), is calculated as: k = populatio size (N) / sample size () 14. Usig tis procedure eac elemet i te populatio as a kow ad equal probabilit of selectio. Tis makes sstematic samplig fuctioall similar to simple radom samplig. It is owever, muc more efficiet (if variace witi sstematic sample is more ta variace of populatio). 15. Te researcer must esure tat te cose samplig iterval does ot ide a patter. A patter would treate radomess. A radom startig poit must also be selected. Sstematic samplig is to be applied ol if te give populatio is logicall omogeeous, because sstematic sample uits are uiforml distributed over te populatio. Example: Suppose a brick maufacturer wats to validate te qualit of its bricks, te usig sstematic samplig it ca coose ever 500 t brick beig produced ad perform te tests o tis sample. Tis is radom samplig wit a sstem. From te samplig frame, a startig poit is cose at radom, ad coices tereafter are at regular itervals. For example, suppose ou wat to sample 0 bricks from a productio ru of bricks, so ever 500 t brick leavig te kil is cose after a radom startig poit betwee 1 ad 500. If te radom startig poit is 11, te te bricks selected are 11, 511, etc. 16. Sstematic samplig ma also be used wit o-equal selectio probabilities. I tis case, rater ta simpl coutig troug elemets of te populatio ad selectig ever k t uit, we

15 Twetiet meetig Page 15 allocate eac elemet a space alog a umber lie accordig to its selectio probabilit. We te geerate a radom start from a uiform distributio betwee 0 ad 1, ad move alog te umber lie i steps of 1. Example: We ave a populatio of 5 uits (A to #). We wat to give uit A a 0% probabilit of selectio, uit B a 40% probabilit, ad so o up to uit E (100%). Assumig we maitai alpabetical order, we allocate eac uit to te followig iterval: A: 0 to 0. B: 0. to 0.6 (=0.+0.4) C: 0.6 to 1. (= ) D: 1. to.0 (=1.+0.8) E:.0 to 3.0 (=.0+1.0) 17. If our radom start was 0.156, we would first select te uit wose iterval cotais tis umber (i.e., A). Next, we would select te iterval cotaiig (elemet C), te.156 (elemet E). If istead our radom start was 0.350, we would select from poits (B), (D), ad.350 (E). 18. Sstematic samplig is a relativel simple metod to appl, wic makes it easil uderstood ad useful we field persoel are asked to sample a subset of uits based o teir ispectios. It is also useful we te populatio is orgaized i some temporal or oter atural order witout a tred or cclical patter. Tat would be te case for te productio of bricks or oter maufacturig processes. Stratified Radom Samplig 19. Aoter metod is called stratified radom samplig. We sub-populatios var cosiderabl, it is advatageous to sample eac subpopulatio (stratum) idepedetl. Uder stratified radom samplig, te populatio is divided ito relativel omogeous subgroups, called strata. Te strata sould be mutuall exclusive: ever elemet i te populatio must be assiged to ol oe stratum. Te strata sould also be collectivel exaustive: o populatio elemet ca be excluded. For example, te populatio of participats i a commercial ligtig program migt be grouped accordig to buildig tpe. Te stratificatio requires tat iformatio o te stratificatio variable, e.g., buildig tpe, be cotaied i te sample frame. Te radom or sstematic samplig is applied witi eac stratum. 0. Allocatio ca be proportioate or optimum. Proportioate allocatio uses a samplig fractio i eac of te strata tat is proportioal to tat of te total populatio. If te populatio cosists of 60% i te male stratum ad 40% i te female stratum, te te relative size of te two samples (tree males, two females) sould reflect tis proportio. Optimum allocatio (or Disproportioate allocatio) - Eac stratum is proportioate to te stadard deviatio of te distributio of te variable. Larger samples are take i te strata wit te greatest variabilit to geerate te least possible samplig variace. 1. If populatio desit varies greatl witi a regio, stratified samplig will esure tat estimates ca be made wit equal accurac i differet parts of te regio, ad tat comparisos of sub-regios ca be made wit equal statistical power. For example, a surve take trougout a particular provice migt use a larger samplig fractio i te less populated ort, sice te disparit i populatio betwee ort ad sout is so great tat a samplig fractio based o te provicial sample as a wole migt result i te collectio of ol a adful of data from te ort. Radomized stratificatio ca terefore be used to improve populatio represetativeess i a stud.. Stratificatio ca icrease te efficiec, i.e., produce a gai i precisio for a give sample size, if te cases witi eac stratum are more omogeeous ta across strata. For example, if ligtig usage witi buildig tpes (office buildigs, retail stores, etc.) varies less ta

16 Twetiet meetig Page 16 across buildig categories, te estimates of ours of operatio usig a stratified sample will produce a estimate wit lower variace for a give sample size. Notatio for Stratified Radom Samplig N deotes te total umber of uits i te stratum deotes te umber of uits i te sample i stratum i deotes te value for te it uit i stratum W N = deotes te stratum weigt N f = deotes te samplig fractio i te stratum N Y N i = deotes te true populatio mea i stratum N N i = deotes te sample mea i stratum N N ( i Y S = N 1 i ) deotes te true populatio variace 3. Te estimate of te mea from a stratified radom sample is: st = W 4. Te estimate is ubiased, regardless of ow te sample is distributed across strata, as log as te cases are draw idepedetl, at radom from eac stratum. If te stratified sample is draw proportioatel from eac stratum, i.e., if / = N / N te stratified sample mea is aritmeticall te same as te radom sample mea. 5. Te variace of te mea estimate is give b: V ( st estimate of S ) = W (1 f ). If a radom sample is take i eac stratum, te a ubiased 1 S sample mea: S is = ( i ). Tis produces a ubiased estimate of te overall variace of te ( 1)

17 Twetiet meetig Page 17 S 1 ) = N ( st N ( N S ). 6. Tat formula ca be used to calculate a sample size ecessar to acieve a give level of precisio i te same maer as described for a radom sample. Te calculatio requires prior estimates of te variaces i eac stratum. Give tose prior estimates, oe ca calculate a optimal allocatio of te total sample across strata ( ), ad te compute te resultig variace (ad stadard error) for te differet total sample sizes. Tat determies te sample ecessar to acieve a specified level of precisio ad cofidece. 7. Te optimal allocatio of a sample across strata depeds o te relative variaces i eac stratum ad te cost of data collectio per sample poit. Te formula for te optimal allocatio is: = W S c ( WS / c ) /, were c is te cost per sample poit i stratum. 8. Eve were it is impossible to obtai accurate estimates of relative stratum variaces ad data collectio costs, te formula provides guidace for improvig te efficiec of sample estimates troug stratificatio. Tose are to allocate larger portios of te sample to strata were: te subpopulatio is larger; tere is more variabilit witi te stratum; ad Samplig is less expesive. Cluster Samplig 9. Clustered samplig refers to a tecique were te populatio is divided ito sub-groups (clusters), ad te sub-groups are sampled, rater ta te idividual elemets to be studied. Cluster samplig is used we atural groupigs are evidece i a statistical populatio. I tis tecique, te total populatio is divided ito sub-groups (clusters) ad a sample of te groups is selected. For example, suppose a project istalls ig efficiec motors i buildigs, wit several motors tpicall i eac buildig. If oe is iterested i estimatig te operatig ours of te motors, oe migt take a sample of te buildigs istead of te motors, ad te meter all of te motors i te selected buildigs. I cotrast to stratified samplig, were te equipmet of iterest is grouped ito a relativel small umber of omogeeous segmets, tere are ma clusters of motors (i.e., buildigs), ad tere is o expectatio tat te motors i eac buildig are more omogeeous ta te overall populatio of efficiet motors. 30. Oe versio of cluster samplig is area samplig or geograpical cluster samplig. Clusters cosist of geograpical areas. Because a geograpicall dispersed populatio ca be expesive to surve, greater ecoom ta simple radom samplig ca be acieved b treatig several respodets witi a local area as a cluster. It is usuall ecessar to icrease te total sample size to acieve equivalet precisio i te estimators, but cost savigs ma make tat feasible. 31. Tere are at least two reasos for usig a clustered samplig approac to collect data. Te first is cost. If a sigificat compoet of te cost of data collectio is travel time betwee sites, te it ma make sese to moitor all of te equipmet at idividual locatios to reduce tat cost compoet. Uder tat approac, it will tpicall be ecessar to meter more pieces of equipmet ta uder radom samplig to acieve a give level of precisio. But te reductio i cost ma more ta offset a egative effects o sample precisio, allowig oe to take a larger sample for a give budget, wit a icrease i precisio.