JSM 013 - Survey Researc Metods Section Variance Estimation of te Design Effect Alberto Padilla Banco de México Abstract Sample size determination is a crucial part of te planning process of a survey and it can be accomplised in different ways, some of tem require information not available or tat may be obtained wit a substantial cost. Sample size calculation can be done by using te design effect estimator proposed by Kis. Tis estimator is also used as an efficiency measure for a probability sampling plan and to build confidence intervals. Even toug te design effect estimator is widely used in practice, little is known about its statistical properties and tere are no variance estimators available. In tis paper we propose a metod to estimate te variance. Wit tis estimator it is possible to assess te precision of te estimators during te planning stage of a survey. An example using stratified sampling is given. Key Words: Ratio estimator, simple random sampling, design effect, sample size; resampling metod 1. Introduction Te design effect, deff K, Kis (1965), is defined as te ratio of te variance of an estimator under a specific design to te variance of te estimator under simple random sampling witout replacement,. Te estimator of te design effect is used for example in te computation of te sample size for complex sample designs and to build confidence intervals. To determine te sample size one must ave an estimation of te deff K for te statistic of interest and a computation of te sample size under. Wit tese two elements at and, setting aside adjustments for nonresponse, te computation of te sample size for a complex designs reduces to te product of te deff K and te sample size under. Confidence intervals can be built by multiplying te standard deviation of a statistic, like a mean or total, computed under wit te square root of te estimated deff K. From tese examples, it can be seen tat te deff K elps us to compare te efficiency of design plans wit. A plan would be more, equal or less efficient tan if te estimated deff K is less, equal or greater tan one. It is wort mentioning tat in te deff estimator, te estimation of te variance under is computed wit te sample obtained from te complex design witout considering te stratification, clustering, unequal probability sampling, etc. Tis metod does not guarantee an unbiased estimation of te population variance under. Tis problem as been analyzed by Rao (196) wo built unbiased estimators for te variance under for tree complex designs; Cocran (1977) illustrated Rao s metod for a stratified population and recently Gambino (009) built an unbiased estimator in terms of te orvitz-tompson estimator (195) of te population caracteristic. Te estimator of te deff K is a widely used quantity but little is known about its properties and to te 603
JSM 013 - Survey Researc Metods Section autor s best knowledge tere are no variance estimators available. We propose a resampling metod to estimate te variance of te design effect, namely Sitter s bootstrap for complex designs, see Sitter (199). Te results obtained are promising and it seems to be an option to estimate te variance of te design effect estimator proposed by Gambino (009). Te article is organized as follows. In section we introduce definitions and notation used trougout te paper. An example of sample size determination is sown in section 3. Section 4 deals wit te proposed resampling metod for estimating te variance of te design effect and an example using stratified random sampling is given.. Notation and Definitions Let U denote a finite population of N elements labeled as k=1,,n, 1<N. It is customary to represent U wit its labels k as U={1,,,k,,N}. Te example used in tis paper refers to stratified random sampling, ereinafter strs, so it is convenient to introduce some definitions and notation for tis design. Stratified random sampling: te variable of interest will be represented by y i, were i stands for te it element of te population in t stratum, wit i { 1,,, N}. N and n denote te total number of elements in te population and te sample size in te t stratum, N 1 N and n 1 population. Te population mean will be denoted by N W N N and y y i 1 1 i N n, were is te total number of strata in te y st 1 W y, were. Te unbiased estimator of te population mean is computed as y ˆ st W yˆ, were yˆ y i i n 1. Te population variance witin strata sall be denoted by s U, its simple estimator as s ˆ and te variance of te stratified mean estimator, using witin strata, will be represented by v strs W ( 1 n N ) s n. Te formula for te variance estimator of tis 1 quantity is vˆ mae wit U s ˆ in place of n s U in te formula for v. 3. Example of sample size determination using deff As it was mentioned above, te design effect, efd K, Kis (1965), is defined as te ratio of te variance of an estimator under a specific design different from simple random v ˆ, to te variance of te estimator under simple random sampling sampling, alt alt witout replacement, following formula deff ˆ v ˆ v ˆ, wit ˆ 0 v ˆ. In tis way, te design effect is computed wit te K alt alt v. During te planning stage of a survey or a sampling plan, design effects are widely used to compute sample sizes. In tis section, an example of sample size determination from an actual consumer survey will be presented. 604
JSM 013 - Survey Researc Metods Section Example 1: sample size computation using deff. In México, INEGI, te official statistics agency determines te sample size for different surveys using te design effect. In particular, for te 008 ouseold Income and Consumption Survey, INEGI used as te target variable to compute te sample size, te ouseold mean current income. Tey employed te following expression, for te adjusted sample size, n: In tis equation: n r X z s deff (1 tnr)pv (1) z α = te αt quantile of a standard normal distribution, INEGI used a two-sided 90% confidence level. s = estimation of te population variance between elements = 1 767,586,178. X = estimation of te mean for te ouseold mean current income = 34,17 mexican pesos. deff = design effect estimator. r = maximum relative aceptable error = 4%. tnr = maximum nonresponse rate = 15%. PV = mean number of ouseolds per dwelling = 1.0. Wit all tese values and using formula (1), te adjusted sample size was 9,711 dwellings, wic was rounded to 10,000. Te design effect, te estimator of te population variance witin elements and te mean for te ouseold mean current income were obtained from te 006 ouseold Income and Consumption Survey. It was required tat te estimations of te variables of interest old for some states, so te final sample size was 35,146 dwellings. Formula (1) can be expressed as follows: n r z s DEFF z s 1 1 DEFF n DEFF X ( 1 tnr ) PV r X ( 1 tnr ) PV ( 1 tnr ) PV () In tis equation, n stands for te sample size obtained by, wic is easy to compute, see Cocran (1977). From te rigt side in equation () it can be easily seen tat te design effect increases, decreases or left unaffected te sample size obtained by. 4. Variance estimation of te design effect As it was mentioned in te introduction, to te autor s best knowledge tere is no variance estimator of te design effect. In survey sampling, te standard approac to obtain a variance estimator of an estimator like a mean, total or te design effect is as follows. First, one as to build an expression for te population variance of te estimator, in tis case te design effect, ten one as to find an estimator of te population variance. In te literature tere are no expressions for te variance and variance estimators for te design effect, but it s posible to estimate te variance using a resampling metod. In Cauduri & Stenger (005) one can find eigt type of bootstraps for samples extracted 605
JSM 013 - Survey Researc Metods Section from finite populations. Some of tese metods were proposed by Sitter (199) and one of tem is te bootstrap used in tis article. Sitter (199) proposed bootstrap estimators for tree sample designs: a) Stratified random sampling wit selection of elements witin strata. b) Two stage cluster sampling wit equal or unequal sizes. c) Te Rao-artley-Cocran metod for probability proportional to size sampling, see Rao et al. (196). In tis article, Sitter proposed tree metods to build confidence intervals. One of tem is te percentile metod, used in tis paper due to its simplicity. Anoter metod is a double bootstrap, a computer intensive metod in wic te bootstrap is applied two times to eac sample. Te tird metod, used by Sitter in its article, is a jackknife estimator of te variance applied to te sample and to te replicated subsample. Te last two metods require a special study in order to compare tem wit te percentile metod and to explore some aspects related to te required number of bootstrap samples tat is closed to te nominal confidence for te design effect estimations. In tis work we use te metod proposed by McCarty & Snowden, see Cauduri & Stenger (005), wic is a special case of te extended bootstrap of Sitter (199) for stratified random samples. Tis metod is described next. Witout considering te fractional part of n ' n (1 f ) wit f n N, te metod is applied as follows: and k 1 f (1 n a) Replicate ( y 1,, y n, ) k times in a separate and independent way, =1,,, to create different pseudo-strata. b) Extract a s of size n from te t pseudo-stratum and repeat independently tis procedure for every =1,,, creating in tis way te bootstrap observations s {( y1,, yn '), 1,, } and let ˆ ˆ( s.) c) Repeat stage (b) a large number of times, say B, and compute for every bt bootstrap sample, ˆ, b b 1,, B. Wit te B estimators at and ˆb, compute te following quantities: 1 B ˆ B ˆ y b1 b B (3) 1 B ˆv BWO ( ˆ ˆ b1 b B ) B 1 Tese two expressions give us te bootstrap estimators of a mean or total and its variance. Te variance estimator BWO can also be used as a variance estimator for te estimator of te original sample. BWO stands for bootstrap in te case of sampling witout replacement. In tis article, ˆb may refer to a stratified estimator of te mean, a ratio estimator, like te design effect estimator, o a variance estimator, like te variance under strs or. N n ) 606
JSM 013 - Survey Researc Metods Section We use Sitter s extended bootstrap, due to its ease of implementation, compared to te metods described in Cauduri & Stenger (005). Noneteless, we are not claiming tat it as better performance tan oter bootstrap metods for estimating te variance of te design effect. Tere is no closed expression available for te variance of te estimator of te design effect, deff G, so we will illustrate ow to compute te bootstrap estimator of te variance in a small stratified population. Example : stratified random sampling. Based on Cocran s (1977) example, page 137, we simulated a small population wit 10 elements and 5 strata. Tables 1 and table below contain summary values of te simulated population and te population variances under strs, and te design effect. Table 1: Simulated population values Stratum 1 N 13 n 9 W 0.11.33 s 1.6 18 7 0.15 1.61 0.08 3 6 6 0. 5.04 1.18 4 6 10 0. 7.01 3.06 5 37 8 0.31 9.86 0.31 Population 10 3.44 Table : Population variances for strs and as well as te design effect Population quantity Value v 0.176 vstrs 0.0196 deffk 0.1114 y U In tis example te following steps were applied to simulate Sitter s bootstrap: a) We drew 5,000 samples, strs, of size 40 from te stratified population. b) For every strs, we simulate B=,000 bootstrap samples wit te above mentioned Sitter s metod. Tis value was used upon recommendation in Stuart et al. (1999) for variance estimation in te case of independent random variables. c) Te design effect estimator was computed using deff G, see Gambino (009), as well as wit te bootstrap. For every strs, te variance estimator of te design effect was obtained from, Cauduri & Stenger (005) or Sitter (199). vˆbwo To te autor s best knowledge, Sitter s bootstrap metods are not available in statistical softwares, so we wrote programs in R, R Development Core Team (010), to extract te samples and apply te bootstrap. Te 5,000 strs samples were drew in R using library pps, Gambino (005) and te 95% two-sided intervals for te population design effect were obtained from te bootstrap istogram built wit te estimated design effects. Wit tis metod and for every strs, te.5% and 97.5% percentiles of every bootstrap 607
JSM 013 - Survey Researc Metods Section istogram are found, so one can determine weter or not te population design effect is contained inside te interval. Tis was done for te 5,000 strs samples, counting te number of times te interval contained te population design effect and dividing by 5,000. In tis way, te coverage for te deff G estimators were obtained. Te results of te simulation are found in te next table. Table 3: Results from te 5,000 strs samples to estimate te variance of te design effect,,000 bootstrap simulations were generated for every strs Estimator Average of estimators (A) Population Value (B) Difference (%) = (A-B)/B Population mean strs BWO 6.144 6.143 0.0 Vstrs BWO 0.018 0.0196-7.60 V BWO 0.176 0.176-0.10 deff BWO 0.103 0.111-7.80 deff G strs 0.11 0.111 0.30 Std deviation deff BWO 0.01 Std deviation deff G strs 0.018 Coverage deff G strs= 90% In table 3, te estimators followed by BWO were obtained from te bootstrap simulations. In te above table, te bootstrap estimators of te population mean, population mean strs BWO, and te variance under, V BWO, ad a relative difference smaller tan 1% compared to te correspondent population value. On te oter and, te bootstrap estimators of te population value under strs, Vstrs BWO, and te population design effect efd K, efd BWO, subestimate by 8% te correspondent population value. Te deff estimator, deff G, ad a small bias, te relative difference to te design effect of te population, efd K, was 0.3%. It can be seen from table 3 tat te square root of te bootstrap estimator of te variance of te design effect as a value of 0.01 wic can be expressed as a coefficient of variation of 18.9%. Tere is a certain amount of variation around te population design effect, but it is not so bad considering te small population and sample size. In order to compare te result obtained from te variance estimator of te design effect generated by te bootstrap, we also compute te variance between te 5,000 deff G estimators. Te square root of tis estimator is 0.018 compared to 0.01 obtained wit te bootstrap. Te coverage, end of table 3, for te design effect obtained wit te bootstrap turned out to be approximately 90%, wic is below from te nominal of 95%. Sitter (199) mentioned tat te coverage can be improved using a metod different from te percentile; noneteless, we found a better coverage by increasing te number of bootstrap samples for every strs. Te first simulations were run wit values of B similar to tose recommended by Sitter in its article, Sitter used B=300, but Stuart et al. (1999), mentioned tat at least,000 bootstrap simulations in te case of variance estimation for random samples are required. Wit tis number of bootstrap simulations, te coverage improved substantially compare 608
JSM 013 - Survey Researc Metods Section to te number of simulations proposed by Sitter. We did not find an article or a reference for te number of simulations recommended for te bootstrap in case of complex simple designs. Below is a istogram for te 5,000 strs samples of te estimator deff G. Figure 1: 5,000 estimators of deff G In tis istogram, te red line corresponds to te population value of te design effect wic is 0.1114. It seems to be a sligt skewness; unfortunately, to te autor s best knowledge, te distribution of te ratio of variance estimates like deff G is not known for finite population sampling. Tis was a simulation exercise, but in practice we only ave one sample and te B bootstrap subsamples from te original strs. Wit te strs we compute te estimator of te design effect, deff G, and wit te B bootstrap subsamples we generate a istogram in order to obtain lower and upper limits from it wit te percentile metod. Tese values give us te interval for te design effect. In figure tere is a istogram of te design effect estimators built wit te B=,000 bootstrap subsamples obtained from te last strs sample of size n=40. Tis sample was selected just to illustrate a istogram one would build in practice. In tis sample te estimator of te design effect is deff G =0.116 and te square root of te variance between te bootstrap estimators of te design effect is equal to 0.0. 609
JSM 013 - Survey Researc Metods Section Figure :,000 bootstrap replicates of one strs sample Te red line corresponds to te population value of te design effect, 0.1114, and te blue one to te deff BWO estimator based on,000 bootstrap replicates. In tis case, te bootstrap estimated variance was 0.00049 or a standard deviation of 0.0. It is wort noting tat we ave not investigated te causes of te asymmetry in figure and it s a topic for future researc. 5. Conclusions We proposed to use te bootstrap for finite population samples, one of Sitter s metods (199), to obtain a variance estimation of te design effect. Te metod was illustrated in a small stratified population. Te results from te simulations suggest te feasibility to estimate te variance of te design effect from te bootstrap istogram. We used te estimator of te design effect proposed by Gambino (009), wic provides an unbiased estimator of te variance of. It is necessary to improve te coverage of te bootstrap istogram. Tis can be done wit Sitter s (199) recommendations of variants of te bootstrap and it is a work for future researc. At te time tis work was done, Sitter s bootstraps were not available in R or in te R survey library, Lumley (010), so we ad to program it in R. References Cauduri, A. & Stenger,. (005) Survey Sampling: teory and metods, nd edn., Capman & all/crc. Cocran, W.G. (1977) Sampling Tecniques, 3rd edn. New York: Wiley. Gambino, J.G. (009) Design effects caveat, Te American Statistician, pp. 141-145. 610
JSM 013 - Survey Researc Metods Section Gambino, J.G. (005) pps: Functions for PPS sampling. R package version 0.94. orvitz, D.G. & Tompson, D. J. (195) A generalization of sampling witout replacement from a finite universe, Journal of te American Statistical Association 47, pp. 663-685. INEGI, Encuesta Nacional de Ingresos y Gastos de los ogares 008. Diseño Muestral. Kis, L. (1965) Survey Sampling, New York: Wiley & Sons. Lumley, T. (010) Survey: analysis of complex survey samples. R package version 3.3-3. Padilla, A.M., Una cota para el sesgo relativo del efecto del diseño, Memorias electrónicas en extenso de la 4ª Semana Internacional de la Estadística y la Probabilidad. Julio 011, CD ISBN: 978-607-487-34-5. Padilla, A.M., A bound for te relative bias of te design effect. ICES IV, Fourt International Conference on Establisment Surveys, June 11-14, 01, Montréal, Québec, Canada. Padilla Terán, A.M., El efecto del diseño: sesgo y estimación de varianza. Documentos de Investigación 01-18, Banco de México. R Development Core Team (010). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051- 07-0, URL ttp://www.r-project.org. Rao, J.N.K. (196) On te estimation of te relative efficiency of sampling procedures, Annals of te Institute of Statistical Matematics, pp. 143-150. Rao, J.N.K., artley,.o. & Cocran, W.G. (196) On a simple procedure of unequal probability sampling witout replacement, Journal of te Royal Statistical Society B 4, pp. 48-491. Särndal, C.E., Swensson, B. & Wretman, J.. (199) Model Assisted Survey Sampling, Springer-Verlag, New York, 199. Sitter, R.R., (199) A resampling procedure for complex survey data, Journal of te American Statistical Association, Vol. 87, pp. 755-765. Stuart, A., Ord, K. & Arnold (1999) S. Kendall s Advanced Teory of Statistics (sixt edn). Volume ª, Classical Inference and te Linear Model. Edward Arnold, London. Tompson, M.E. (1997). Teory of Sample Surveys. Capman & all, London. 611