U.S. Department of Commerce

Monthly & Annual Retail Trade

You are here: Census.govBusiness & IndustryRetailMonthly & AnnualHow the Data are Collected › Annual Retail Trade Survey Methodology
Skip top of page navigation

Annual Retail Trade Survey Methodology

Overview

The Census Bureau introduced a new sample with the 2011 Annual Retail Trade Survey (ARTS). This new sample is designed to produce estimates based on the 2007 North American Industry Classification System (NAICS). This section describes the design, selection, and estimation procedures for the new samples. For descriptions of the prior samples, see prior ARTS releases.

The U.S. Census Bureau conducts the Annual Retail Trade Survey (ARTS) to produce national estimates of total annual sales, e-commerce sales, end-of-year inventories, purchases, total operating expenses, inventories held outside the United States, gross margins, and end-of-year accounts receivable for retail businesses and annual sales and e-commerce sales for accommodation and food service firms located in the United States. Estimates are summarized by industry classification based on the North American Industry Classification System (NAICS) and are released approximately 15 months after the reference year. Firms without paid employees, or nonemployers, are included in the estimates through imputation or administrative data provided by other federal agencies.

Sampling Frame

The sampling frame used for the ARTS has two types of sampling units: Employer Identification Numbers (EINs) and large, multiple-establishment firms. Both sampling units represent clusters of one or more establishments owned or controlled by the same firm. The information used to create these sampling units was extracted from data collected as part of the 2007 Economic Census and from establishment records contained on the Census Bureau's Business Register as updated to December 2010. The next few paragraphs give details about the Business Register; the distinction between firms, EINs, and establishments; and the construction of the sampling units. Though important, they are not essential to understanding the basic sample design and readers may continue to the Stratification, Sampling Rates, and Allocation section.

The Business Register is a multi-relational database that contains a record for each known establishment that is located in the United States or one of its territories and has paid employees. An establishment is a single physical location where business transactions take place and for which payroll and employment records are kept. Groups of one or more establishments under common ownership or control are firms. A single-unit firm owns or operates only one establishment. A multiunit firm owns or operates two or more establishments. The treatment of establishments on the Business Register differs according to whether the establishment is part of a single-unit or multiunit firm. In particular, the structure of an establishment's primary identifier on the Business Register differs depending on whether it is owned by a single-unit firm or by a multiunit firm.

A single-unit firm's primary identifier is its EIN. The Internal Revenue Service (IRS) issues the EIN, and the firm uses it as an identifier to report social security payments for its employees under the Federal Insurance Contributions Act (FICA). The same act requires all employer firms to use EINs. Each employer firm is associated with at least one EIN and only one firm can use a given EIN. Because a single-unit firm has only one establishment, there is a one-to-one relationship between the firm and the EIN. Thus the firm, the EIN, and the establishment all reference the same physical location and all three terms can be used interchangeably and unambiguously when referring to a single-unit firm.

For multiunit firms however, a different structure connects the firm with its establishments via the EIN. Essentially a multiunit firm is associated with a cluster of one or more EINs and EINs are associated with one or more establishments. A multiunit firm consists of at least two establishments. Each firm is associated with at least one EIN and only one firm can use a given EIN. However, one multiunit firm may have several EINs. Similarly, there is a one-to-many relationship between EINs and establishments. Each EIN can be associated with many establishments but each establishment is associated with only one EIN. Because of the possibility of one-to-many relationships, we must distinguish between the firm, its EINs, and its establishments. The multiunit firm that owns or controls a particular establishment is identified on the Business Register by way of the establishment's primary identifier.

The primary identifier of an establishment owned by a multiunit firm consists of a unique combination of an alpha number and a plant number. The alpha number identifies the multiunit firm, and the plant number identifies a particular establishment within that firm. All establishments owned or controlled by the same multiunit firm have the same alpha number. Different multiunit firms have different alpha numbers, and different establishments within the same multiunit firm have different plant numbers. The Census Bureau assigns both the alpha number to the multiunit firm and plant numbers to the corresponding establishments based on the results of the quinquennial economic census and the annual Company Organization Survey.

To create the sampling frame, we extract the records for all employer establishments located in the United States and classified in the Retail Trade or Food Service and Accommodation sectors as defined by the 2007 NAICS. For these establishments we extract sales, payroll, employment, name and address information, as well as primary identifiers and, for establishments owned by multiunit firms, associated EINs. To create the sampling units for multiunit firms, we aggregate the economic data of the establishments owned by these firms to an EIN level by tabulating the establishment data for all retail establishments associated with the same EIN. Similarly, we aggregate the data to a multiunit firm level by tabulating the establishment data for all retail establishments associated with the same alpha number. No aggregation is necessary to put single-unit establishment information on an EIN basis or a firm basis. Thus, the sampling units created for single-unit firms simultaneously represent establishment, EIN, and firm information. In summary, the sampling frame is a complex amalgam of establishments, EINs, and firms.

Stratification, Sampling Rates, and Allocation

The primary stratification of the sampling frame is by industry group based on the detail required for publication. We further stratify the sampling units within industry group by a measure of size (substratify) related to their annual sales. Sampling units expected to have a large effect on the precision of the estimates are selected "with certainty." This means they are sure to be selected and will represent only themselves (i.e., have a selection probability of 1 and a sampling weight of 1). Within each industry stratum, we determine a substratum boundary (or cutoff) that divides the certainty units from the noncertainty units. We base these cutoffs on a statistical analysis of data from the 2007 Economic Census. Accordingly, these values are on a 2007 sales basis. We also used this analysis to determine the number of size substrata for each industry stratum and to set preliminary sampling rates needed to achieve specified sampling variability constraints on sales estimates for different industry groups. The size substrata and sampling rates are later updated through analysis of the sampling frame.

Sample Selection

The first step in the sample selection identified firms selected with certainty. If a firm's annual sales or end-of-year inventories were greater than the corresponding certainty cutoff, that firm was selected into the ARTS sample with certainty.

All firms not selected with certainty were subjected to sampling on an EIN basis. If a firm had more than one EIN, we treated each of its EINs as a separate sampling unit. To be eligible for the initial sampling, an EIN had to have nonzero payroll in 2009. The EINs were stratified according to their major industry and their estimated sales (on a 2007 basis). Within each noncertainty stratum, a simple random sample of EINs was selected without replacement. The selected noncertainty EINs were divided into two approximately equal groups. One group is canvassed for both the monthly and the annual survey, the other group is canvassed for only the annual survey.

Sample Maintenance

Periodically, we update the sample to represent EINs issued since the initial sample selection. These new EINs, called births, are EINs, recently assigned by the IRS, that have an active payroll filing requirement on the IRS Business Master File (BMF). An active payroll filing requirement indicates that the EIN is required to file payroll for the next quarterly period. The Social Security Administration attempts to assign industry classification to each new EIN.

EINs with an active payroll filing requirement on the IRS Business Master File are considered to be "BMF active" and EINs with an inactive payroll filing requirement are said to be "BMF inactive."

EIN births are sampled on a quarterly basis using a two-phase selection procedure. To be eligible for selection, a birth must either have no industry classification or be classified in an industry within the scope of the Service Annual Survey, the Annual Wholesale Trade Survey, or the Annual Retail Trade Survey, and it must meet certain criteria regarding its quarterly payroll. In the first phase, births are stratified by broad industry groups and a measure of size based on quarterly payroll. A relatively large sample is selected using equal probability systematic sampling within the payroll strata. The selected births are canvassed to obtain a more reliable measure of size, consisting of sales in two recent months, company affiliation information, and a new or more detailed industry classification code. Births that have not returned their questionnaire after 30 days are contacted by telephone.

Using this more reliable information, the selected births from the first phase are subjected to probability proportional-to-size sampling with overall probabilities equivalent to those used in drawing the initial ARTS sample from the Business Register. Because of the time it takes for a new employer firm to acquire an EIN from the IRS, and because of the time needed to accomplish the two-phase birth-selection procedure, births are added to the samples approximately nine months after they begin operation.

Births that are selected in the quarterly birth-selection procedure in November of the annual survey reference year are included in the initial mailing of the annual survey questionnaires in January of the following year. To better represent all EIN births in the reference year, and specifically to account for the lag between the time a business starts operation and the time it takes to acquire an EIN and identify and select the EIN into the ARTS sample, births are added to the annual survey sample that are selected in February, May, and August of the year following the annual survey reference year. We mail annual survey forms to these births in June and August to supplement the initial annual survey mailings.

To be eligible for the sample canvass and tabulation, an EIN selected in the noncertainty sampling operations must meet both of the following requirements:

  • It must have an active payroll filing requirement on the IRS Business Master File.
  • It must have been selected from the Business Register in either the initial sampling or during the quarterly birth-selection procedure.

If a firm was selected with certainty and had more than one establishment at the time of initial sampling, any new establishments that the firm acquires, even if under new or different EINs, are included in the sample with certainty. However, if a single-unit firm was selected with certainty, only future establishments associated with that firm's originally-selected EIN are included in the sample with certainty; any new EINs that might later be associated with that firm are subjected to sampling through the quarterly birth-selection procedure.

Each quarter, we check against the current Business Register to determine if any EINs on ARTS have become BMF inactive. Typically, we do not canvass BMF inactive EINs during the reference year. Likewise, if any EIN on ARTS that was BMF inactive in a previous reference year is now BMF active on the current Business Register, we again include these EINs in the canvass. In both cases, we only tabulate data for that portion of the reference year that these EINs reported payroll to the IRS.

Singleunit EINs selected into the sample with certainty are not dropped from canvass and tabulation if they are no longer on the IRS mailing list. Rather, the firm that used the EIN is contacted, and if a successor EIN is found, it is added to the survey. For both inactive and reactivated EINs, data are tabulated for only the portion of the reference year that these EINs reported payroll to the IRS.

Estimation and Sampling Variance

Total estimates are computed using the Horvitz-Thompson estimator (i.e., as the sum of weighted data (reported or imputed) for all selected sampling units that meet the sample canvass and tabulation criteria). The weight for a given sampling unit is the reciprocal of its probability of selection into the ARTS sample. These estimates are input to a benchmarking procedure, as described below. Variances are estimated using the method of random groups and are used to determine if measured changes are statistically significant.

Historical Estimates

Annual estimates prior to 1999 are derived from data that were collected and published based on the Standard Industrial Classification (SIC) system. For a description of how these estimates were derived, see: http://www2.census.gov/retail/releases/benchmark/annpub00.pdf

Because of the method used to derive annual estimates prior to 1999, these estimates should be used with caution. It is expected that for estimates for NAICS codes, that, by definition, are the same or nearly the same as a given SIC code, the quality of the estimates will be similar to that of the estimates released on an SIC basis. Estimates may be of less quality for NAICS codes that consist of more than one SIC component. Additionally, for reference years further from 1997, estimates are likely of less quality than for those years close to 1997. Note, however, that estimated year-to-year changes for 1992 through 1998 are dependent on the underlying SIC-based year-to-year changes. Year-to-year changes for 1999 and subsequent years are derived from data collected on a NAICS basis.

Special caution should be exercised when using the end-of-year retail inventory estimates prior to 1999. Retail inventory data has historically been analyzed at much broader industry levels than for sales. Determining clear relationships between NAICS and SIC codes was much more difficult at broader levels.

Non-employers

Firms without paid employees (nonemployers) are included in the ARTS estimates through administrative data provided by other Federal agencies and through imputation. Imputed nonemployer estimates for reference year 2011 have been updated to reflect values published by the Nonemployer Statistics program where applicable. ARTS nonemployer estimates for reference year 2012 are imputed because values from the Nonemployer Statistics program are not yet available.

ARTS estimates of total and detailed expenses and accounts receivable are not affected by this revision as these estimates are based only on employer businesses.

Benchmarking

The current sample was introduced with the 2011 ARTS. This sample is designed to produce estimates based on the 2007 North American Industry Classification System (NAICS). In order to maintain the time series for each industry, an operation was performed to link estimates from the prior and new samples. For the linking operation to occur, two years of data were collected (2010 and 2011) from units in the new sample. The linking is done so that the new sample estimates are implicitly benchmarked using results of the 2007 Economic Census. For more information on how the samples were linked and what was changed between samples, see the methodology section of the archived 2011 ARTS release.

Linking Samples

Because the ARTS sample is selected from a sampling frame of firms with paid employees, the following methodology, which is used to link the samples, is applied to employer-only estimates. Then, the published ARTS estimates are obtained by summing the estimates for employers and nonemployers. See the "Nonemployers" section for more details.

Sales estimates from the new sample for reference year 2010 and subsequent years are linked to the prior sample estimates by multiplying the Horvitz-Thompson estimates from the new sample by a ratio. The ratio is calculated as follows:

  • The numerator is the 2010 census-adjusted sales estimate for employer firms for the industry on a 2007 NAICS basis from the prior sample.
  • The denominator is the 2010 Horvitz-Thompson sales estimate for employer firms for the industry on a 2007 NAICS basis from the new sample.

The resulting sales estimates (call these "modified" sales estimates) are implicitly benchmarked to 2007 Economic Census results via this linking procedure.

The following method is used to produce "modified" estimates for the following items: end-of-year inventories and purchases. First the sales ratio described above is multiplied by the Horvitz-Thompson estimate for the given item for 2010 and subsequent years. Then the estimates for 2004 through 2010 from the prior sample are input into the benchmarking program. Using this program, the estimates for 2005 through 2010 for each detailed industry are revised in a manner that:

  • Uses the benchmarked estimate for 2004 from the prior sample as a constraint, resulting in no revision to the 2004 estimate.
  • Uses the 'modified' estimate for 2010 from the new sample as a constraint
  • Minimizes the sum of squared differences between the year-to-year changes of the input and revised estimates for 2005 through 2010.

A similar method is used to produce "modified" estimates for total expenses and inventories held outside the U.S. For total expenses, the benchmarked estimate for 2006 from the prior sample is used as a constraint because total expenses were not collected as part of ARTS for 2004 or 2005. For inventories held outside the U.S., the estimates are benchmarked at the 3-digit NAICS aggregate industry levels because detail industry estimates can be very small. Inventories held inside the U.S. are calculated by subtraction from "modified" total inventory.

Sales tax estimates for 2010 and subsequent years are "modified" by first multiplying the Horvitz-Thompson estimate of sales tax by the sales ratio described above. Estimates for 2009 and earlier years are linked to the new sample estimates by multiplying the Horvitz-Thompson sales tax estimates from the prior sample by a ratio. The ratio is calculated for each detailed industry as follows:

  • The numerator is the "modified" 2010 sales tax estimate for the industry on a 2007 NAICS basis from the new sample.
  • The denominator is the 2010 Horvitz-Thompson sales tax estimate for the industry on a 2007 NAICS basis from the prior sample.

Total accounts receivable estimates for 2009 and earlier years are linked to the new sample estimates by multiplying the Horvitz-Thompson accounts receivable estimates from the prior sample by a ratio. The ratio is calculated as follows:

  • The numerator is the 2010 Horvitz-Thompson accounts receivable estimate for the industry on a 2007 NAICS basis from the new sample.
  • The denominator is the 2010 Horvitz-Thompson accounts receivable estimate for the industry on a 2007 NAICS basis from the prior sample.

Open-end and Closed-end accounts receivable estimates are raked to the "modified" total accounts receivable to ensure they sum properly to the total.

For e-commerce estimates, other than NAICS 4541 (Electronic Shopping and Mail-Order Houses), the following method is used to produce "modified" estimates. First the sales ratio described above is multiplied by the Horvitz-Thompson e-commerce estimate for 2010 and subsequent years. Because these e-commerce estimates tend to make up a small percentage of total e-commerce and are subject to larger dollar volume revisions across samples, a longer revision span was chosen to better minimize the revisions to the year-to-year percent changes. The estimates for 1998 through 2010 from the prior sample are input into the benchmarking program. Using this program, the estimates for 1999 through 2010 for each detailed industry are revised in a manner that:

  • Uses the benchmarked estimate for 1998 from the prior sample as a constraint, resulting in no revision to the 1998 estimate.
  • Uses the 'modified' estimate for 2010 from the new sample as a constraint.
  • Minimizes the sum of squared differences between the ratios of input and revised estimates for 1999 through 2010, which results in an approximation of the preservation of the year-to-year percent changes.

For NAICS 4541 (Electronic Shopping and Mail-Order Houses) e-commerce estimates, the above methodology is applied using the benchmarked estimate for 2004 from the prior sample as a constraint rather than 1998. Modified merchandise lines sales and e-commerce estimates within the NAICS 4541 industry group for 2005 and subsequent years are then obtained using methodology similar to that used for inventories and purchases.

Modified estimates at aggregate industry levels are computed by summing the modified estimates for the appropriate detailed industries comprising the aggregates.

Reliability of the Estimates

The published estimates may differ from the actual, but unknown, population values. For a particular estimate, statisticians define this difference as the total error of the estimate. When describing the accuracy of survey results, it is convenient to discuss total error as the sum of sampling error and nonsampling error. Sampling error is the error arising from the use of a sample, rather than a census, to estimate population values. Nonsampling error encompasses all other factors that contribute to the total error of a sample survey estimate. The sampling error of an estimate can usually be estimated from the sample; whereas, the nonsampling error of an estimate is difficult to measure and can rarely be estimated. Consequently, the actual error in an estimate exceeds the error that can be estimated. Further descriptions of sampling error and nonsampling error are provided in the following sections. Data users should take into account the estimates of sampling error and the potential effects of nonsampling error when using the published estimates.

Sampling Error

Because the estimates are based on a sample, exact agreement with results that would be obtained from a complete enumeration of firms on the sampling frame using the same enumeration procedures is not expected. However, because each firm on the sampling frame has a known probability of being selected into the sample, it is possible to estimate the sampling variability of the survey estimates.

The particular sample used in this survey is one of a large number of samples of the same size that could have been selected using the same design. If all possible samples had been surveyed under the same conditions, an estimate of a population parameter of interest could have been obtained from each sample. For the parameter of interest, estimates derived from the different samples would, in general, differ from each other. Common measures of the variability among these estimates are the sampling variance, the standard error, and the coefficient of variation (CV). The sampling variance is defined as the squared difference, averaged over all possible samples of the same size and design, between the estimator and its average value. The standard error is the square root of the sampling variance. The CV expresses the standard error as a percentage of the estimate to which it refers. For example, an estimate of 200 units that has an estimated standard error of 10 units has an estimated CV of 5 percent. The sampling variance, standard error, and CV of an estimate can be estimated from the selected sample because the sample was selected using probability sampling. Note that measures of sampling variability, such as the standard error and CV, are estimated from the sample and are also subject to sampling variability. (Technically, we should refer to the estimated standard error or the estimated CV of an estimator. However, for the sake of brevity we have omitted this detail.) It is important to note that the standard error and CV only measure sampling variability. They do not measure any systematic biases in the estimates.

The Census Bureau recommends that individuals using published estimates incorporate this information into their analyses, as sampling error could affect the conclusions drawn from these estimates.

The estimate from a particular sample and its associated standard error can be used to construct a confidence interval. A confidence interval is a range about a given estimator that has a specified probability of containing the average of the estimates for the parameter derived from all possible samples of the same size and design. Associated with each interval is a percentage of confidence, which is interpreted as follows. If, for each possible sample, an estimate of a population parameter and its approximate standard error were obtained and using a t-statistic with 15 (16 random groups - 1) degrees of freedom, then:

  1. For approximately 90 percent of the possible samples, the interval from 1.753 standard errors below to 1.753 standard errors above the estimate would include the average of the estimates derived from all possible samples of the same size and design.
  2. For approximately 95 percent of the possible samples, the interval from 2.131 standard errors below to 2.131 standard errors above the estimate would include the average of the estimates derived from all possible samples of the same size and design.

To illustrate the computation of a confidence interval for an estimate of total sales, assume that an estimate of total sales is $10,750 million and the CV for this estimate is 1.8 percent, or 0.018. First obtain the standard error of the estimate by multiplying the total sales estimate by its CV. For this example, multiply $10,750 million by 0.018. This yields a standard error of $193.5 million. The upper and lower bounds of the 90-percent confidence interval are computed as $10,750 million plus or minus 1.75 times $193.5 million. Consequently, the 90-percent confidence interval is $10,411 million to $11,089 million. If corresponding confidence intervals were constructed for all possible samples of the same size and design, approximately 9 out of 10 (90 percent) of these intervals would contain the average of the estimates derived from all possible samples.

Nonsampling Error

Nonsampling error encompasses all other factors, other than sampling error, that contribute to the total error of a sample survey estimate and may also occur in censuses. It is often helpful to think of nonsampling error as arising from deficiencies or mistakes in the survey process. Nonsampling errors are difficult to measure and can be attributed to many sources: the inclusion of erroneous units in the survey (overcoverage), the exclusion of eligible units from the survey (undercoverage), nonresponse, misreporting, mistakes in recording and coding responses, misinterpretation of questions, and other errors of collection, response, coverage, or processing. Although nonsampling error is not measured directly, the Census Bureau employs quality control procedures throughout the process to minimize this type of error.

A potential source of bias in the estimates is nonresponse. Nonresponse is defined as the inability to obtain all the intended measurements or responses about all selected units. Two types of nonresponse are often distinguished. Unit nonresponse is used to describe the inability to obtain any of the substantive measurements about a sampled unit. In most cases of unit nonresponse, the questionnaire was never returned to the Census Bureau after several attempts to elicit a response. Item nonresponse occurs either when a question is unanswered or the response to the question fails computer or analyst edits.

Economic surveys at the Census Bureau are required to compute two different types of response rates: a unit response rate and weighted item response rates. Read more[PDF] about ARTS response rates (including the 2012 rates).

An estimate with a coefficient of variation (CV) greater than 30 percent or with a total quantity response rate (TQRR) less than 50 percent has been suppressed from publication, unless the estimate has consistently been published for prior years and the CV and TQRR are acceptably close to the thresholds. A suppressed estimate and its corresponding measure of sampling variability has been replaced with an "S" in the published tables. For a description of the Census Bureau's standards for Releasing Information Products, see http://www.census.gov/quality/standards/standardf1.html

Further explanation of the quality of the data and the estimates can be made available upon request.


[PDF] or PDF denotes a file in Adobe’s Portable Document Format. To view the file, you will need the Adobe® Reader® Off Site available free from Adobe. [Excel] or the letters [xls] indicate a document is in the Microsoft® Excel® Spreadsheet Format (XLS). To view the file, you will need the Microsoft® Excel® Viewer Off Site available for free from Microsoft®.
Source: U.S. Census Bureau | Monthly & Annual Retail Trade | (301) 763-2747 |  Last Revised: March 31, 2014