Purpose: The purpose of this standard is to ensure that measures and indicators of nonsampling error are computed and documented to allow users to interpret the results in information products, to provide transparency regarding the quality of the data, and to guide improvements to the program.
Scope: The Census Bureau’s statistical quality standards apply to all information products released by the Census Bureau and the activities that generate those products, including products released to the public, sponsors, joint partners, or other customers. All Census Bureau employees and Special Sworn Status individuals must comply with these standards; this includes contractors and other individuals who receive Census Bureau funding to develop and release Census Bureau information products.
In particular, this standard applies to activities associated with producing measures or indicators of nonsampling error associated with estimates for Census Bureau information products. Examples of nonsampling error sources include:
In addition to the global exclusions listed in the Preface, this standard does not apply to:
Key Terms: Convenience sample, coverage, coverage error, coverage ratio, equivalent quality data, item allocation rate, item nonresponse, key variables, latent class analysis, longitudinal survey, measurement error, nonresponse bias, nonresponse error, nonsampling error, probability of selection, quantity response rate, reinterview, release phase, respondent debriefing, response analysis survey, total quantity response rate, and unit nonresponse.
Requirement D3-1: Throughout all processes associated with producing measures and indicators of nonsampling error, unauthorized release of protected information or administratively restricted information must be prevented by following federal laws (e.g., Title 13, Title 15, and Title 26), Census Bureau policies (e.g., Data Stewardship Policies), and additional provisions governing the use of the data (e.g., as may be specified in a memorandum of understanding or data-use agreement). ( See Statistical Quality Standard S1, Protecting Confidentiality.)
Requirement D3-2: A plan must be developed that addresses:
Note: Statistical Quality Standard A1, Planning a Data Program, addresses overall planning requirements, including estimates of schedule and costs.
Requirement D3-3: Except in the situations noted below, weighted response rates must be computed to measure unit and item nonresponse. The weights must account for selection probabilities, including probabilities associated with subsampling for nonresponse follow-up.
Response rates may be computed using unweighted data when:
Note: In general, computing response rates is not appropriate for samples that are not randomly selected (e.g., convenience samples or samples with self-selected respondents).
Sub-Requirement D3-3.1: For demographic surveys and decennial censuses, when computing unit response rates, item response rates or item allocation/imputation rates (for key variables), and total item response rates:
Sub-Requirement D3-3.2:For economic surveys and censuses, when computing unit response rates, quantity response rates (for key variables), and total quantity response rates:
Sub-Requirement D3-3.3: Rates for the types of nonresponse (e.g., refusal, unable to locate, no one home, temporarily absent, language problem, insufficient data, or undeliverable as addressed) must be computed to facilitate the interpretation of the unit response rate and to better manage resources.
Sub-Requirement D3-3.4: For panel or longitudinal surveys, cumulative response rates must be computed using weighted data or cumulative total quantity response rates must be computed to reflect the total attrition of eligible units over repeated waves of data collection. If a survey uses respondents from another survey or census as its sampling frame, then the response rate of the survey (or census) serving as the frame must be included in the computation of the cumulative response rate.
Sub-Requirement D3-3.5: Cumulative response rates must be computed using weighted data over successive stages of multistage data collections (e.g., a screening interview followed by a detailed interview). If estimated probabilities of selection must be used and the accuracy of the response rate might be affected, then a description of the issues affecting the response rate must also be provided.
Note: In most situations, a simple multiplication of response rates for each stage is appropriate. In other situations, a more complex computation may be required.
Sub-Requirement D3-3.6: Nonresponse bias analyses must be conducted when unit, item, or total quantity response rates for the total sample or important subpopulations fall below the following thresholds.
Note: If response rates fall below these thresholds in a reimbursable data collection, the sponsor is responsible for conducting the nonresponse bias analysis.
Requirement D3-4: Coverage ratios must be computed to measure coverage error, as an indicator of potential bias, using statistically sound methods (e.g., computing coverage ratios as the uncontrolled estimate of population for a demographic-by-geographic group divided by the population control total for the demographic-by-geographic cell used in post-stratification adjustments or using capture-recapture methods).
Note: If computing coverage ratios is not appropriate, a description of the efforts undertaken to ensure high coverage must be made available.
Requirement D3-5: Measures or indicators of nonsampling error associated with data from administrative records must be computed to inform users of the quality of the data.
Examples of measures and indicators include:
Requirement D3-6: Measures or indicators of nonsampling error associated with data collection and processing activities must be computed to inform users of the quality of the data.
Examples of indicators of nonsampling error include:
Examples of analyses or studies that generate measures or indicators of nonsampling error include:
Requirement D3-7: Methods and systems for calculating measures and indicators of nonsampling error must be verified and tested to ensure all components function as intended.
Examples of verification and testing activities include:
Requirement D3-8: Measures and indicators of nonsampling error must be evaluated to guide improvements to the program.
Examples of evaluation activities include:
Requirement D3-9: Documentation needed to replicate and evaluate the activities associated with producing measures and indicators of nonsampling error must be produced. The documentation must be retained, consistent with applicable policies and data use agreements, and must be made available to Census Bureau employees who need it to carry out their work. (See Statistical Quality Standard S2, Managing Data and Documents.)
Examples of documentation include: