Purpose: The purpose of this standard is to ensure that statistically sound practices are used for producing direct estimates from samples for information products.

Scope: The Census Bureau’s statistical quality standards apply to all information products released by the Census Bureau and the activities that generate those products, including products released to the public, sponsors, joint partners, or other customers. All Census Bureau employees and Special Sworn Status individuals must comply with these standards; this includes contractors and other individuals who receive Census Bureau funding to develop and release Census Bureau information products.

In particular, this standard applies to the production of direct estimates from samples and estimates of their variances for Census Bureau information products. The standard applies to estimates derived from:

• Samples selected for surveys or the Economic Census.
• Samples or subsamples selected for data analyses, evaluations, or quality assessments of surveys, censuses, or programs using administrative records.

Exclusions:
In addition to the global exclusions listed in the Preface, this standard does not apply to:

• 100 percent enumerations.
• Activities related to producing estimates from models. (See Statistical Quality Standard D2, Producing Estimates from Models.)

Key Terms: Calibration, coefficient of variation (CV), coverage error, cross-sectional studies, direct estimates, estimation, generalized variance function, imputation, longitudinal studies, post-stratification, raking, ratio estimation, replication methods, sanitized data, and Taylor series method for variance estimation.

Requirement D1-1: Throughout all processes associated with estimation, unauthorized release of protected information or administratively restricted information must be prevented by following federal laws (e.g., Title 13, Title 15, and Title 26), Census Bureau policies (e.g., Data Stewardship Policies), and additional provisions governing the use of the data (e.g., as may be specified in a memorandum of understanding or data-use agreement). (See Statistical Quality Standard S1, Protecting Confidentiality.)

Requirement D1-2: A plan must be developed that addresses:

1. Key estimates that will be produced.
2. Estimation methodologies (e.g., population controls, post-stratification, nonresponse adjustments, ratio estimation, calibration, and raking).
3. Variance estimation methodologies (e.g., sampling formula variances, Taylor series (linearization) methods, replication methods, and generalized variance functions).
4. Verification and testing of the systems for generating estimates.
5. Verification of the estimates and evaluation of their quality.

Note: Statistical Quality Standard A1, Planning a Data Program, addresses overall planning requirements and development of schedules and costs.

Requirement D1-3: Estimates and their variances must be produced using statistically sound practices that account for the sample design and reduce the effects of nonresponse and coverage error.

Examples of statistically sound practices include:

• Calculating estimates and variances in ways that take into account the probabilities of selection, stratification, and clustering.
• Developing generalized variance formulas for computing variances.
• Using auxiliary data or performing post-sampling adjustments to improve the precision and the accuracy of estimates (e.g., ratio or raking weighting adjustments for unit nonresponse and post-stratification).
• Accounting for post-sampling adjustments when computing variances (e.g., imputation effects in variance estimates).
• Generating weights or adjustment factors to allow both cross-sectional and longitudinal estimates for longitudinal surveys.

Note: Statistical Quality Standard A3, Developing and Implementing a Sample Design, specifies requirements for the design and selection of probability samples used to produce estimates or make inferences.

Sub-Requirement D1-3.1: Specifications for the estimation systems must be developed and implemented.

Examples of issues that specifications might address include:

• Methodological requirements for generating the estimates and variances.
• Data files used or saved during the estimation process (e.g., files used for program validation, verification, and research).

Sub-Requirement D1-3.2: Estimation systems must be verified and tested to ensure that all components function as intended.

Examples of verification and testing activities include:

• Verifying that specifications conform to the estimation methodologies.
• Validating computer code against specifications.
• Verifying that the estimates are computed according to the specifications.
• Using subject matter and statistical experts to review the estimation methodology.
• Conducting peer reviews (e.g., reviews of specifications, design documents, and programming code).
• Conducting verification and validation tests.
• Conducting internal user acceptance tests for estimation software.

Sub-Requirement D1-3.3: Methods and systems must be developed and implemented to verify the estimates and evaluate their quality.

Examples of verification and evaluation activities include:

• Comparing current estimates against historical results.
• Comparing the estimates derived from the survey to other independent collections of similar data.
• Comparing coefficients of variation (CVs) or variances of the estimates against historical results.
• Examining relationships among the estimates.
• Conducting studies to evaluate the performance of variance estimates.

Note: Statistical Quality Standard D3, Producing Measures and Indicators of Nonsampling Error, provides requirements for measuring and evaluating nonsampling error.

Requirement D1-4: Documentation needed to replicate and evaluate the estimation operations must be produced. The documentation must be retained, consistent with applicable policies and data-use agreements, and must be made available to Census Bureau employees who need it to carry out their work. (See Statistical Quality Standard S2, Managing Data and Documents.)

Examples of documentation include:

• Plans, requirements, specifications, and procedures for the estimation systems.
• Final weighting specifications, including calculations for how the final sample weights are derived.
• Final variance estimation specifications.
• Computer source code.
• Data files with weighted data and any design parameters that would be needed to replicate estimates and variances.
• Methodological documentation.
• Quality measures and evaluation results. (See Statistical Quality Standard D3, Producing Measures and Indicators of Nonsampling Error.)

Notes:

1. The documentation must be released on request to external users, unless the information is subject to legal protections or administrative restrictions that would preclude its release. (See Data Stewardship Policy DS007, Information Security Management Program.)
2. Statistical Quality Standard F2, Providing Documentation to Support Transparency in Information Products, contains specific requirements about documentation that must be readily accessible to the public to ensure transparency of information products released by the Census Bureau.