Statistical Quality Standard A3: Developing and Implementing a Sample Design
Purpose: The purpose of this standard is to ensure that statistically sound frames are designed and samples are selected to meet the objectives of the survey.
Scope: The Census Bureau's statistical quality standards apply to all information products released by the Census Bureau and the activities that generate those products, including products released to the public, sponsors, joint partners, or other customers. All Census Bureau employees and Special Sworn Status individuals must comply with these standards; this includes contractors and other individuals who receive Census Bureau funding to develop and release Census Bureau information products.
In particular, this standard applies to the design and selection of statistically sound samples used to produce estimates or make inferences. This standard covers:
- Frame development for censuses and sample surveys.
- The design and selection of samples or subsamples for surveys.
- The design and selection of samples or subsamples for secondary data analysis, evaluations, or quality assessments.
In addition to the global exclusions listed in the Preface, this standard does not apply to:
- Selection of focus groups.
- Cognitive interviewing.
- Samples that will not be used to produce estimates or make inferences (e.g., samples used for operational tests, pilot studies, or quality control).
- Frames and samples provided to the Census Bureau by a sponsor.
- Activities performed to produce sample estimates (e.g., weighting, estimation, and variance estimation). Statistical Quality Standard D1, Producing Direct Estimates from Samples, addresses requirements related to producing estimates.
Key Terms: Cluster, coverage, cut-off samples, estimate, estimation, frame, housing unit, peer review, precision, primary sampling unit (PSU), probability of selection, probability sampling, sample design, sample size, sampling frame, sampling weights, sequential sampling, strata, stratification, systematic sampling, target population, unduplication, variance, and weights.
Requirement A3-1: Throughout all processes associated with frame development and sample design, unauthorized release of protected information or administratively restricted information must be prevented by following federal laws (e.g., Title 13, Title 15, and Title 26), Census Bureau policies (e.g., Data Stewardship Policies), and additional provisions governing the use of the data (e.g., as may be specified in a memorandum of understanding or data-use agreement). (See Statistical Quality Standard S1, Protecting Confidentiality.)
Requirement A3–2: A plan must be developed that addresses:
- Statistical requirements of the program using the sample (e.g., the target population, the key estimates, the required precision of the estimates, and the expected response rates).
- Development of the sampling frame.
- Sampling methodologies that improve efficiency and minimize the costs of data collection (e.g., probability sampling, oversampling, stratification, sorting, unduplication requirements, and cluster sizes).
- Verification and testing of systems associated with the sampling operations.
- Monitoring and evaluating the accuracy of the frame and the sample (e.g., the coverage of the target population by the frames, timeliness of the frames, efficiency of stratification, and verification of the sample).
- The Census Bureau Guideline Sample Design and Selection identifies steps to follow and issues to consider when designing and selecting a sample.
- Statistical Quality Standard A1, Planning the Data Program, addresses overall planning requirements, including the development of schedules and costs
Requirement A3-3: Sampling frames that meet the data collection objectives must be developed using statistically sound methods.
Requirement A3–4: The sample design must be developed to meet the objectives of the survey, using statistically sound methods. The size and design of the sample must reflect the level of detail needed in tabulations and other information products and the precision required of key estimates. Any use of nonprobability sampling methods (e.g., cut–off) must be justified statistically.
Requirement A3-5: Sampling frames must be implemented and samples selected to ensure high quality data.
Sub–Requirement A3–5.1: Specifications and procedures for creating frames and selecting samples, based on the statistical requirements, must be developed and implemented.
Examples of issues that specifications and procedures might address include:
- Stratum definitions, stratification algorithms, and clustering algorithms.
- Addition or deletion of records to update frames.
- Algorithms for creating PSUs.
- Sampling algorithms.
- Unduplication of the sample between surveys or between different waves of the same survey.
- Creation of sample replicates needed for weighting, estimation, and variance estimation.
- Assignment of sampling weights appropriate for the sample design to selected units.
Sub–Requirement A3–5.2: Systems and procedures must be verified and tested to ensure all components function as intended.
Examples of verification and testing activities include:
- Verifying that specifications conform to the technical requirements for the frame and sample design (e.g., using walk–throughs and peer reviews).
- Validating computer code against specifications.
- Performing tests of the individual modules and an integrated test of the full sample selection operation.
- Verifying the accuracy of frame information.
- Verifying the selection of the sample for accuracy (e.g., sample sizes are as expected).
Sub–Requirement A3–5.3: Systems and procedures must be developed and implemented to monitor and evaluate the accuracy of the frame development and sample selection operations and to take corrective action if problems are identified.
Examples of activities to monitor and evaluate the accuracy include:
- Comparing weighted sample counts with frame counts.
- Verifying that sample sizes are within expectations.
- Evaluating the accuracy and coverage of the frames against the target population.
- Evaluating changes in the sample design to understand how the revisions might affect the estimates.
Requirement A3–6: Documentation needed to replicate and evaluate frame development and sample design operations must be produced. The documentation must be retained, consistent with applicable policies and data–use agreements, and must be made available to Census Bureau employees who need it to carry out their work.(See Statistical Quality Standard S2, Managing Data and Documents.)
Examples of documentation include:
- Plans, requirements, specifications, and procedures for the systems and processes of frame development and sample selection.
- Sampling design information needed to produce estimates and variance estimates.
- Descriptions of the frame and its coverage.
- Techniques used to evaluate the coverage of the frame and the adequacy of the sample design.
- Quality measures and evaluation results. (See Statistical Quality Standard D3, Producing Measures and Indicators of Nonsampling Error.)
Back to Main
- The documentation must be released on request to external users, unless the information is subject to legal protections or administrative restrictions that would preclude its release. (See Data Stewardship Policy DS007, Information Security Management Program.)
- Statistical Quality Standard F2, Providing Documentation to Support Transparency in Information Products, contains specific requirements about documentation that must be readily accessible to the public to ensure transparency of information products released by the Census Bureau.