Purpose: The purpose of this standard is to ensure that data collection instruments and supporting materials are designed to promote the collection of high quality data from respondents.
Scope: The Census Bureau’s statistical quality standards apply to all information products released by the Census Bureau and the activities that generate those products, including products released to the public, sponsors, joint partners, or other customers. All Census Bureau employees and Special Sworn Status individuals must comply with these standards; this includes contractors and other individuals that receive Census Bureau funding to develop and release Census Bureau information products.
In particular, this standard applies to the development or redesign of data collection instruments and supporting materials. The types of data collection instruments and supporting materials covered by this standard include:
In addition to the global exclusions listed in the Preface, this standard does not apply to:
Key Terms: Behavior coding, CAPI, CATI, cognitive interviews, data collection instrument, field test, focus group, graphical user interface (GUI), imputation, integration testing, methodological expert review, nonresponse, pretesting, questionnaire, record linkage, respondent burden, respondent debriefing, split panel test, and usability testing.
Requirement A2-1: Throughout all processes associated with data collection, unauthorized release of protected information or administratively restricted information must be prevented by following federal laws (e.g., Title 13, Title 15, and Title 26), Census Bureau policies (e.g., Data Stewardship Policies), and additional provisions governing the use of the data (e.g., as may be specified in a memorandum of understanding or data-use agreement). (See Statistical Quality Standard S1, Protecting Confidentiality.)
Requirement A2-2: A plan must be produced that addresses:
Requirement A2-3: Data collection instruments and supporting materials must be developed and tested in a manner that balances (within the constraints of budget, resources, and time) data quality and respondent burden.
Sub-Requirement A2-3.1: Specifications for data collection instruments and supporting materials, based on program requirements, must be developed and implemented.
Examples of topics that specifications might address include:
Note: The Census Bureau Guideline Presentation of Data Edits to Respondents in Electronic Self-Administered Surveys presents recommendations for designing editing functionality, presentation, and wording in both demographic and economic self-administered electronic surveys.
Sub-Requirement A2-3.2: Data collection instruments and supporting materials must clearly state the following required notifications to respondents:
Sub-Requirement A2-3.3: Data collection instruments and supporting materials must be pretested with respondents to identify problems (e.g., problems related to content, order/context effects, skip instructions, formatting, navigation, and edits) and then refined, prior to implementation, based on the pretesting results.
Note: On rare occasions, cost or schedule constraints may make it infeasible to perform complete pretesting. In such cases, subject matter and cognitive experts must discuss the need for and feasibility of pretesting. The program manager must document any decisions regarding such pretesting, including the reasons for the decision. If no acceptable options for pretesting can be identified, the program manager must apply for a waiver. (See the Waiver Procedure for the procedures on obtaining a waiver.)
Note: Pretesting is not required for questions that performed adequately in another survey.
Meaningful changes to questions to accommodate mode differences include changes to the presentation of the question or response format to reflect mode-specific functional constraints or advantages. In these cases, the proposed wording of each version must be pretested to ensure consistent interpretation of the intent of the question across modes, despite structural format or presentation differences. As long as the proposed wording of each version is pretested, testing of the mode (e.g., paper versus electronic) is not required, although it may be advisable.
Sub-Requirement A2-3.4: Data collection instruments and supporting materials must be verified and tested to ensure that they function as intended.
Examples of verification and testing activities include:
Requirement A2-4: Documentation needed to replicate and evaluate the development of data collection instruments and supporting materials must be produced. The documentation must be retained, consistent with applicable policies and data-use agreements, and must be made available to Census Bureau employees who need it to carry out their work. (See Statistical Quality Standard S2, Managing Data and Documents.)
Examples of documentation include: