|
Pretesting Questionnaires and Related Materials for Surveys and Censuses Census Bureau
Standard Authored by: Nancy Bates Diane Willimack Jane Ingold Introduction The Census Bureau's Methodology and Standards Council sets statistical standards for the Census Bureau's surveys and censuses. This responsibility encompasses setting standards for the pretesting of Census Bureau questionnaires and related materials. The standards set forth here are based on a document entitled "Pretesting Policy and Options: Demographic Surveys at the Census Bureau." While that document pertains only to the demographic area, the standards set here apply to all areas of the Census Bureau and will supersede the existing document. By requiring pretesting of new and revised questions, the Census Bureau aims to improve the quality of the data collected in its demographic and economic censuses and surveys. The pretesting required by the standards will further the Census Bureau's ability to provide high-quality data for its sponsors and data users. Scope These pretesting standards apply to instruments and materials implemented using paper or automated methods for all surveys and censuses conducted within all program areas of the Census Bureau, including those sponsored by other federal agencies. They apply to pilot tests of new or redesigned forms, methods panels associated with the redesign of a specific survey, and the decennial census testing and evaluation program (site and national tests). Mandatory administrative forms, such as the Shipper's Export Declaration, are not covered by these standards. Standards This document sets forth four standards: a minimal standard required for all demographic and economic censuses and surveys, and three recommended standards pertaining to specific situations in which consequences of lapses in data quality are especially high. Minimal Standard When an internal or external sponsor proposes to ask questions which have not been included in previous Census Bureau surveys, or to modify questions which have been previously asked, the Census Bureau's minimal testing standard applies. A questionnaire pretest to test and refine questions is required before they are put into production. Sponsors may contract with an independent organization to conduct the required testing, may conduct their own pretesting if they have the capability, may procure Census Bureau services, or may submit evidence from existing research showing that the questions meet the minimal standard. The minimal standard is that a census or survey questionnaire must "work." This means that the questions (new or revised) can be administered properly by interviewers (if relevant) and understood and answered by respondents, and do not adversely affect survey cooperation. Evidence that a question works may be based on results of its use in prior surveys, or on tests with respondents conducted by the Census Bureau or by other organizations. If there is insufficient evidence about how well a question performs, the question must be subjected to some form of questionnaire pretest. A description of the acceptable pretest methods is included in Supporting Document A. When surveys are conducted using multiple modes of questionnaire administration, all questionnaire versions should work. When meaningful changes are made to questions to accommodate mode differences (e.g., changes to the presentation of question format to reflect mode-specific functional constraints or advantages), each version must be tested to ensure consistent interpretation of question purpose across modes, despite structural format or presentation differences. As long as the proposed wording of each version is tested, duplication of the mode (e.g., paper vs. CATI or CAPI) is not required, although it may be advisable. This minimal standard also applies to non-English language census and survey forms. That is, questionnaires and guides that will be administered in any language other than English must be pretested in the languages that will be used in the production survey or census. Therefore, to the extent feasible, evidence that a non-English language census or survey questionnaire "works" must be demonstrated either through its use in that language in prior surveys or tests conducted by the Census Bureau or by other organizations. This pretesting should follow the development of these instruments in accordance with the Census Bureau Guideline: Language Translation of Data Collection Instruments and Supporting Materials. Examples of circumstances that would invoke the minimal standard include: * if a sponsor proposes to ask questions that have not been asked in previous Census Bureau surveys, or to modify questions that have been asked previously; * if review of nonresponse rates, measures of reliability and validity, or other evidence (e.g., benchmarking) indicates that the data are of poor quality; * if subject-matter experts decide that the content of a survey is outdated and revisions are required to address issues of validity or relevance; * if policy or program changes require new surveys, new questions, or other changes to the data collection instrument, including the addition of new data collection technology; * if legislative changes mandate new surveys, new questions, or other changes to the data collection instrument. If the necessary tests are not conducted, or if sufficient evidence from existing research is not presented, the Census Bureau will not include the questions in the survey. If the tests are conducted and the results show that a question performs poorly or adversely affects responses to other survey questions or survey response rates, the Census Bureau will not use the survey question without modifying and testing it further. Recommended Standards These standards are not required for all data collections, but are recommended for surveys and censuses in which special circumstances apply. In these cases, being held to standards higher than the minimal standard may be warranted. The Census Bureau plans to phase in these standards as required at some point in the future. Recommended Standard With Important Policy Implications: This standard is recommended when changes in questions or modes of data collection are made to surveys and censuses for which the data have important policy uses. It is recommended that additional questionnaire testing be conducted when the data of interest are key economic and socio-economic indicators (a list of these is included in Supporting Document B). Changes in survey procedures, including the questions asked, may affect the continuity of time series data. Seemingly minor changes in question wording or sequence sometimes can affect survey responses in important and unexpected ways. Similarly, any systematic change in the survey process (such as mode of interviewing, sample design, edit routines, or field procedures) can also result in a change in survey estimates. When a key economic or socio-economic indicator may be affected by a questionnaire or procedural change, the Census Bureau recommends that an experimental field test be conducted to evaluate the effects of the proposed changes on survey estimates. The Census Bureau's statistical experts can recommend designs for implementing changes in survey questions and procedures in such a way that effects on time series are minimized, and/or can be statistically estimated. Recommended Standard for Supplemental Instruments and Materials: These standards are recommended for surveys and censuses in which data collection instruments and activities have a high profile and the impact of procedural failures would be large. * Because the wording of supplemental materials is important to response rates, it is recommended that other survey materials (for example, advance letters, reminder postcards, supplemental instruction sheets, letters with replacement questionnaires) either be pretested or based on prototypes/boiler plate language that has been pretested. * It is recommended that data collection instruments for pre-census operations that involve contact with respondents be pretested. This includes, but is not limited to, automated mapping and listing, permanent address listing, block canvassing, group quarters frame development, and Local Update of Census Addresses (LUCA). * When the physical or structural features of a self-administered form are tested (for example, questionnaire folding pattern, number of person spaces included on the form, size of the pages in the form), testing with an actual, mock, or prototype mailing package, including questionnaire, envelope, and any supplementary materials, is recommended. This is because slight changes between the tested pieces and the production pieces may affect the comparability of the results. All pieces in the pre-test mailing package should be the same size and color as those proposed for mailout. Recommended Standard for Electronic Self-Administered Questionnaires: This standard is recommended for electronic questionnaires, to test the performance of the entire programming operation rather than just the form itself. It is recommended that the interface for electronic self-administered questionnaires (sometimes known as CSAQs) be pretested, including log-in or access mechanisms, navigation interfaces, automated edit functions, help screens, and submission procedures. In program areas that have tested and adopted a style guide for electronic instruments, interfaces designed according to style guide criteria need not be pretested. However, it is recommended that proposed changes to, or deviations from, the style guide be pretested. Responsibilities Program areas will be responsible for the following: * development of project schedules that allow adequate time for adhering to these standards; * preparation of the appropriate information or materials based on these standards; * consultation with survey methodologists to develop the prescribed course of action; * conducting pretesting activities when appropriate expertise is available internally; and * ensuring the proper use of the standards. Methodological research areas in SRD and ESMPD will be responsible for the following: * review of all pretesting activities that are conducted within the program areas; * consultation, expert reviews, and conduct of pretesting activities when expertise is not available within the program area; and * documenting, archiving, and making available the results of pretesting activities. The Methodology and Standards Council will be responsible for the following: * the initiation of periodic evaluations, reviews, and updates to the standards, as necessary; and * guidance to program areas in the application of the standards. Implementation Implementation of these standards depends on the availability of resources. Implementation should be phased in to allow for attention to staffing and funding issues involved in complying with these standards. Surveys With Existing OMB Clearance Approval. All surveys will be grandfathered in until the expiration of the OMB clearance effective at the issue date of this standard. That is, any changes made before the expiration of the current clearance are not subject to this standard. Surveys Seeking OMB Clearance Approval. Any changes made in these surveys need to be pretested unless the next clearance process comes up less than 12 months from the issue date of this standard. After the expiration of the current clearance, no testing is necessary if no changes are made to the survey. Exceptions. If there are situations in which these standards cannot be met, the situation should be brought to the attention of the Methodology and Standards Council so that a reasonable alternative can be developed. Inquiries Inquiries relating to interpretation of this standard should be addressed to the Census Bureau's Methodology and Standards Council. Supporting Documents: A Questionnaire Testing and Evaluation Methods for Censuses and Surveys B Key Indicators for Which Split Panel Testing is Recnded
Recommended by Census Bureau Methodology and Standards Council:
Signed Robert N. Tinari, July 25, 2003 Signed Rajendra P. Singh, July 25,
2003 Signed Alan R. Tupek, July 25, 2003 Signed Howard Hogan, July 25, 2003 Signed Ruth Ann Killion, July 25, 2003 Signed Tommy Wright, July 25, 2003 Concurrence: Signed Lawrence A. Neal, July 25,
2003 Signed Gloria Gutierrez, July 25,
2003 Marvin D. Raines Associate Director for Field Operations Signed Frederick T. Knickerbocker, July 25,
2003 Signed Preston Jay Waite, July 25,
2003 Signed Nancy M. Gordon, July 25,
2003 Signed Cynthia Z.F. Clark ,July 25,
2003 Signed Gerald W. Gates ,July 25,
2003
Document Management & Control
The most current version of
this document is maintained on the Census Bureau Intranet and may Category:
Standard |