Skip Main Navigation Skip To Navigation Content

Research Reports

You are here: Census.govSubjects A to ZResearch Reports Sorted by Year › Abstract of RSM2008/07
Skip top of page navigation

Multilingual Questionnaire Evaluation and Development through Mixed Pretesting Methods: The Case of the U.S. Census Nonresponse Followup Instrument

Jennifer Childs and Patricia Goerman

KEY WORDS: Multilingual Questionnaire Development, Pretesting, Cognitive Testing, Behavior Coding, Observational Study, Usability Testing


The objective of the research described in this paper was to develop and improve a Nonresponse Followup (NRFU) instrument for the 2010 U.S. Census. This research is unique in that multiple pretesting methods were used in the development of a single instrument in two different languages: English and Spanish.
The NRFU instrument was originally developed and tested as a computer-assisted personal interview (CAPI) instrument. The U.S. Census Bureau’s NRFU operation is critical to gathering data from both English- and Spanish-speaking households who do not return their self-administered census forms. The instrument was developed using a series of pretesting methods, including usability and cognitive testing, behavior coding, and an observational study of the administration of test versions of both the English and Spanish instruments. Though many of the questions had already been tested in the context of a self-administered paper form, this was the first time pretesting was conducted using the interviewer-administered questions as they were scripted to be read from a CAPI instrument.
This paper discusses overarching results of three rounds of English cognitive testing, two rounds of Spanish cognitive testing, two rounds of usability testing in both languages, two rounds of behavior coding of the instrument in both languages, and an observational study of the administration of the NRFU interview in the field in both languages. The application of mixed pretesting methods to the development of one survey instrument is an all-too-uncommon situation. This paper presents lessons learned about the types of findings made possible by the different pretesting methods, and offers the unique opportunity to examine issues of equivalency between a source and a translated version of a survey instrument through multiple measures.

CITATION: Childs, J. and Goerman, P. Multilingual Questionnaire Evaluation and Development through Mixed Pretesting Methods: The Case of the U.S. Census Nonresponse Followup Instrument. Statistical Research Division Research Report Series (Survey Methodology #2008-07). U.S. Census Bureau.

Source: U.S. Census Bureau, Statistical Research Division

Created: September 30, 2008
Last revised: September 26, 2008

[PDF] or PDF denotes a file in Adobe’s Portable Document Format. To view the file, you will need the Adobe® Reader® Off Site available free from Adobe.

This symbol Off Site indicates a link to a non-government web site. Our linking to these sites does not constitute an endorsement of any products, services or the information found on them. Once you link to another site you are subject to the policies of the new site.

Source: U.S. Census Bureau | Statistical Research Division | (301) 763-3215 (or |   Last Revised: October 08, 2010