U.S. Department of Commerce

Research Reports

You are here: Census.govSubjects A to ZResearch Reports Sorted by Year › Abstract of RSM2007/39
Skip top of page navigation

Respondent Debriefings Conducted by Experts: A New Qualitative Methodology for Questionnaire Evaluation

Elizabeth Nichols and Jennifer Hunter Childs

KEY WORDS: Conversational interviewing; Pretesting; Coverage Improvement

ABSTRACT

In theory, experts in a domain of interest could perform an in-depth interview using scripted and unscripted questions to obtain information needed to answer research questions. These experts would feel confident that the data they collected was of high quality since they could supplement the scripted probes with unscripted probes to determine the truth (see Conrad and Schober, 2000, and Schober and Conrad, 1997 for studies on conversational interviewing). If they felt the respondent was confused by questions, or if the respondent indicated there was more pertinent information available, then experts could inquire further. This is largely the method employed by experts doing cognitive testing of a questionnaire. However, this method is impractical for a large data collection.

This paper explores a method that uses expert respondent debriefings as a means of evaluating the quality of the production data. This is especially important for surveys for which an assessment of "truth" is not otherwise available. In July of 2006, a small group of experts in Census residence rules observed 169 Census Coverage Measurement Person Interviews in two sites for the 2006 Census Test. Immediately following 50 interviews where a complex living situation was described during the interview, these experts conducted qualitative, ethnographictype respondent debriefings asking open-ended questions about where the person of interest had been staying over the last year. The goal of the respondent debriefing was to determine the "true" residence for each person. We compare this "true" residence against the residence status obtained by the questionnaire alone. By comparing the two sets of data, we determine if the questionnaire was collecting accurate information, and for what situations the questionnaire needs to be modified. This paper discusses our methodology for these expert respondent debriefings.

CITATION:

Source: U.S. Census Bureau, Statistical Research Division

Created: October 3, 2007
Last revised: October 3, 2007


[PDF] or PDF denotes a file in Adobe’s Portable Document Format. To view the file, you will need the Adobe® Reader® Off Site available free from Adobe.

This symbol Off Site indicates a link to a non-government web site. Our linking to these sites does not constitute an endorsement of any products, services or the information found on them. Once you link to another site you are subject to the policies of the new site.

Source: U.S. Census Bureau | Statistical Research Division | (301) 763-3215 (or chad.eric.russell@census.gov) |   Last Revised: October 08, 2010