Research Reports

Respondent Debriefings Conducted by Experts: A New Qualitative Methodology for Questionnaire Evaluation

Elizabeth Nichols and Jennifer Hunter Childs

KEY WORDS: Conversational interviewing; Pretesting; Coverage Improvement

ABSTRACT

In theory, experts in a domain of interest could perform an in-depth interview using scripted and unscripted questions to obtain information needed to answer research questions. These experts would feel confident that the data they collected was of high quality since they could supplement the scripted probes with unscripted probes to determine the truth (see Conrad and Schober, 2000, and Schober and Conrad, 1997 for studies on conversational interviewing). If they felt the respondent was confused by questions, or if the respondent indicated there was more pertinent information available, then experts could inquire further. This is largely the method employed by experts doing cognitive testing of a questionnaire. However, this method is impractical for a large data collection.

This paper explores a method that uses expert respondent debriefings as a means of evaluating the quality of the production data. This is especially important for surveys for which an assessment of "truth" is not otherwise available. In July of 2006, a small group of experts in Census residence rules observed 169 Census Coverage Measurement Person Interviews in two sites for the 2006 Census Test. Immediately following 50 interviews where a complex living situation was described during the interview, these experts conducted qualitative, ethnographictype respondent debriefings asking open-ended questions about where the person of interest had been staying over the last year. The goal of the respondent debriefing was to determine the "true" residence for each person. We compare this "true" residence against the residence status obtained by the questionnaire alone. By comparing the two sets of data, we determine if the questionnaire was collecting accurate information, and for what situations the questionnaire needs to be modified. This paper discusses our methodology for these expert respondent debriefings.

CITATION:

Source: U.S. Census Bureau, Statistical Research Division

Created: October 3, 2007
Last revised: October 3, 2007