The quality of data collected in a survey rests on, among other things, the notion that the meaning of the question as written by the questionnaire designer is consistent with the way respondents interpret it. Pretesting attempts to ensure this by subjecting the questionnaire to some type of evaluation to assess its ability to collect the desired data. As the popularity and variety of pretesting methods grow, the question remains as to whether different methods yield consistent and/or complementary results. A commonly used method of pre testing, the cognitive interview, has traditionally been used as a front-end tool for identifying problems in a questionnaire at a relatively early stage of its development. Other “front-end” methods include expert review and forms appraisal system (which is a detailed, formal system of coding questions in questionnaires in order to detect problems). On the other hand, respondent debriefing (which involves incorporating follow-up questions in a standardized interview), and behavior coding (which is the systematic coding of the interactions between an interviewer and a respondent) have typically been used as a final stage question evaluation method during a field test or the actual production implementation of the survey itself. This paper compares three common methods of pretesting – cognitive interviewing, behavior coding , and respondent debriefing – to evaluate a series of computer-crime questions from the National Crime Victimization Survey (NCVS).

The remainder of this paper is organized as follows: Section I briefly reviews the literature to see how other research has assessed the strengths and weaknesses of these three pretesting methods, both separately and in comparison with each other; Section II lay s out the objectives of the present research; Section III describes the methods used in this research; Section IV presents the results of cognitive interviews conducted in the lab, the respondent debriefing questions administered in the field, and behavior coding of interviews tape recorded in the field; and Section V offers some conclusions about the effective ness of using each of these methods as a pretesting tool.