The U.S. Census Bureau’s Usability Lab conducted the third round of usability testing on the 2011 American Community Survey (ACS) Internet instrument from June 30th, 2010 to July 29th, 2010. This third usability test focused on the Login and Rostering sections of the ACS.
Some of the mailing materials associated with the instrument were also evaluated. The Initial mailing packet for the prominent choice, not prominent choice, or regular/modified push panel was given to the participant to use to access the online instrument and some feedback on the materials was gathered. These three panels were the ones that offered an Internet option for responding to the ACS, so the materials could be used in the lab by participants to access the online survey. A detailed description of the experimental panels from the 2011 ACS Internet test can be found in Ashenfelter, Holland, Quach, Nichols, & Lakhe (2011). A vignette methodology using fictitious complex living situations was applied in addition to having participants complete the survey based on their own living situations. This allowed the researchers to glean information related to a wider variety of coverage issues than those of the participants.
The coverage issues addressed in this test have been referenced in previous research and are described in Chan (2007) as, “Unlike, perhaps other types of undercoverage, errors due to problems in the application of rules and definitions would seem to offer some fertile ground for coverage improvements based on improved questionnaire design” (pg. 1) (see also Childs, Carter, Norris, Hanaoka, & Schwede, 2007; Childs, 2008; Hunter & de la Puente, 2005). The results from Rounds 1 and 2 of testing of the ACS Internet instrument are documented in Ashenfelter et. al (2011).
This report contains the results of usability testing and eye-tracking analysis of the instrument, results of the vignette completion of alternative living situations, and participant debriefing about the instrument and mailing materials as well as recommendations for improving the instrument. Aggregate data from the demographic questionnaire, satisfaction questionnaire, and eye tracking data are presented in tables throughout the report. We report content issues that were identified where appropriate.
Kathleen Ashenfelter, Victor Quach, Temika Holland, Elizabeth Nichols, and Sabin Lakhe. (2012). Report for Round 3 of Usability Testing of the 2011 American Community Survey Online Instrument: Focus on Login and Roster Features. Research and Methodology Directorate, Center for Survey Measurement Study Series (Survey Methodology #2012-10). U.S. Census Bureau. Available online at <http://www.census.gov/srd/papers/pdf/ssm2012-10.pdf>.