Research Reports

You are here: Census.govSubjects A to ZResearch Reports Sorted by Year › Abstract of SSM2009/01
Skip top of page navigation

Final Report for First-Round Usability Testing of Data-Reliability Indicator Prototypes

Kathleen T. Ashenfelter, Jenna Beck, and Elizabeth D. Murphy


All American Community Survey (ACS) estimates are currently released with their associated margin of error (MOE). However, many users do not think that the MOE provides enough information about an estimate’s reliability. We conducted usability testing to examine whether the addition of reliability indicators to new prototypes of ACS data tables helped people use the tables with greater accuracy, efficiency, and satisfaction. All three prototypical reliability indicators were based on the coefficient of variation (CV), which is defined as the standard error divided by the mean of the estimate.

Each prototypical table with a reliability indicator tested in this study implemented a color-coded “Reliability” column and a legend explaining the meaning of the color codes to provide users with guidance as to whether the proportion of error to the estimate itself might be considered unacceptably high.

Each prototype table was defined by the number of “levels” that its Data Reliability Indicator used to label the estimates. Specifically, the two-level indicator had the levels “blank” (CV<=0.30) and “use caution” (yellow; CV>0.30); the three-level indicator included the levels “good” (green; CV<=0.30), “fair” (yellow; 0.30<CV<=0.61), and “poor” (red; CV>0.61); and the four-level indicator had the levels “excellent” (green; CV<=0.10), “good” (yellow; 0.10<CV<=0.30), “fair” (orange; 0.30<CV<=0.61), and “poor” (red; CV>0.61). Nine participants completed usability tasks using the prototypes, and three participants completed the tasks using baseline versions of the current ACS data tables without the indicators. Full versions of the prototype tables can be found in Appendix A.

The results showed that the four-level indicator was associated with the highest accuracy and satisfaction scores. In fact, eight of the twelve participants indicated that they preferred the four-level indicator overall. Users were able to complete the tasks more efficiently using the prototypes than the baseline tables, and they also expressed a strong preference for all of the prototypes over the baseline tables.

CITATION: Ashenfelter, Kathleen T., Beck, Jenna, and Murphy, Elizabeth D. (2009). Final Report for First-Round Usability Testing of Data-Reliability Indicator Prototypes. Statistical Research Division Study Series (Survey Methodology #2009-01). U.S. Census Bureau. Available online at <>.

Source: U.S. Census Bureau, Statistical Research Division

Published online: February 18, 2009
Last revised: February 12, 2009

[PDF] or PDF denotes a file in Adobe’s Portable Document Format. To view the file, you will need the Adobe® Reader® Off Site available free from Adobe.

This symbol Off Site indicates a link to a non-government web site. Our linking to these sites does not constitute an endorsement of any products, services or the information found on them. Once you link to another site you are subject to the policies of the new site.

Source: U.S. Census Bureau | Statistical Research Division | (301) 763-3215 (or |   Last Revised: October 08, 2010