This study was the second round of usability testing for the Data Reliability Indicator for American Community Survey (ACS) data tables proposed by the sponsor team. Three prototype tables with a color-coded indicator based on an estimate's coefficient of variation were compared to a baseline table that represented a past production version of an ACS data profile. One prototype had three levels of reliability and the other two had four levels of reliability. One version of the four-level table was labeled with the terms "excellent," "good," "fair," and "poor." The other was labeled with the terms "reliable," "mostly reliable," "somewhat reliable," and "unreliable." No differences were found in users' accuracy and effciency (time on task) across the four tested table designs. Overall, 19 of the 21 participants indicated that they preferred the prototype tables over the baseline tables, although their preference for either the three- (n=10) or four-level (n=9) tables or either of the two alternative wording options was fairly evenly split (nine preferred the "good" wording, ten preferred the "reliable" wording, and two had no preference). However, there were some significant dfferences in satisfaction scores between the tables. There is evidence that satisfaction as measured by the QUIS instrument was significantly higher for both the baseline and the four-level tables than for the three-level table. An analysis of how likely participants were to mention the margin of error (MOE) or report it along with the estimate showed no significant differences by table design. Eye-tracking heat maps showed that participants looked at the MOE column on the table fewer times for the three-level condition.
Participants did use the reliability indicator and frequently said that they would report the message contained in the reliability column along with the estimate (12% indicator reporting for the three-level condition and 43% reporting for each of the four-level conditions). Participants in both the four-level "good" table and the four-level "reliable" table conditions were significantly more likely to report the message from the color-coded reliability indicator than the three-level indicator in a post-hoc test. Also, participants in both the four-level "good" table and the four-level "reliable" table conditions were significantly more likely to explicitly report the message from the color-coded reliability indicator along with the estimate than participants in the three-level indicator condition. There was no significant difference between the two four-level tables themselves on this variable. More detailed results and potential usability issues are discussed.
data reliability indicator, coefficient of variation, color-coded data tables, usability
Ashenfelter, Kathleen T. (2010). Data Reliability Indicator Based on the Coefficient of Variation: Results from the Second Round of Testing. Statistical Research Division Study Series (Survey Methodology #2010-04). U.S. Census Bureau. Available online at <http://www.census.gov/srd/papers/pdf/ssm2010-04.pdf>.
Source: U.S. Census Bureau, Statistical Research Division
Published online: March 31, 2010
Last revised: March 9, 2010
This symbol indicates a link to a non-government web site. Our linking to these sites does not constitute an endorsement of any products, services or the information found on them. Once you link to another site you are subject to the policies of the new site.