U.S. Department of Commerce

Research Reports

You are here: Census.govSubjects A to ZResearch Reports Sorted by Year › Abstract of SSM2010-04
Skip top of page navigation

Data Reliability Indicator Based on the Coefficient of Variation: Results from the Second Round of Testing

Kathleen T. Ashenfelter

 

ABSTRACT

This study was the second round of usability testing for the Data Reliability Indicator for American Community Survey (ACS) data tables proposed by the sponsor team. Three prototype tables with a color-coded indicator based on an estimate's coefficient of variation were compared to a baseline table that represented a past production version of an ACS data profile. One prototype had three levels of reliability and the other two had four levels of reliability. One version of the four-level table was labeled with the terms "excellent," "good," "fair," and "poor." The other was labeled with the terms "reliable," "mostly reliable," "somewhat reliable," and "unreliable." No differences were found in users' accuracy and effciency (time on task) across the four tested table designs. Overall, 19 of the 21 participants indicated that they preferred the prototype tables over the baseline tables, although their preference for either the three- (n=10) or four-level (n=9) tables or either of the two alternative wording options was fairly evenly split (nine preferred the "good" wording, ten preferred the "reliable" wording, and two had no preference). However, there were some significant dfferences in satisfaction scores between the tables. There is evidence that satisfaction as measured by the QUIS instrument was significantly higher for both the baseline and the four-level tables than for the three-level table. An analysis of how likely participants were to mention the margin of error (MOE) or report it along with the estimate showed no significant differences by table design. Eye-tracking heat maps showed that participants looked at the MOE column on the table fewer times for the three-level condition.

Participants did use the reliability indicator and frequently said that they would report the message contained in the reliability column along with the estimate (12% indicator reporting for the three-level condition and 43% reporting for each of the four-level conditions). Participants in both the four-level "good" table and the four-level "reliable" table conditions were significantly more likely to report the message from the color-coded reliability indicator than the three-level indicator in a post-hoc test. Also, participants in both the four-level "good" table and the four-level "reliable" table conditions were significantly more likely to explicitly report the message from the color-coded reliability indicator along with the estimate than participants in the three-level indicator condition. There was no significant difference between the two four-level tables themselves on this variable. More detailed results and potential usability issues are discussed.

KEYWORDS:

data reliability indicator, coefficient of variation, color-coded data tables, usability

CITATION:

Ashenfelter, Kathleen T. (2010). Data Reliability Indicator Based on the Coefficient of Variation: Results from the Second Round of Testing. Statistical Research Division Study Series (Survey Methodology #2010-04). U.S. Census Bureau. Available online at <http://www.census.gov/srd/papers/pdf/ssm2010-04.pdf>.

Source: U.S. Census Bureau, Statistical Research Division

Published online: March 31, 2010
Last revised: March 9, 2010


[PDF] or PDF denotes a file in Adobe’s Portable Document Format. To view the file, you will need the Adobe® Reader® Off Site available free from Adobe.

This symbol Off Site indicates a link to a non-government web site. Our linking to these sites does not constitute an endorsement of any products, services or the information found on them. Once you link to another site you are subject to the policies of the new site.

Source: U.S. Census Bureau | Statistical Research Division | (301) 763-3215 (or chad.eric.russell@census.gov) |   Last Revised: October 08, 2010