Test Objective

• In January through March of 2006, the American Community Survey (ACS) conducted the first test of new and modified content since the ACS reached full implementation levels of data collection. The results of that testing will determine the content for the 2008 ACS.
• Heating fuel showed the highest multiple response (2.1 percent) to a single response item in Census 2000. This finding suggests that some respondents in households using multiple heat sources may have overlooked identifying only one type of heating fuel as the primary source or the source used most as the question states.
• For heating fuel, the objective was to evaluate whether adding an instruction to mark only one box will maintain or improve the non-response rate for heating fuel while decreasing the multiple response rate.

Methodology

• The control version replicated the current ACS question. The test group modified the heating fuel question by including an instruction “Mark (X) one box”.

Research Questions and Results

• Research Question 1: Will adding the instruction “Mark (X) one box” decrease the number of multiple responses?
• The results did not support the hypothesis that the instruction will decrease the percent of multiple responses. There was no significant difference in the percentages of respondents who entered multiple answers for the heating fuel question at the national level (1.4 percent for the control panel and 1.6 percent for the test panel) or in the high response areas (1.5 percent vs. 1.6 percent). There was a significantly higher percent of multiple entries in the low response areas (1.2 percent vs. 1.7 percent).
• Research Question 2: Will adding the instruction “Mark (X) one box” affect the item non-response rate?
• The results did not support the hypothesis that the instruction maintained or improved the item non-response rate. We suspect that respondents who may use more than one type of house heating fuel may be more inclined not to answer the house heating fuel question when the instruction “Mark (X) one box” is added. There was a significantly higher percent of respondents who did not answer the heating fuel question at the national level (1.6 percent for the control panel and 2.0 percent for the test panel), however, there were no differences in the high (1.3 percent vs. 1.7 percent) or low (2.6 percent vs. 3.1 percent) response areas.
• Research Question 3: Will adding the instruction “Mark (X) one box” change the distribution of coded responses for sources of heating fuel?
• The results support the hypothesis that the addition of the instruction had minimal impact on the distribution of heating types.

Summary of Empirical Results

Even though the item non-response rate was significantly higher for the test version, the differences are not meaningful since the ACS telephone follow-up operation used in production can supplement the information reported on mail questionnaires by contacting respondents and obtaining missing data. However, the increase in multiple response rates, although not significant for the nation and high response areas, is problematic. It suggests that some respondents may be confused about the test question. Continued research using the ACS production files, which are now keying all reported (multiple) entries for all items could help clarify the heating fuel issue in the future.

Empirically, the control version performed better than the test version.