Test Objective

• In January through March of 2006, the American Community Survey (ACS) conducted the first test of new and modified content since the ACS reached full implementation levels of data collection. The results of that testing will determine the content for the 2008 ACS.
• The ACS uses the employment status concept (as officially defined by the Department of Labor) to classify the civilian working-age population (16 years and over) into three major categories: employed, unemployed, not in the labor force.
• Research comparing ACS data on employment status with corresponding data from the Current Population Survey (CPS) revealed substantial differences between their estimates. In relation to the CPS, the ACS underestimates employment and overestimates unemployment.
• Since the CPS is the “gold standard” for household estimates of employment status, the overall objective of the ACS 2006 Content Test for the employment status items was to bring the ACS estimates closer to the CPS estimates, without a significant decrease in data quality as reflected by an increase in response errors (bias), response variability, and nonresponse rates.

Methodology

• The estimates of employment and unemployment in the ACS are developed by recoding the pattern of responses to a battery of six questions. The 2006 Content Test focused on three of the questions in this battery: at work last week, temporarily absent, and looking for work.
• The test versions differed from the control versions in the following ways:
  o For the at work last week question, we separated the inquiry into two parts, in an effort to give respondents -- particularly people with irregular kinds of work arrangements -- two opportunities to grasp and respond to the correct intent of the question. We removed a supposedly confusing reference to work for profit; and we expressed the meaning of work in the question itself rather than parenthetically.
  o For the temporarily absent question, we revised the list of examples of reasons for work absences to include maternity/paternity leave, family or personal reasons, and bad weather.
  o For the looking for work question, we modified the expression ‘looking for work’ by adding the word ‘actively,’ emphasized in all capital letters, to provide respondents with a clearer idea of who should be counted as looking for work.
• The changes to the above three questions result in a different battery of employment status questions for the test panel than for the control panel. The Content Test focused on comparing the employment status data derived from each of these batteries.
• The reinterview part of the Content Test (that is, the Content Follow-up Survey) attempted to measure differences in response bias between the test and control panels. To do so, it asked a series of probing questions about employment status taken from the more detailed employment status sequence of questions in the CPS.

Research Questions and Results

Summary

The selection criteria for the battery of test questions were based on the results for the three major categories of the employment status concept. The results showed that the test questions produced a higher estimate of employed people. While the test questions did not produce a lower estimate of unemployed people, the overall unemployment rate was lower for the test panel. The unemployment rate is the more useful measure because it excludes people who are not in the labor force. Finally, the net difference rate, a measure of response error, for the estimates of ‘employed’ and ‘not in the labor force’ was lower in the test panel and the rate for ‘unemployed’ was not significantly different between the control and test panels. Our findings indicate that, empirically, the test version of the questions performed better than the existing ACS questions.

However, while the test questions may have produced more favorable estimates than the control questions, there remains room for improvement. The comparison of the test and the Control Follow-up results show that respondents are answering the employment series of questions inconsistently. The structure and wording of these questions should be revisited in future ACS content tests.