Skip Header

How Does the Pandemic Affect Survey Response: Using Administrative Data to Evaluate Nonresponse in the Current Population Survey Annual Social and Economic Supplement

September 15, 2020
Written by: Jonathan Rothbaum

In its new report, Income and Poverty in the United States: 2019, the U.S. Census Bureau provides estimates of the distribution of household and personal income and poverty. We collected data for this report (and the accompanying microdata and summary tables) in February, March and April of 2020 as part of the Current Population Survey (CPS) Annual Social and Economic Supplement (ASEC). Unfortunately, the COVID-19 pandemic affected survey collection. Interviewing for the March portion of data collection began on March 15 (which comprises about 75% of the CPS ASEC sample). To protect the health and safety of Census Bureau staff and respondents, we suspended in-person interviewing and closed the two computer-assisted telephone interview (CATI) centers just five days later. For the rest of March and April, we conducted all interviews by phone.  

While the Census Bureau went to great lengths to complete interviews by telephone, the response rate for the CPS basic household survey in March was 73%, about 10 percentage points lower than in preceding months and during the same period in 2019. Further, as the Bureau of Labor Statistics (2020) noted in early April when it released the March Employment Situation, “Response rates for households normally more likely to be interviewed in person were particularly low.  The response rate for households entering the sample for their first month was over 20 percentage points lower than in recent months, and the rate for those in the fifth month was over 10 percentage points lower.”

 

Figure 1. Unweighted Current Population Survey Monthly Response Rates for March 2016 Through April 2020

Source: U.S. Census Bureau, Current Population Survey, internal data files, March 2016-April 2020.

 

The CPS ASEC response rate includes different months and samples. It also includes an adjustment factor to account for those who responded to the basic survey but not the supplement.  The Census Bureau estimates that the unweighted combined supplement response rate was 61.1% in 2020, down from 67.6% in 2019.[1]

The Census Bureau creates weights designed to adjust for nonresponse and to control weighted counts to independent population estimates by age, sex, race and Hispanic origin.  But the magnitude of the increase in (and differential nature of) nonresponse related to the pandemic likely reduced their efficacy.

To understand the possible impact of these changes and the lower response rates, we evaluated the CPS ASEC sample, including comparisons between responses in 2020 and those in prior years. First, we looked at characteristics like race that do not change over time and those like age that change predictably to answer the question, “Do people that respond to the 2020 CPS ASEC differ from respondents in prior years?” We found that they do. The 2020 sample is older, more educated, has fewer Hispanics, and fewer noncitizens. 

The standard weighting procedure to control for nonresponse in the CPS ASEC only controls for differences by sex, age, race and Hispanic origin. However, if nonresponse is related to other characteristics, such as education, citizenship or income, the standard procedure may not be sufficient.

Next, we also used administrative, 2010 decennial census, and prior American Community Survey (ACS) data to determine whether households that answer the CPS ASEC differ from those that don’t respond to it. We found that 2020 was unique: respondents, unlike those in previous years, have higher earnings than nonrespondents.

 

Figure 2. Earnings Difference Between Respondents and Nonrespondents

Source: 2017-2020 Current Population Survey Annual Social and Economic Supplement linked to W-2 earnings data by address.

 

Still, the normal nonresponse adjustment could control for this. We tested whether household response in 2020 still correlates with income after we control for all the factors in our weighting adjustment (and others that we can observe in the linked data). We found that it is, as shown in Figure 3.

 

Figure 3. Probability of Response by W-2 Earnings

Source: 2017-2020 Current Population Survey Annual Social and Economic Supplement linked to administrative and survey data.[2]

 

To understand the potential impact of nonresponse on our income statistics, we developed and implemented an alternative nonresponse adjustment. This adjustment controls for nonresponse bias using information not available in survey processing from the linked administrative, census and survey data available for respondent and nonrespondent households. As we found that low-income households were less likely to respond to the CPS ASEC in 2020, this adjustment gives more weight to low-income households that did respond relative to high-income households that did respond.

Using these alternative weights, we calculate alternative income and poverty statistics and compare them against the official (survey-only) estimates. For real median household income and poverty, the estimates are not statistically different in 2017, 2018 or 2019, indicating that in pre-pandemic years, the nonresponse adjustment in the survey weights appears sufficient to address nonresponse bias for these statistics.[3]

Figure 4 shows estimates across the income distribution compared to the official estimates. A value of one indicates the survey and alternative-weight estimates do not differ at that point in the distribution. For 2017 to 2019, the estimates were always near 0.99 and 1. For 2020, the estimates are 2% to 4% lower than the survey estimates, reflecting the unique pattern of nonresponse in the 2020 CPS ASEC. In 2020, median household income is 2.8% lower and poverty is about a half a percentage point higher with the alternative weights.

 

Figure 4. Comparison of Nonresponse-Adjusted Income to Survey-Only Income over Time

Source: 2017-2020 Current Population Survey Annual Social and Economic Supplement linked to administrative and survey data.

 

Finally, we look at year-to-year changes using the alternative weights for both 2019 and 2020. We estimate that real median household income increased by 4.1% from 2019 to 2020 (compared to 6.8% estimated from the survey data), and poverty declined by 0.9 percentage points (compared to 1.3 percentage points from the survey data).

Even with this change, we estimate that real median household income in 2019 (estimated from the 2020 CPS ASEC) is the highest ever measured in the CPS ASEC.

For more information, see the Census Bureau working paper at <www.census.gov/library/working-papers/2020/demo/SEHSD-WP2020-10.html>.

For information on how the pandemic and nonresponse affected estimates of health insurance coverage, see the Census Bureau working paper at <www.census.gov/library/working-papers/2020/demo/SEHSD-WP2020-13.html>.


[1] For more information about the design of the survey, including weighting for nonresponse, see Technical Paper 77, <https://www2.census.gov/programs-surveys/cps/methodology/CPS-Tech-Paper-77.pdf> and a summary of weighting in the CPS at <www.census.gov/programs-surveys/cps/technical-documentation/methodology/weighting.html>.

[2] This figure comes from a regression of household response on income and various demographic and socioeconomic $25,000 to $50,000 is the baseline group. Therefore, the positive coefficient for “>=200,000” in 2020 indicates that households with $200,000 or more in earnings were more likely to respond to the CPS ASEC than households with $25,000 to $50,000 in earnings, after controlling for race, Hispanic origin, age, etc.

[3] Note that we continue to refer to the survey years in the text, tables, and figures to keep the year references consistent across table and more clearly identify the 2020 CPS ASEC as the one affected by the pandemic. Keep in mind that the reference period is the prior year in the CPS ASEC. Therefore, for example, when we discuss statistics for the 2020 CPS ASEC, we are discussing income earned or received in 2019.

 

Back to Header