All survey information released by the Census Bureau must adhere to the Census Bureau’s Statistical Quality Standards for publication. This document sets thresholds for several metrics of sampling and non-sampling error below which there may be concerns regarding data quality. Subject Table S9801 presents the ACS values for those metrics at the national level, to provide more transparency on the quality of the ACS for data users.

Is this statistical quality standards assessment for the ACS new?

The publication of the metrics in S9801 is new for 2023, but the ACS has been calculating these metrics for internal use and assessment against the Census Bureau standards since 2011. Historical data for 2011 through 2022 are available as a downloadable file.

What metrics are reported in S9801?

There are four non-sampling error metrics and one sampling error metric that are reported in S9801. The metrics are listed below, as well as the thresholds that must be met, which are taken from section F1-7 in the Standards document.

Non-Sampling Error Metric	Required Threshold
Unit response rate	60% or higher
Coverage rates	70% or higher
Item response rates for “key estimates”	all 70% or higher
Combined rates for “key estimates”	all 50% or higher

Sampling Error Metric	Required Threshold
Coefficient of Variation for “key estimates”	majority 30% or less

While the Statistical Quality Standards set these thresholds, the individual surveys are allowed to determine how to best measure them. The details of how the ACS calculates the metrics follow. Notably, all of the ACS metrics are calculated using the ACS 5-year data.

What unit response rates are used?

The median of the housing unit response rate, as published in table B98021, is taken across all counties in the U.S. and all municipios in Puerto Rico. For more information on the response rates, visit the Response Rate Definitions.

What coverage rates are used?

The housing unit coverage rate, as published in table B98011, and the total population coverage rate, as published in table B98012, are used. The median housing unit coverage rate is computed across all counties in the U.S., excluding Puerto Rico because housing coverage rates cannot be calculated in Puerto Rico. The median total population coverage rate is computed across all counties in the U.S. and all municipios in Puerto Rico. Both median coverage rates must meet the 70 percent threshold. For more information on coverage rates, visit the Coverage Rate Definitions.

What are “key estimates”?

When developing the quality standards metrics for the ACS in 2011, this was the most important question to answer. With such a wide variety of estimates covering many topics, there are no “key estimates” for which the ACS is specifically designed to measure. Ultimately, seventeen estimates were selected, which met all (or most) of these guidelines:

Occur in about 10 percent of the population at the national level
Are used for critical policy purposes at subnational levels
Are published in the Data Profiles
Cover a spectrum of topic areas including social, economic, housing, and demographic characteristics

The seventeen key estimates listed in S9801 have been used since 2011. The key estimates are subject to change if a key estimate no longer meets the general guidelines above.

What are item response rates?

Item response rates are the complement of item allocation rates:

Item Response rate = 100% - Item Allocation rate

The metrics are the median item response rates across all counties and municipios.

Each of the seventeen key estimates is paired with an item response rate based on the question on the ACS form used to create the estimate. For example, for the key estimate of percent of the household population with the relationship of spouse, the corresponding item response rate is based on the relationship question on the form. For more information on the item allocation rates that the item response rates are based on, visit the Item Allocation Rate Definitions.

What are the combined rates?

Combined rates are defined as:

Combined rate = Response rate × Coverage rate × Item Response rate

The total population coverage rate is used for the population-based key estimates, and the housing unit coverage rate is used for the housing-based key estimates. The 17 median county combined rates are calculated across all counties for the housing-based estimates, and across all counties and municipios for the population-based estimates.

What are the coefficients of variation?

The coefficient of variation (CV) is defined as:

CV = (Standard Error of an Estimate) / Estimate

Where the Standard Error is the Margin of Error divided by 1.645.

Unlike the other metrics which are calculated as county-level medians, the median of each of the 17 key estimate CVs is taken across all tracts in the United States and Puerto Rico. To meet the specified threshold (“majority of the key estimates”), 9 or more of the 17 median CVs must be 30 percent or less.

Does the ACS always meet the Statistical Quality Standards thresholds?

From 2011 through 2019, yes, all thresholds were met in each of those nine years. However, problems were encountered with the CVs in 2020 and one item response rate in 2022.

What happened with the CVs in 2020?

Due to the data collection issues related to the pandemic in 2020, the characteristics of the respondents in 2020 differed in important ways from the respondents for a typical year, thus potentially distorting many ACS estimates. To diminish these effects on the 2016-2020 ACS 5-year estimates, new weighting methods were employed that, while improving the quality of the estimates, increased their margins of error. Due to this increase in the margins of error, fewer than nine of the key estimates had median CVs less than 30 percent, and the ACS did not pass that metric.

Because these adjustments to the weighting would be in place for each set of 5-year estimates including 2020, the ACS received a waiver on the sampling error metric from the Census Bureau’s Methodology and Standards Council. This allowed publication through 2024, with the expectation that the ACS would be back in compliance with the 2025 data, when the 5-year period would no longer include 2020. For more information visit the User Note: Increased Margins of Error in the 5-Year Estimates Containing Data Collected in 2020.

What happened with the item response rate in 2022?

The median total income item response rate paired with the household income key estimate fell below 70 percent. The item response rate that had been used for this key estimate was uniquely ill-suited for this purpose among the 17, as it looked at whether any of eight income questions on the form were allocated, instead of being focused on one question as the others were.

The Methodology and Standards Council again approved a waiver for 2022, with the expectation that the ACS would be back in compliance in 2023. Research suggested that the percent of household income allocated was a more appropriate base for this item response rate, and this new rate is included in S9801 for 2023. The 2023 median item response rates for all measures are again above 70 percent. For more information, visit the User Note: Item Allocation Rate for Total Income.