Statistical Quality Standard E2: Reporting Results
Purpose: The purpose of this standard is to ensure that information products meet statistical reporting requirements; that they provide understandable, objective presentations of results and conclusions; and that conclusions are supported by the data.
- Requirement F1–4 of Statistical Quality Standard F1, Releasing Information Products, contains reporting requirements regarding information products affected by serious data quality issues that may impair the suitability of the products for their intended uses.
- Department Administrative Order (DAO) 219–1 establishes the policy for Commerce Department employees engaging in public communications.
Scope: The Census Bureau's statistical quality standards apply to all information products released by the Census Bureau and the activities that generate those products, including products released to the public, sponsors, joint partners, or other customers. All Census Bureau employees and Special Sworn Status individuals must comply with these standards; this includes contractors and other individuals who receive Census Bureau funding to develop and release Census Bureau information products.
In particular, this standard applies to the reporting of results in information products such as:
- News releases.
- Census Bureau publications (i.e., information products that the program’s Associate Director has reviewed and approved and the Census Bureau has affirmed their content).
- Working papers (e.g., technical papers and division reports intended for release to the public).
- Professional papers (e.g., journal articles, book chapters, conference papers, poster sessions, and written discussant comments).
- Research reports used to guide decisions about Census Bureau programs.
- Presentations at public events, such as seminars or conferences. (Statistical Quality Standard E3, Reviewing Information Products, defines public events.)
- Handouts for distribution at public events.
- Tabulations, including custom tabulations, estimates, and their associated documentation.
- Statistical graphs, figures, and thematic maps.
In addition to the global exclusions listed in the Preface, this standard does not apply to:
- Papers, presentations, or other public communications prepared or delivered by Census Bureau employees that are not related to programs, policies, or operations of the Department of Commerce (DOC) or the Census Bureau. (The DOC Summary of Ethics Rules state that you may use your Census Bureau affiliation in non–official contexts only if it is used as part of general biographic information, and it is given no more prominence than other significant biographical details. Contact the Office of Analysis and Executive Support (OAES) for additional guidance.)
Key Terms: Census Bureau publications, coefficient of variation (CV), confidence interval, custom tabulations, derived statistics, design effect, direct comparison, estimate, implied comparison, information products, margin of error (MOE), metadata, nonsampling error, policy view, sampling error, significance level, standard error, statistical inference, statistical significance, synthetic data, transparency, and working papers.
Requirement E2–1: Throughout all processes associated with reporting results, unauthorized release of protected information or administratively restricted information must be prevented by following federal laws (e.g., Title 13, Title 15, and Title 26), Census Bureau policies (e.g., Data Stewardship Policies), and additional provisions governing the use of the data (e.g., as may be specified in a memorandum of understanding or data–use agreement). (See Statistical Quality Standard S1, Protecting Confidentiality.)
Requirement E2–2: All information products must provide accurate and reliable information that promotes transparency and must present that information in an unbiased manner.
Key estimates in the text must be accompanied by confidence intervals or margins of error (MOEs) or their equivalents (e.g., equivalents for Bayesian inferences or for error arising from synthetic data) for the information products indicated in the table below. Providing a URL to these measures of statistical uncertainty is not sufficient.
- Information products based on data that have "serious quality issues" are not permitted except under the restrictions in Sub–Requirement F1–5.2 of Statistical Quality Standard F1, Releasing Information Products.
Note: Requirement F1–5 in Statistical Quality Standard F1 describes serious data quality issues.
- Except as noted below, information products (including each table, graph, figure, and map within an information product, and including stand–alone tables, such as custom tabulations) must include a source statement that:
- Indicates the program(s) that provided the data.
- Indicates the date of the source data.
Note: Abstracts and presentation slides do not need source statements.
- Except as noted below, information products (including tables, graphs, figures, and maps that stand alone) must indicate that the data are subject to error arising from a variety of sources, including (as appropriate) sampling error, nonsampling error, model error, and any other sources of error. Including one of the following in the information product will satisfy this requirement:
- An explicit statement indicating that the data are subject to error arising from a variety of sources.
- A description of the error sources.
- A discussion of the error sources.
Note: Abstracts and presentation slides do not need to indicate that the data are subject to error.
- Except as noted below, information products must include a reference (i.e., URL) to the full methodological documentation of the program(s).
Note: Abstracts and presentation slides do not need to include a reference to the full methodological documentation.
All inferences and comparisons of estimates based on sample data must include appropriate measures of statistical uncertainty, such as margins of error, confidence intervals, or p–values for hypothesis tests.
Results that are not statistically significant must not be discussed in a manner that implies they are significant.
The same significance or confidence level must be used throughout an information product. Table A shows the
requirements for specific information products:
Table A. Significance and Confidence Levels by Information Product
|Census Bureau publications
|All other information products (e.g., working papers, professional papers, and
||0.10 or less
||0.90 or more
- Direct comparison statements that are not statistically significant must include a statement conveying the lack of
statistical significance, such as:
“The 90 percent confidence interval for the change includes zero. There is insufficient evidence to conclude that the actual change is different from zero.”
Such a statement may be given in a footnote. For example, "Sales of nondurable goods were down
0.6 percent (+/– 0.8 %)*." Footnote: "*The 90 percent confidence interval
includes zero. There is insufficient evidence to conclude that the actual change is different from zero."
The text must clearly state whether each comparison (direct or implied) is statistically significant. This must be done
- Using a blanket statement such as, "All comparative statements in this report have undergone statistical testing, and, unless otherwise noted, all comparisons are statistically significant at the 10 percent significance level," and specifically noting any implied comparison statements that are not significant.
- Reporting a p–value for each comparison.
- Stating whether or not the confidence interval includes 0.
- Statements of equality between population quantities that are being estimated with sampling error are not allowed.
For example, the following statements are not acceptable, since they refer to unknown underlying
- "The poverty rate for state A equals the rate for state B."
- "The poverty rate remained statistically unchanged" (for a comparison across time).
It is acceptable to say that the estimates are "not statistically different" or (for
comparisons over time) "statistically unchanged," if the difference in the estimates is not
statistically significant. For example, the following statements are acceptable, since they refer to
the estimates of population quantities:
- "The estimated poverty rate for state A, 8.1 percent (± 0.2), is not statistically different
from the estimated poverty rate for state B, 8.1 percent (± 0.2)."
- "The estimated poverty rate remained statistically unchanged for non–Hispanic whites at 8.2 percent
(± 0.2)." However, this statement must be accompanied by the abovementioned footnote.
Table B. Confidence Intervals or MOEs for Key Estimates by Information Product
||Confidence intervals or MOEs
|Census Bureau publications
|News releases for the economic data items listed in Appendix E2
|News releases for all other data (e.g., economic data items not in Appendix E2, household–level data, and person–level data)
|Abstracts and presentations slides
|All other information products (e.g., working papers and professional papers)
Except as noted below, information products must include or make available by reference (URL) information that
allows users to assess the statistical uncertainty of derived statistics as well as of the estimates themselves.
- In working papers and professional papers, p–values, standard errors, coefficients of variation (CV), or other appropriate measures of statistical uncertainty may be used instead of confidence intervals or MOEs.
- If the width of a confidence interval rounds to zero, the interval may be replaced by a statement such as "The width of the confidence interval for this estimate rounds to zero."
- Measures of statistical uncertainty (e.g., variances, CVs, standard errors, error arising from synthetic
data, or their Bayesian equivalents).
- Methods to estimate the measures of statistical uncertainty (e.g., generalized variance functions or equations
and design effects).
- Methods to approximate the measures of statistical uncertainty for derived statistics, such as estimates of
change or ratios of estimates.
If needed for readers to assess the results presented, the information product must include:
- This requirement does not apply to response rates, unless the information product analyzes the response
rates or draws conclusions from them.
- Abstracts and presentation slides need not make available information on statistical uncertainty. Custom
tabulations must provide information on statistical uncertainty as specified in Sub–Requirement E2–2.2, item 4.
- Maps need not portray or indicate information on statistical uncertainty, but if not, they must include a
URL at which users can access measures of statistical uncertainty and other information about statistical
- When information on statistical uncertainty is made available by referencing a URL, the URL must direct
users specifically to the location of the information.
The information presented must be technically and factually correct.
The information must be presented logically and any results must follow from the data and the analysis.
Any anomalous findings must be addressed appropriately.
The subject matter and methodological literature must be referenced, as appropriate.
Policy views must never be expressed.
Except as noted in Sub–Requirement E2–2.1 (item 3), personal views must not be expressed.
- A discussion of the assumptions made.
- The limitations of the data.
- A description of the methodology used to generate the estimates.
- An explanation of how the methodology and the limitations might affect the results.
Sub–Requirement E2–2.1: In addition to the requirements for all information products, the requirements for working papers, professional papers, research reports, presentation slides, handouts for distribution at presentations, and similar products include the following:
- Except as noted below, a disclaimer must be included on the title page. The author may determine the wording of the disclaimer as long as it indicates that any views expressed are those of the author and not necessarily those of the Census Bureau. An example of a disclaimer is: "Any views expressed are those of the author(s) and not necessarily those of the U.S. Census Bureau."
Note: The disclaimer is not needed for:
Working papers published on the Census Bureau’s Web site and written entirely by non–Census Bureau individuals (e.g., external researchers at the Census Bureau’s Research Data Centers) must incorporate the disclaimer described above, with an additional statement indicating that the Census Bureau has not reviewed the paper for accuracy or reliability and does not endorse its contents or conclusions.
Personal views may be expressed only if they are appropriate for the paper or presentation because they are on statistical, methodological, technical, or operational issues.
Working papers and professional papers that discuss the results of qualitative research not supported by statistical testing (e.g., based on samples that are not random, are nonrepresentative, or are too small to provide statistical support of the results) must include a caveat explaining why the qualitative methods used do not support statistical testing. The caveat also must address how the findings can (or cannot) be extended to wider populations.
Information products based on data with "serious data quality issues" related to nonsampling error may be written only when their purpose is not to report, analyze, or discuss characteristics of the population or economy, but to:
- Census Bureau publications, new releases, abstracts, and handouts for advisory committee meetings.
- Information products that are distributed internally.
- Information products that have been reviewed and approved by the Associate Director as not needing a disclaimer
because the documents do not contain personal views (e.g., working papers).
- Presentation slides, unless they will be distributed as handouts or published (e.g., in conference proceedings).
Note: Statistical Quality Standard F1, Releasing Information Products describes serious data quality issues and the restrictions on releasing information products with such issues.
- Analyze and discuss data quality issues or research on methodological improvements, or to
- Report results of evaluations or methodological research.
Note: Although not a requirement of the statistical quality standards, the Census Bureau requires presentation slides to use the PowerPoint templates featuring the Census Bureau wordmark provided at the Customer Liaison and Marketing Services Office Intranet Web site.
Sub–Requirement E2–2.2: In addition to the requirements for all information products, the requirements for tabulations include the following:
- The level of detail for tabulations must be appropriate for the level of sampling error, nonsampling error, and any other error associated with the estimates.
- All tabulations, except as noted for custom tabulations in item 4 below, must present estimates that take into account the sample design (e.g., weighted estimates).
- All tabulations, except as noted for custom tabulations in item 4 below, must account for missing or invalid data items (e.g., use imputed data, adjust weights, or display the weighted total of the cases where the data were not reported).
- Custom tabulations must:
- Present weighted estimates unless a client requests unweighted tabulations. If unweighted tabulations are produced for a client, a discussion of the issues associated with using unweighted counts must be provided with the tabulations. Providing a reference (URL) citing the discussion is not sufficient.
- Account for missing or invalid data items unless a client requests custom tabulations that exclude imputed data. If tabulations are produced for a client that exclude imputed data, additional metadata must be provided with the tabulations to describe and quantify the level and the extent of the missing data. Providing a reference (URL) citing the metadata is not sufficient.
- Include measures of statistical uncertainty (e.g., CVs, standard errors, MOEs, confidence intervals, or their Bayesian equivalents) with weighted tabulations, or include a reference (URL) to the measures of statistical uncertainty. If a program manager thinks that computing estimates of sampling error is not feasible (e.g., for reasons of cost, schedule, or resources), the program manager must work with their research and methodology Assistant Division Chief (ADC) to provide the client with acceptable measures of statistical uncertainty or the means to compute them.
Note: Although not a requirement of the statistical quality standards, program managers who produce custom tabulations must refer to and follow the requirements of Data Stewardship Policy DS021, Custom Tabulations.
- If any differences are identified (e.g., with a footnote) as statistically significant in any table within an information product, then all statistically significant differences must be similarly identified in all the tables. However, it is not required to identify statistically significant differences in tables.
- Tabulations must be formatted to promote clarity and comprehension of the data presented.
Examples of formatting practices that promote clarity and comprehension include:
Displaying estimates that equals zero and symbols in tables must be appropriate for the content/subject matter being presented and according to acceptable statistical practice. An estimate that equals zero should be shown as a numeric value, e.g., 0.00 for two-decimal accuracy. The exception is when the estimate is less than half of a unit of measurement from zero and there is a meaningful difference between an actual zero and a rounded zero for the particular statistics. Use the symbol without additional punctuation such as parenthesis. Use an "X" instead of "(X)".
- Presenting at most four dimensions in a cross–tabulation.
- Labeling all variables.
- Using row or column percentages to reinforce the text description of the relationships involved.
- Labeling the type of statistics being presented (e.g., frequency, percentage, means, and standard errors).
- Presenting totals and subtotals when appropriate.
- Labeling header columns for each page in multi–page tabulations.
- Indicating when a data value is suppressed because of disclosure issues.
- Footnoting anomalous values (e.g., outliers).
Examples of approved standard symbols:
- A ‘Z’ means the estimate rounds to zero.
- An ‘S’ means that the estimate is withheld because estimate did not meet publication standards.
- An ‘X’ means that the estimate is not applicable.
- An ‘N’ means that the estimate is not available or not comparable
- A ‘D’ means that the estimate is withheld to avoid disclosing data for individual companies; data
are included in higher level totals
Sub–Requirement E2–2.3: In addition to the requirements for all information products, the requirements for statistical graphs, figures, and maps include the following:
- The dimensions of graphs, figures, and maps must be consistent with the dimensions of the data (e.g., three–dimensional effects must not be used when displaying only two dimensions of data).
- Graphs, figures, and maps must be formatted to promote clarity and comprehension of the data presented.
Examples of formatting practices that promote clarity and comprehension include:
- Labeling axes and including the unit of measure.
- Including a legend that defines acronyms, special terms, and data values.
- Preparing graphs, figures, and maps in the same format throughout the information product.
- Using consistent scales across graphs, figures, or maps that are likely to be compared.
- Using units of measure appropriate to the scale of the graph, figure, or map.
- Starting the base of the graph or figure at zero to avoid giving an inappropriate visual impression.
- Ensuring that color hues correspond to the level of measurement (e.g., a light–to–dark color scheme corresponds
with low–to–high values).
- Complying with accessibility requirements of Section 508 of the U.S. Rehabilitation Act.
Note: The Census Bureau Guideline on the Presentation of Statistical Graphics and the Administrative Customer Service Division (ACSD) Chart Publishing Guidelines provide additional guidance on presenting graphics.
Back to Main