Purpose: The purpose of this standard is to ensure that statistically sound practices are used to generate estimates from models for information products.

Scope: The Census Bureau’s statistical quality standards apply to all information products released by the Census Bureau and the activities that generate those products, including products released to the public, sponsors, joint partners, or other customers. All Census Bureau employees and Special Sworn Status individuals must comply with these standards; this includes contractors and other individuals who receive Census Bureau funding to develop and release Census Bureau information products.

In particular, this standard applies to the production of estimates from models for Census Bureau information products. This standard applies to models (e.g., regression, economic, and log-linear) used to produce estimates, such as:

• Small domain estimates, including small area estimates.
• Demographic estimates and projections.
• Seasonal adjustment of estimates.
• Census coverage estimates.
• Synthetic data to protect microdata from disclosure.

Exclusions:
In addition to the global exclusions listed in the Preface, this standard does not apply to:

• Models that are not used to produce estimates for Census Bureau information products (e.g., models used for imputation or disclosure avoidance which are addressed in Statistical Quality Standard C2, Editing and Imputing Data, and Statistical Quality Standard S1, Protecting Confidentiality, respectively).
  

Key Terms: Autocorrelation function, autoregressive integrated moving average (ARIMA), cross-validation, goodness-of-fit, heteroscedastic, homoscedastic, model, model validation, Monte Carlo simulation, multicollinearity, projection, regression, revisions history, residual, sanitized data, seasonal adjustment, sensitivity analysis, sliding spans, small area estimation, and spectral graphs.

Requirement D2-1: Throughout all processes associated with estimation, unauthorized release of protected information or administratively restricted information must be prevented by following federal laws (e.g., Title 13, Title 15, and Title 26), Census Bureau policies (e.g., Data Stewardship Policies), and additional provisions governing the use of the data (e.g., as may be specified in a memorandum of understanding or data-use agreement). (See Statistical Quality Standard S1, Protecting Confidentiality.)

Requirement D2-2: A plan must be developed that addresses:

1. Purpose and rationale for using a model (e.g., data to compute precise estimates are not available, or modeling with additional data will provide more accuracy).
2. Key estimates that will be generated and the domain of application for the model.
3. Methodologies and assumptions related to the model, such as the:
   1. Model structure (e.g., functional form, variables and parameters, error structure, and domain of interest).
   2. Model estimation procedure (e.g., least squares estimation, maximum likelihood estimation, and demographic estimation methods).
   3. Data source and how the data will be used in the model, including key modifications to the data.
4. Criteria for assessing the model fit (e.g., goodness-of-fit statistics and R-squared) and the model specification (e.g., measures of multicollinearity).
5. Verification and testing of the systems for generating estimates.
6. Verification of the modeled estimates and evaluation of their quality.
   
   Note: Statistical Quality Standard A1, Planning a Data Program, addresses overall planning requirements, including estimates of schedule and costs.

Requirement D2-3: Models must be developed and implemented using statistically sound practices.

Examples of statistically sound model development practices include:

• Ensuring definitions of variables are accurate (e.g., definitions of the geographic areas used in the model, and eligibility criteria in administrative records).
• Specifying a model that has a basis in verified empirical relationships.
• Examining preliminary model results for internal consistency and to ensure that logical relationships among the data are maintained (e.g., population estimates are not negative, and sub-domains (e.g., counties) sum to super-domains (e.g., states).
• Estimating measures of statistical uncertainty (e.g., prediction error variances, measures of error associated with using synthetic data, or the Bayesian equivalents of these measures).
• Modifying the functional form, the variables, or the parameters of the model to address problems revealed by the model diagnostics and error estimates.
• Having experts perform a methodological review.
• Producing estimates using weighted data, when appropriate.
• Providing justification that the sample design and selection are adequately accounted for in the estimation process.

Examples of statistically sound practices for demographic estimates and projections include:

• Basing assumptions about future relationships among variables on empirical data or on assumptions that are considered statistically sound.
• Comparing raked and unraked data to ensure logical relationships are maintained.
• Providing quantitative or qualitative assessments of uncertainty for each estimated or projected data point, whenever possible.

Examples of statistically sound practices for seasonal adjustments include:

• Before the first seasonal adjustment of a series, conducting a seasonal analysis to determine whether seasonal patterns exist and periodically repeating the analysis.
• Seasonally adjusting a time series only when data exhibit seasonal patterns.
• Seasonally adjusting only those component series that show identifiable seasonality for aggregate series derived from the combination of component series.
• Using autoregressive integrated moving average (ARIMA) extrapolations in calculating seasonal factors (e.g., the X-12-ARIMA method).
• Reviewing appropriate modeling and seasonal adjustment diagnostics (e.g., revisions history, spectral graphs, plots of the sample autocorrelation function of the model residuals, forecast performance, and sliding spans) for valuable information about model adequacy and adjustment stability.

Sub-Requirement D2-3.1: Model results must be evaluated and validated, and the results of the evaluation and validation must be documented.

Examples of evaluation and validation activities include:

• Validating the model by comparing with independent information sources.
• Generating and reviewing goodness-of-fit statistics (e.g., R-squared and F-tests).
• Generating and reviewing model diagnostics and graphical output (e.g., reviewing for outliers, multicollinearity, heteroscedasticity, homoscedasticity, and influential observations).
• Cross-validating the model using a subset of data withheld from the model fitting.
• Conducting sensitivity analyses to violations of the assumptions (e.g., Monte Carlo simulations).

Note: Evaluation and validation is required when the model is developed.  Models used in a continuing production setting must be re-evaluated periodically as appropriate.

Sub-Requirement D2-3.2: Specifications for the modeling and estimation systems must be developed and implemented.

Examples of issues that specifications might address include:

• Descriptions of data files to be used in the model.
• Equations for computing estimates and variances.
• Instructions for running production software.
• Estimation algorithms.
• Convergence criteria for iterative models.

Sub-Requirement D2-3.3:; Estimation systems must be verified and tested to ensure that all components function as intended.

Examples of verification and testing activities include:

• Using subject matter and statistical experts to review the estimation methodology.
• Checking that the appropriate equations were used.
• Verifying that the specifications reflect requirements.
• Validating computer code against specifications.
• Assessing computer code to ensure that the appropriate data and variables are used and the code is correctly programmed.
• Performing test runs and debugging computer code.
• Using different random starts to ensure models using maximum likelihood estimates converge consistently.

Sub-Requirement D2-3.4: Methods and systems must be developed and implemented to verify the modeled estimates and evaluate their quality.

Examples of verification and evaluation activities include:

• Performing sensitivity analyses using alternative assumptions to inform users of model stability.
• Examining measures of statistical uncertainty.
• Ensuring that variances reflect both sampling error and modeling error.
• Comparing production estimates against comparable data from other sources, including previous estimates for the program or projections from prior cycles.
• Reviewing goodness-of-fit statistics and model diagnostics and documenting unexpected results to aid the revision of the model for the next cycle.
• Reviewing (during each seasonal adjustment run) newly identified outliers and changes to previously identified extreme values that may cause large revisions in the seasonally adjusted series.

Note: Statistical Quality Standard D3, Producing Measures and Indicators of Nonsampling Error, provides requirements for measuring and evaluating nonsampling error.

Sub-Requirement D2-3.4.1: The seasonal adjustment process and results must be reviewed annually by the program manager (or the appropriate mathematical statistician) to identify needed changes in the X-12-ARIMA specification files. Using the required secure data transmission protocols, the program manager (or the appropriate mathematical statistician) must provide the following to the Time Series Methods Staff (TSMS) of the Office of Statistical Methods and Research for Economic Programs (OSMREP):

1. The new final X-12-ARIMA specification files and the data used.
2. The revised X-12-ARIMA specification file and the data used, whenever the seasonal adjustment options must be changed outside of the annual review period. This information must be provided immediately after release of the adjusted data.

Sub-Requirement D2-3.4.2: For indicator releases, any routine revisions to the annual review process, such as benchmarking and updating of seasonality factors, must be consolidated and released simultaneously. See Statistical Policy Directive No. 3. Deviations from this requirement must be approved as specified in the directive.

Requirement D2-4: Documentation needed to replicate and evaluate the modeling activities must be produced. The documentation must be retained, consistent with applicable policies and data use agreements, and must be made available to Census Bureau employees who need it to carry out their work. (See Statistical Quality Standard S2, Managing Data and Documents.)

Examples of documentation include:

• Plans, requirements, specifications, and procedures for the estimation systems.
• Data files with weighted and unweighted data.
• Computer source code.
• Results of outlier analyses, including information on cause of outliers, if available.
• Results of model diagnostics.
• Output data file with “predicted” results for every unit of analysis.
• Seasonal adjustment diagnostic measures (e.g., revisions history values and graphs, spectral graphs, forecast error values and graphs, and sliding spans results).
• Error estimates, parameter estimates, and overall performance statistics (e.g., goodness-of-fit and other such statistics).
• Methodologies used to improve the estimates.
• Quality measures and evaluation results. (See Statistical Quality Standard D3, Producing Measures and Indicators of Nonsampling Error.)

Notes:

1. The documentation must be released on request to external users, unless the information is subject to legal protections or administrative restrictions that would preclude its release. (See Data Stewardship Policy DS007, Information Security Management Program.)
2. Statistical Quality Standard F2, Providing Documentation to Support Transparency in Information Products, contains specific requirements about documentation that must be readily accessible to the public to ensure transparency of information products released by the Census Bureau.