Processing and Editing of Data

Census Bureau Principle
Version 1.2
Issued: 05 Jan 06

Authored for:
Census Bureau Methodology & Standards Council
Cynthia Z.F. Clark , Chair

Introduction

The Census Bureau’s commitment to quality and professional standards of practice includes the goal of processing survey and census data to prepare the data as collected for meaningful consumption by end users.

Scope

The term “processing” encompasses a variety of activities, including administrative functions (such as parameter development and management information systems); post-collection processes (such as editing, imputation, data review and correction, data query, weighting, estimation and variance estimation, and application of disclosure limitation rules); and support functions for collection technologies (such as mailout, check-in, data capture, and follow-up). Of these various processes, editing and imputing data are described by this set of principles, while weighting is described within the Production of Estimates or Projections principles.

Principles

1. Processing systems will be as integrated as possible (that is, output from one processing activity should be useable as input to another).
2. Editing will clean up the data and reduce inconsistent or illogical entries, resulting in more accurate, cohesive and comprehensive survey data results, and provide the basis for future improvement of the survey vehicle, and provide information about the quality of the survey data.
3. While inconsistent or invalid entries will be removed from the datasets to maintain the credibility of the issuing agency and to facilitate further automated data processing and analysis, this goal must be balanced against the costs of the process and the potential to bias the data.
4. A data item or series of items that are missing or deemed to be inconsistent or illogical by the editing process may be subjected to imputation processes that substitute valid values for the missing or illogical values.
5. Imputation processes will be designed to limit the bias caused by missing or incorrect data.
6. Imputation processes are to be repeatable and objective. This is best accomplished through automation.
7. To preserve the reported data, the minimum number of edit-failing data fields required will be imputed to ensure that the imputed record satisfies the edits.
8. The choice of treatment for missing or failed-edit data will depend on the end-use of the data and the level of reporting for a given item or series of items.
9. Imputation methods will include audit functions that identify the imputation methods used for a particular data item and the source of data used for each imputed value.
10. Exemption from the requirements of these Principles may be sought in special circumstances using the procedure described below under “Responsibilities.”
    

Responsibilities

Program areas will be responsible for the following:

• Developing requirements to satisfy these Principles in the design, schedule, and budget of new or re-designed statistical activities, programs, or products; and
• Submitting applications for exemption from the requirements of these Principles to the Census Bureau Methodology and Standards Council.

The Methodology and Standards Council will be responsible for the following:

• Providing standards and guidelines on the application of these Principles to program areas;
• Producing reports as necessary on the state of compliance with these Principles;
• Initiating evaluations of the application of these Principles within particular program areas and ensuring that such evaluations are co-ordinated with other program evaluation activities;
• Initiating a review of these Principles and accompanying standards and guidelines when deemed necessary; and
• Reviewing applications for exemption from the requirements of the Principles.

Inquiries

Inquiries relating to the interpretation of these Principles should be addressed to the Census Bureau Methodology and Standards Council.

Approved by Census Bureau Methodology and Standards Council:

                  Signed Cynthia Z.F. Clark, August 6, 2002  
                  Cynthia Z.F. Clark
                  Associate Director for Methodology and Standards
                  Chair, Methodology and Standards Council

                  Signed Robert N. Tinari, August 06, 2002
                  Robert Tinari
                  Chief, Computer Assisted Survey Research Office

                  Signed David Whitford, August 6, 2002
                  David C. Whitford
                  Acting Chief, Decennial Statistical Studies Division

                  Signed Al R. Tupek, August 6, 2002
                  Alan R. Tupek
                  Chief, Demographic Statistical Methods Division

                  Signed Howard Hogan, August 6, 2002
                  Howard Hogan
                  Chief, Economic Statistical Methods and Programming Division

                  Signed Ruth Ann Killion, August 6, 2002
                  Ruth Ann Killion
                  Chief, Planning, Research and Evaluation Division

                  Signed Tommy Wright, August 6, 2002
                  Tommy Wright
                  Chief, Statistical Research Division

Concurrence:

                  Signed Preston Jay Waite, June 11, 2003
                  Preston Jay Waite
                  Associate Director for Decennial Programs

                  Signed Nancy M. Gordon, June 11, 2003
                  Nancy M. Gordon
                  Associate Director for Demographic Programs

                  Signed Frederick T. Knickerbocker, June 11, 2003
                  Frederick T. Knickerbocker
                  Associate Director for Economic Programs

                  Signed Marvin D. Raines, June 11, 2003
                  Marvin D. Raines
                  Associate Director for Field Operations

                  Signed Gloria Gutierrez ,June 11, 2003
                  Gloria Gutierrez
                  Assistant Director for Marketing and Customer Liaison

                  Signed Cynthia Z.F. Clark ,June 11, 2003
                  Cynthia Z.F. Clark
                  Associate Director for Methodology and Standards

Document Management & Control

The most current version of this document is maintained on the Census Bureau Intranet and may
be accessed from the Quality Management Repository.

Category: Principle   
Filename: P05-0_v1.2 Process Edit