Contents | Introduction | Part A | Part B | Part C

The Census 2000 Testing, Experimentation, and Evaluation Program

May, 2002

Part B:The Census 2000 Testing and Experimentation Program

Introduction

A successful decennial census, one that is responsive to the nation's changing needs, cannot be achieved without early planning. Many key issues for Census 2000, such as declining public cooperation and tighter funding restrictions, were already being studied in the late 1980s. Fundamental operational changes, such as those designed to improve the process for capturing information on the census questionnaires, came from this early research. For a decennial census, much lead time is needed to first identify and test promising new procedures, make adjustments, and retest as needed. Substantial lead time is particularly necessary for the procurement and testing of many different types of equipment that must be in place to conduct the decennial census.

Early in 1997, the Census Bureau formed a team to develop the Census 2000 program of testing and experimentation. The tests and experiments were conducted concurrently with Census 2000 because the decennial census environment provided the best possible conditions to learn about the value of new or different methodologies. Research conducted during the decennial census is expected to guide future decennial census designs, but also provide valuable information for use by other areas of the Census Bureau.

Planned Tests and Experiments

Summary descriptions of the tests and experiments conducted in Census 2000 are provided on the following pages.

Census 2000 Alternative Questionnaire Experiment (AQE2000)

Overview

This experiment was designed to manipulate three independent questionnaire design components. The first component evaluates the effects of the amount and presentation of the residence rules on the short form; that is, in comparison to the current presentation would a briefer and reformatted presentation of the rules improve data quality? Since this is a coverage issue, a reinterview was conducted. The second component examined the presentation of the race question to determine whether changes in the way the race questions were asked in the 1990 and 2000 censuses affect the quality and content of race data. Specifically, it evaluated the combined effects of variant question wording, format, content, and design on race data quality and content. The third component pertains to the long form, specifically in regard to the design of the skip instructions to determine whether the current format facilitated respondent's navigation through the form correctly. "Skip to" and "go to" instruction variations were examined. Information learned about the long form will advise implementation of the American Community Survey.

Objectives

The objectives of this experiment were to continue efforts to develop a user friendly mail-out questionnaire that can be completed accurately by respondents and to evaluate the effects of questionnaire changes on the data. Corresponding to the variables described above, the specific objectives were: 1) to compare the Census 2000 short form, defined as containing a full set of residence rules, with a revised form that contains an alternate presentation of the rules, 2) to compare the Census 2000 short form presentation and sequencing of the race question, including its provision for marking multiple categories, to that of the 1990 presentation and instructions, and 3) to compare the standard skip instruction on the Census 2000 long form with four revised formats.

Administrative Records Census 2000 Experiment (AREX 2000)

Overview

The Plan for Census 2000 explicitly called for experimentation with an administrative records census for two reasons: 1) use of administrative records as the primary data collection method has tremendous potential for cost savings, and 2) significant testing of administrative records was not done as part of the 1990 Census testing and experimentation program and, as a result, the Census Bureau was not sufficiently prepared to include administrative records in the Census 2000 design. The potential benefit of an administrative records census is to reduce the cost and response burden of direct data collection.

The AREX 2000 explored two methods for conducting an administrative records census. In both methods, national-level administrative records were assembled, unduplicated using Social Security Numbers, and assigned block-level geographic codes. Records for the selected test sites (approximately one million housing units in five counties) were extracted and tallied at the census block level. The two methods differ in their use of the Master Address File to create a universe (frame) of housing units. The first method did not use the Master Address File but provided only population counts at the block level. The second method matched administrative records to addresses on the Master Address File and reconciled differences through field operations. This method provided both population and housing unit counts at the block level.

The experiment included the following field/mailout operations: 1) a clerical geocoding operation to be conducted at selected Regional Census Centers, 2) a field address verification operation, and 3) a mailout to P.O. Box and rural-style addresses to obtain geocodable house number/street name information.

Objectives

The AREX 2000 compared two methods for conducting an administrative records census to Census 2000 and evaluated the results and costs. The data analysis for the experiment included comparisons of site, census tract and block level population and housing counts from AREX 2000 and Census 2000. The analysis also examined the similarities and differences of population characteristics (age, gender, race/ethnicity) and simulate the replacement of Census 2000 nonresponse household enumerations with administrative record information. Secondary objectives included collecting relevant information that was only available in 2000 to be used for ongoing testing and planning for administrative records use in the 2010 Census and for comparing an administrative records census to other potential 2010 methodologies.

Social Security Number, Privacy Attitudes, and Notification Experiment

Overview

The purpose of the SPAN was to obtain behavioral and attitudinal data on several topics related to the use of administrative records. This included how the public responded to requests for Social Security Numbers (SSNs) on decennial census questionnaires, how the public responds to differently worded notifications about the Census Bureau's use of administrative records, and on what were the public's attitudes on privacy and confidentiality pertaining to the notion of an administrative records census.

The SPAN consisted of two components. The first component collected data on requesting the SSN and the use of differently worded notifications. The second component involved a telephone survey that measured the public's attitudes on privacy and confidentiality issues.

Objectives for Component 1: Specific objectives were to determine: 1) what effect a request for the SSN for every household member has on mail and item response, 2) what effect a request for an SSN for only the person completing the questionnaire has on mail and item response, 3) the accuracy of the respondent-provided SSNs, and 4) what effect different notifications on the Census Bureau's possible use of administrative records has on mail and item response rates. The methodology for achieving these objectives involved the mailout of seven short form and three long form panels -- each panel containing a 5,000 sample -- for a total of 50,000 forms during Census 2000. The long form panels included only the notification test with no requests for SSNs.

There were two notifications -- referred to as "general" and "specific." Each notification was included in the cover letter and described how or why the Census Bureau may use administrative records data from other Federal agencies. A "general" notification mentioned the Census Bureau's possible use of statistical data from other Federal agencies, while the "specific" notification goes further to mention actual Federal agencies, such as the Internal Revenue Service, Social Security Administration, and "agencies providing public housing assistance."

Objectives for Component 2: The second component of the SPAN was a telephone survey that measured the public's attitudes on privacy and confidentiality issues pertaining to an administrative records census. This survey included pre- and post-measurements to Census 2000 to enable examination of the census environment's effect on privacy attitudes. The pre-measurement took place before the national paid advertising and field recruiting campaigns. The post-measurement occurred shortly after Census Day, April 1, 2000. Each measurement group was a national sample of 2,000 households. Specific objectives were to: 1) determine the

public's opinion of the Federal government and the Census Bureau in general, 2) assess change in the public's attitudes on privacy-related issues using results from studies conducted in 1995 and 1996, and 3) determine the public's opinion of the Census Bureau's testing on expanding the use of administrative records, possible interest in collecting SSNs in the future, and the notion of an "administrative records census."

Response Mode and Incentive Experiment (RMIE)

Overview

This experiment measured the effect of an incentive and/or option of alternative electronic modes of collection on response to the census short form. Since 1970, response to the mailed form has declined and labor costs to visit nonresponding households have greatly increased. To address these problems, the Census Bureau explored other methods and technologies to count the population, such as incentives, telephones, and the Internet. This experiment determined what effect an incentive has on getting respondents to answer the census using one of three electronic modes of collection. The effectiveness of the incentive also was measured on households not responding to the mailout of the standard Census 2000 questionnaire.

The alternative modes of collection were:

Employing the collection modes listed above, this experiment incorporated two treatments: response mode and incentive, each with three panels. There also was a control group consisting of three panels which served as the universe for the response phase of the experiment. The total mailout for all panels was 35,380 households. The incentive was a telephone calling card worth thirty minutes of free long distance service which was activated after response. Data analysis was conducted on seven experimental components that included: 1) initial mailout/operator assistance, 2) nonresponse, 3) ASQ - name recognition, 4) ASQ - customer satisfaction, 5) Internet Usage Survey (telephone followup), 6) Internet Customer Satisfaction Survey (administered on the Internet), and 7) Internet administrative data.

Objectives

This experiment has the following key objectives:

Census 2000 Supplementary Survey

Overview

Census 2000 included a long form for 1-in-6 households across the country. Essentially the same process has been used since 1940 to collect basic socioeconomic information (such as educational, marital, and veterans' status; housing characteristics; and commuting patterns) for all geographic areas of the United States, ranging from the national down to the census tract level.

In spite of the efficiencies of using the decennial census to collect this critical socioeconomic data, there is a strong interest in moving away from this approach -- both to simplify the census process and to provide more current and more accurate data for federal, state and local users. In response to this interest, Census 2000 included, in addition to a traditional long form, a supplementary survey designed as the operational feasibility test of collecting long form data throughout the country during the same time frame but in a process separate from the census.

Objectives

The objective of the Census 2000 Supplementary Survey was to demonstrate the operational and technical feasibility of collecting the full range of socioeconomic data gathered on the decennial census long form using a different questionnaire and estimation methodology. To accomplish this objective, the Census 2000 Supplementary Survey was conducted during Census 2000 using an existing questionnaire -- that of the American Community Survey. Results will inform the process of removing the long form from the census.

Use of the Employee Reliability Inventory File for Nonresponse Followup Enumerators

Overview

The Office of Personnel Management (OPM) reported that the tests used to hire decennial staff, while valid, do not assess an important aspect of the knowledge, skills, and abilities needed for successful performance -- interpersonal skills. This experiment will help determine if the Employee Reliability Inventory (ERI) meets each of three criteria for a valid selection test and if it is appropriate for use in selecting decennial census enumerators. To be considered a valid selection aid, the personality-based competencies measured by a noncognitive test should: 1) be job-relevant, 2) have no between-group differences, and 3) be subject-related. To measure the noncognitive competencies of census new hires, an already existing noncognitive instrument from the testing market (ERI) was administered to a sample of people hired to be nonresponse followup enumerators. The research will answer the following questions:

Objectives

The overall objective of this study was to determine if an existing noncognitive test provided a reliable and valid measure of interpersonal skills that can be used by the census to make more precise employee hiring decisions. The goal was to determine if the Census Bureau could reduce interviewer turnover and improve interviewer work performance by improving enumerator selection tools through the use of a commercial noncognitive test -- the ERI.

Ethnographic Studies

Overview

The Census Bureau used ethnographic techniques to study survey coverage as early as 1971. The National Academy of Science's Panel on Decennial Census Methodology, established by the Census Bureau in 1984, recommended that the Census Bureau undertake a series of participant observation coverage studies in selected areas. Exploratory ethnographic research was initiated in a number of communities to identify and explain the complex behavioral processes that lead to underenumeration. Based on the experience obtained in preliminary research, the Census Bureau launched its most ambitious phase of ethnographic studies associated with the 1990 Census. More than 40 additional exploratory and ethnographic studies and evaluations were conducted on a wide range of populations-such as the homeless, migrant workers, African Americans, Latinos, American Indians, and Asians-and issues such as respondents' understanding of census language and concepts, and other types of communications.

Objectives

Ethnographic studies conducted in association with Census 2000 provided new insights that can be used to improve coverage of selected segments of this nation's population. The following studies reflect selected social and demographic aspects in American society that are important to explore from an ethnographic perspective. This perspective, grounded in the actual behavior of respondents, can offer insights which other methods may not capture.

Protecting Privacy: Information, Trust and Technology in the Decennial Census and Demographic Surveys: The goal of this project was to conduct a qualitative study of belief structures that influence survey respondents' perceptions of, and reactions to, survey information requests, focusing on privacy concerns. This study explored how respondents assess the consequences of survey participation and survey response, their sense of information ownership, their reactions to confidentiality statements, and their reasons for choosing to participate in survey data collections.

Complex Households and Relationships in the Decennial Census and Demographic Surveys: This ethnographic research project had three objectives: 1) to explore the range and functioning of complex households within different ethnic groups in the United States, 2) to examine how the response categories of the decennial relationship question capture the emerging diversity of household types, and 3) to compare the household composition and relationship information collected by the ethnographic interviews to those in Census 2000. This study was designed to assess how well census methods, questions, relationship categories, and household composition typologies describe the emerging diversity of household types in this country. Six ethnographers or teams each conducted 25 ethnographic interviews with a selected ethnic/race group: African Americans, Hispanics, Inupiaq Eskimos, Koreans, Navajos, or Whites.

Generation X Speaks Out on Censuses, Surveys, and Civic Engagement: An Ethnographic Approach: The purpose of this nationwide ethnographic research was to examine civic engagement, behaviors, and attitudes towards censuses and surveys among Gen-Xers (individuals born during the years 1968-1979) from varied socioeconomic backgrounds and ethnicities, including individuals from hard-to-enumerate categories, such as young minority males and immigrants. Patterns of civic engagement have consequences for government data collection efforts in terms of survey nonresponse, trust and privacy concerns, policy-oriented issues and effective educational outreach campaigns. Millennial Generation individuals (14-18 years of age) were also interviewed in order that comparative life-cycle experiences and cultural explanations emerge with regard to census and survey nonresponse, government engagement, and civic responsibility and obligation.

Patterns of civic engagement have consequences for government data collection efforts in terms of survey nonresponse, trust and privacy concerns, policy-oriented issues and effective educational outreach campaigns. Millennial Generation individuals (14-18 years of age) were also interviewed in order that comparative life-cycle experiences and cultural explanations emerge with regard to census and survey nonresponse, government engagement, and civic responsibility and obligation.