SIPP Nonsampling Error

All surveys—including SIPP—are subject to nonsampling errors from various sources. SIPP contains nonsampling errors common to most surveys, as well as errors that stem from SIPP's longitudinal design. Undercoverage in household surveys is due primarily to within-household omissions; the omission of entire households is less frequent. SIPP experiences some differential undercoverage of demographic subgroups; for example, the coverage ratio of black males over 15 years of age is much lower than that for white males in the same age group. To compensate for this differential undercoverage, the Census Bureau adjusts SIPP sample weights to population control totals. Little is known, however, about how effective those adjustments are in reducing biases.

Sample attrition is another major concern in SIPP because of the need to follow the same people over time. Attrition reduces the available sample size. To the extent that those leaving the sample are systematically different from those who remain in the sample, survey estimates could be biased.

Response errors in SIPP take on a number of forms. Recall errors are thought to be the source of the "seam phenomenon." This effect results from the respondent's tendency to project current circumstances back onto each of the 4 prior months that constitute the SIPP reference period. When that happens, any changes in respondent circumstances that occurred during that 4-month period appear to have happened in the first month of the reference period. A disproportionate number of changes appear to occur between the fourth month of one wave and the first month of the following wave, which is the "seam" between the two waves—hence the name.

Another potential source of response error is the time-in-sample effect. This effect refers to the tendency of sample members to "learn the survey" over time. The more times a sample member is interviewed, the better he or she learns the questionnaire. The concern is that sample members will alter their responses to the survey questions in an effort to conceal sensitive information or to minimize the length of the interview.

Undercoverage

One source of error in SIPP, as in other household surveys, is differential undercoverage of demographic subgroups. Black males over 15 years of age are most affected by undercoverage. The coverage ratio for this subgroup was about 0.82 in the 1990 and 1991 SIPP Panels. (Coverage ratio is computed as the survey estimate of the number in the subgroup before post- stratification, divided by a population estimate for the subgroup from population projections based on the most recent census.) For black males in their mid to late 20s, the coverage ratio was lower, about 0.65 in the same panels (SIPP Quality Profile, 3rd Ed. [U.S. Census Bureau, 1998a, Chapter 3]; hereinafter in this chapter, SIPP Quality Profile, 3rd Ed). These coverage ratios may understate the magnitude of the coverage problems because census undercounts are not reflected in the coverage ratios before 1992. Undercoverage in household surveys is attributed mainly to within-household omissions; the omission of entire households is less frequent. Shapiro et al. (1993) estimated that about 70 percent of the undercoverage for young black males consists of within-household omissions; the corresponding percentage for the white population is about 60 percent. To compensate for undercoverage, the Census Bureau uses population controls to adjust SIPP weights. Little is known about the effectiveness of the adjustments in reducing biases.

Nonresponse

Nonresponse is a major concern in SIPP because of the need to follow the same people over time. In SIPP, nonresponse can occur at several levels: household nonresponse at the first wave and thereafter; person nonresponse in interviewed households; and item nonresponse, including complete nonresponse to topical modules. At the household level, the rate of sample loss for the 1991 Panel rose from about 8 percent at Wave 1 to more than 21 percent by Wave 8. For the same panel, 23 percent of the original sample persons who participated in Wave 1 missed one or more interviews for which they were eligible in later waves. At the item level, the nonresponse rate is typically around 10 percent or less for items on income amounts but somewhat higher for items on asset amounts. Nonresponse reduces the effective sample size (and, therefore, increases sampling error) and introduces bias in the survey estimates. The Census Bureau uses a combination of weighting and imputation methods to reduce the biasing effects of nonresponse at all three levels in SIPP. The effectiveness of those procedures remains a matter of ongoing review and research (SIPP Quality Profile, 3rd Ed., Chapters 4, 5, and 8).

Measurement Errors

Measurement errors are associated with the data collection phase of the survey. They may vary across SIPP panels because of changes in data collection procedures over the years. Most core survey items in SIPP are used consistently at every panel, although there have been occasional changes to improve the clarity of some items. The data collection method, which was face-to- face interviewing for the early panels, was changed to a maximum use of telephone interviewing in February 1992. Telephone interviewing was used as the primary mode of data collection between February 1992 and January 1996 for all waves except Waves 1, 2, and 6, for which face-to-face interviewing was used. The switch to telephone interviewing has had no known adverse effects on data quality.

Computer-assisted interviewing (CAI) was introduced with the 1996 SIPP Panel. The effects of CAI on survey responses have yet to be determined (SIPP Quality Profile, 3rd Ed., Section 11.3). For the 1996 Panel, computer-assisted personal interviewing (CAPI) was used for Waves 1 and 2. After Wave 2, the field representatives used the CAI instrument in face-to-face interviews with approximately one-third of the respondents; for the remaining interviews, the field representatives used the CAI instrument but conducted telephone interviews from their homes.

The combination of face-to-face interviews and telephone interviews used across waves is prespecified and varies for different subgroups of the sample according to the following scheme (Waite, 1996). Sample members are assigned to one of three interviewing mode subgroups. For each subgroup, a pattern of interviewing modes is designated and repeated every three waves. Thus, for Waves 3, 4, and 5, subgroup 1 is assigned the sequence face-to-face, telephone, telephone; subgroup 2, the sequence telephone, face-to-face, telephone; and subgroup 3, the sequence telephone, telephone, face-to-face. Under this scheme, which is applied with each rotation group, one-third of the sample is interviewed in person each wave and each month, and every household is interviewed in person once a year. The same sequence is repeated for Waves 6 and beyond, with a cycle of three waves (SIPP Quality Profile, 3rd Ed.).

Response errors in SIPP include errors of recall, errors in proxy respondents. reports, and other errors associated with the panel nature of SIPP. SIPP uses a 4-month recall period to reduce memory error, and respondents are encouraged to use financial records and an event calendar to facilitate recall. Although the level of accuracy for self-response is generally believed to be higher than for proxy response (see Moore, 1988, for a contrary view), achieving a higher proportion of self-response would increase data collection costs and might lead to some increase in person nonresponse rates (SIPP Quality Profile, 3rd Ed., Section 4.5.3).

A potential source of response error that arises from the panel nature of SIPP is the time-in- sample effect (or panel conditioning). This effect occurs when the responses given at later waves are affected by the respondents. experiences of being interviewed in previous waves. The extent of this error is difficult to evaluate because it is often confounded with other sources of error, particularly attrition. Thus far, studies have found little evidence of systematic biases resulting from time-in-sample effects (Pennell and Lepkowski, 1992; McCormick et al., 1992).

Measurement errors can also occur when respondents misinterpret questions. For example, when asked about earnings, some respondents may have reported take-home pay instead of gross earnings. There is also some evidence of confusion in regard to welfare programs, such as the old Aid to Families with Dependent Children and general assistance programs.

Research has consistently indicated that respondents tend to report the same status (e.g., employment or program participation) and the same amounts (e.g., Social Security income) for all 4 months within a wave, with most reported changes occurring between the last month of one wave and the first month of the subsequent wave. This phenomenon results in an overstatement of changes at the on-seam months (the boundary between interviews in successive waves of a panel) and an understatement of changes at the off-seam months. The seam phenomenon affects most variables for which monthly data are collected. As a result of the rotation group pattern, the phenomenon has relatively small effects on cross-sectional estimates based on all four rotation groups. That is because there is only one rotation group (or one-fourth of the sample) that is on seam and three rotation groups off seam for any given pair of calendar months. The effects of the seam phenomenon on longitudinal estimates are not well known (SIPP Quality Profile, 3rd Ed., Chapter 6).

Effects of Nonsampling Error on Survey Estimates

A considerable amount of research has been conducted to investigate the various sources of nonsampling error in SIPP. The results of the research are summarized in the SIPP Quality Profile, 3rd Ed.). The research includes, for example, the SIPP Record Check Studies (Marquis and Moore, 1989a,b, 1990; Marquis et al., 1990) that compared SIPP responses on program participation with administrative records. Despite the volume of this methodological research, it remains difficult to quantify the combined effects of nonsampling errors on SIPP estimates. The problem is made more complex because the effects of nonsampling error of different types on survey estimates vary, depending on the estimate under consideration. There are, however, some findings about nonsampling error that SIPP users should bear in mind when conducting their analyses and examining their results. Those findings include the following:

Some demographic subgroups are underrepresented in SIPP because of undercoverage and nonresponse. They include young black males, metropolitan residents, renters, people who changed addresses during a panel (movers), and people who were divorced, separated, or widowed. The Census Bureau uses weighting adjustments and imputation to correct the underrepresentation. Those procedures, however, may not fully correct for all potential biases (SIPP Quality Profile, 3rd Ed., Chapter 8).
The SIPP estimates of income from Social Security, Railroad Retirement, and Supplemental Security programs represent more than 95 percent of the amounts reported by administrative sources. The SIPP estimates of unemployment income, workers. compensation income, veteran.s income, and public assistance income, however, are low relative to the amounts reported by administrative sources (Coder and Scoon-Rogers, 1996).
Evaluation studies typically find that SIPP estimates (as well as other survey estimates) of property income are generally poor. Among the different types of property income, reports of interest and dividend income are most prone to error. Respondents are often confused about those two sources of income, and both sources tend to be underreported (Coder and Scoon- Rogers, 1996).
SIPP estimates of assets, liabilities, and wealth are low relative to estimates from the Federal Reserve Board (Eargle, 1990).
For SIPP panels before 1996, the estimates of the percentages of people in poverty were lower than those found in the Current Population Survey (CPS) (Shea, 1995a).
SIPP estimates of the working population differ from those produced from CPS. The differences may be explained largely by substantial conceptual and operational differences in the collection of labor force data in the two surveys (SIPP Quality Profile, 3rd Ed., Chapter 10).
The SIPP estimates of people without any health insurance coverage are much lower than the CPS estimates. There are reasons to believe that the SIPP estimates are more accurate (McNeil, 1988).
The SIPP estimates of the number of births compare favorably with the CPS estimates. Both surveys, however, provide estimates that are low relative to the records from the National Center for Health Statistics (NCHS). The SIPP estimates of the number of marriages are fairly comparable with the NCHS counts, but the SIPP estimates of the number of divorces are consistently lower than the NCHS estimates (SIPP Quality Profile, 3rd Ed., Chapter 10).

In spell analyses, Kalton et al. (1992) found that spell durations of multiples of 4 months (e.g., 4 months, 8 months, 12 months) were particularly common, a feature that can be explained by the seam phenomenon.

Page Last Modified: May 9, 2006

Census Bureau Links: Home · Search · Subjects A-Z · FAQs · Data Tools · Catalog · Census 2000 · Quality · Privacy Policy · Contact Us