All surveys, including SIPP, are subject to nonsampling errors from various sources. SIPP contains nonsampling errors common to most surveys, as well as errors that stem from SIPP's longitudinal design. SIPP experiences some differential undercoverage of demographic subgroups. To compensate for this undercoverage, the Census Bureau adjusts SIPP sample weights to population control totals. Little is known, however, about how effective these adjustments are in reducing biases in surveys.
SIPP also experiences sample attrition. This is a common concern in longitudinal surveys because of the need to follow the same people over time. Attrition reduces the available sample size, and to the extent that those leaving the sample are systematically different from those who remain in the sample, survey estimates could be biased.
Response errors also occur in SIPP and take on a number of forms. Recall errors, for example, are thought to be the source of the "seam phenomenon." This effect occurs when a respondent projects current circumstances back onto the survey’s reference period (the prior calendar year for the 2014 Panel and after, and the prior 4 months for 2008 and earlier panels). This causes any changes in respondent circumstances that occurred during the reference period to appear to have happened at the beginning of the reference period. The effect is a disproportionate number of changes that appear to occur between the last month of one wave and the first month of the following wave – the "seam" between their two interview waves.
Another potential source of nonsampling error is the time-in-sample effect. This effect refers to the tendency of sample members to "learn the survey" over time. The more times a sample member is interviewed, the better they learn the questionnaire. The concern is that sample members will alter their responses to the survey questions to conceal sensitive information or to minimize the length of the interview.
A considerable amount of research has been conducted to investigate the various sources of non-sampling error in SIPP. The results of this research are summarized in the SIPP Quality Profile, 3rd edition, available at www.census.gov/library/working-papers/1998/demo/SEHSD-WP1998-11.html Additional findings about SIPP data quality, especially for more recent panels, can be found in the National Research Council’s 2018 report, The 2014 Redesign of the Survey of Income and Program Participation: An Assessment, and in Appendix A of the National Research Council’s 2009 report, Reengineering the Survey of Income and Program Participation. Despite the volume of methodological research, it remains difficult to quantify the combined effects of non-sampling errors on SIPP estimates. This problem is made more complex because the effects of different types of non-sampling error on survey estimates vary, depending on the estimate under consideration. However, there are some findings about non-sampling error that SIPP users should bear in mind when conducting their analyses and examining their results. Those findings include:
The Census Bureau has done nonresponse bias studies to investigate the effect of decreasing response rates, but more work needs to be done to truly quantify that bias (McMillan & Culver, 2013). These studies are available at www.census.gov/programs-surveys/sipp/tech-documentation/nonresponse-reports.html