I.   Introduction

The most significant feature of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA or the welfare reform) has directed Bureau of the Census (BOC) to undertake the task of collecting data that permits researchers and analysts to measure the impact of the welfare reform.  The most significant feature of the welfare reform may be the shift of the burden of welfare program administration from the Federal government to state governments. To this end, BOC is conducting the Survey of Program Dynamics (SPD).  The SPD represents the most substantial effort toward creating a nationally representative longitudinal data set that can be used to examine how the welfare reform has affected families and children who were previously dependent on the Federal entitlement programs, over a ten-year period.

Based on the objective of the SPD described above, the current SPD 1997 (Bridge) sample was drawn from the samples of the Survey of Income and Program Participation (SIPP) Panel 1992 and 1993. The current SPD sample selected only  the sample households that were respondent ones in the last interview waves of the SIPP Panels 1992 and 1993. Some of our SPD data users have stated that the SPD data will not be viable because we have not followed every sample person from the SIPP Panel 1992 and 1993.  As an initial step toward resolving this concern, an exploratory attrition study was conducted. Hereinafter, this study will be referred to as the SPD Exploratory Attrition Study (EAS). The objective of the SPD EAS is to conduct a test designed to measure the possible success rate and cost of tracking down the SIPP Panel 1992 and 1993 sample people who attrited as far in the past as five or six years ago and lived in a household with a total household income below 200% of the poverty threshold.

II.  Experimental Design

For timeliness, a revised SIPP 1993 Wave 9 paper questionnaire and control card was used instead of the SPD CAPI instrument for the SPD EAS interviews.  Three incentive amounts of $0, $50, and $100 were tested to determine the effect of the incentive sizes on the response rates.

Since the primary interest of the SPD is in the low income households, the SPD EAS was conducted based only on the noninterview SIPP  Panel 1992 and 1993 sample households with the total household income below 200% of the poverty threshold. The Panel Study of Income Dynamics (PSID) at the University of Michigan reportedly experienced a success rate of 50% using an incentive of $50 when they attempted to bring the attritors back into the sample. The success rate was defined as the ratio (or percent) of the interviewed sample households to the sample households assigned for an interview. The same definition of a success rate was used in the SPD EAS as well.  On the basis of the PSID 50% success rate, a sample size of 120 households for each of the three ($0, $50, and $100) incentive groups was selected.  For a response rate around 50% in at least one incentive group, a sample size of 120 per incentive group enables us to determine a minimum detectable difference (MDD) of 12.5% tested at 10% significance level with the design effect accounted for by a factor of 1.2 in the standard error estimate. Thus, the SPD EAS began with selecting, from SIPP Panel 1992 and 1993, 360 noninterview sample households with the total household income below 200% of the poverty threshold at the interview wave right before they attrited, in the manner described below.

Only Type A noninterview (e.g., refused, no one home) and Type D noninterview (e.g., moved - address unknown) sample households were selected into the SPD EAS sample. A sample household attrited prior to wave 5 in the SIPP was defined as an early attrited sample household, and a sample household attrited at wave 5 or a later wave was defined as a late attrited household.  Since the sample sizes of the SIPP Panels 1992 and 1993 are virtually the same, half (180) of the SPD EAS 360 sample households were selected from the SIPP Panel 1992 and the other half (180) were selected from the SIPP Panel 1993. The 180 SPD EAS sample households from each of the two SIPP panels generally consisted of 15 households selected from each of the 12 regional offices (ROs). Each of the 12 sets of 15 sample households consisted of  Sample Group A formed by either three or four households randomly selected (without replacement) from the Type A early attrited sample households, Sample Group B formed by either three or four  households randomly selected from Type A late attrited sample households, Sample Group C formed by either three or four households selected randomly selected from Type D early attrited sample households, and Sample Group D formed by either three or four households selected randomly from Type D late attrited households. Once the sample households were selected, the three level incentives ($0, $50, and $100) were assigned equally within each sample group (Sample Groups A, B, C and D).

The households that spawned from the SPD EAS sample households were also added to the original sample and were assigned the same incentive amount as the originating sample household.

III. Actual Sample

Based on the sample design for the SPD EAS specified in Section II, the size of the final sample assigned for fielding is 406 households (cases). Among the 406 sample households, 373 households were eligible for interview, and 33 households were not eligible for interview because they were Type B  (entire household institutionalized or temporarily ineligible) and Type C (entire household deceased or moving out of the survey universe, e.g., moved out of the country) noninterview households. A classification of 373 eligible sample households by combinations of the following SIPP attrition characteristics: Type A  noninterview, Type D noninterview, early attrition, and late attrition, and incentive levels ($0, $50, and $100) were made as exhibited in Table 1. The classification in Table 1 also served as a prototype framework for the analysis of the data from the SPD EAS sample in Section III.

IV.      Analysis of the Experimental Data

The analysis of the experimental data obtained from the SPD EAS actual sample described in Section III includes the calculation various response rates and their statistical significance of the eligible sample households.  As discussed in Section II, a response rate expressed in percent is defined as shown in the equation below.

Response Rate  =  (Number of Interviewed Eligible Assigned Households)(100)
                                                Number of Eligible Assigned Households

Five Sets (Sets 1 through 5) of the response rate analysis of the SPD EAS experimental data were conducted as described below.  Pairwise statistically significant difference tests of the response rates were conducted for the classifications within each set and across sets. The statistically significant difference tests were one-sided tests at 10% significance level, and a design effect of 1.2 was factored in the calculation of the standard errors used for the tests.

Set 1 - Among the assigned and eligible sample households with the total household income less than 200% of the poverty threshold (the entire sample), calculate the response rates among these households classified by incentive amounts ($0, $50, and $100), SIPP nonresponse types ( Type A, Hard Type A - direct refusal for interview, Soft Type A - not a direct refusal for interview, and Type D), and SIPP attrition stages (early attrition and late attrition) as shown in Table 1.  The results of the pairwise statistically significant difference tests of the response rates for the classifications within this set and across sets  were indicated in Table 1, namely, only the response rates of the pairs of classifications that are statistically significantly different will have one of the following footnote marks: *,  ^, and &.

Set 2 - Among the eligible sample households with the total household income less than 100% of the poverty threshold, calculate the response rates among these households classified by incentive amounts, nonresponse types, and attrition stages as shown in Table 3A.  In the same manner as Set 1, the results of the pairwise statistically significant difference tests of the response rates for the classifications within this set and across sets  were indicated in Table 3A.

Set 3 - Among the eligible sample households with the total household income at or more than 100% of the poverty threshold, calculate the response rates among these households classified by incentive amounts, nonresponse types, and attrition stages as shown in Table 3B.  In the same manner as Set 1, the results of the pairwise statistically significant difference tests of the response rates for the classifications within this set and across sets  were indicated in Tables 3B.

Set 4 - Among the eligible sample households with the total household income less than 150% of the poverty threshold, calculate the response rates among these households classified by incentive amounts, nonresponse types, and attrition stages as shown in Table 4A.  In the same manner as Set 1, the results of the pairwise statistically significant difference tests of the response rates for the classifications within this set and across sets  were indicated in Table 4A.

Set 5 - Among the eligible sample households with the total household income at or more than 150% of the poverty threshold, calculate the response rates among these households classified by incentive amounts, nonresponse types, and attrition stages as shown in Table 4B.  In the same manner as Set 1, the results of the pairwise statistically significant difference tests of the response rates for the classifications within this set and across sets  were indicated in Tables 4B.

In addition, a sample unit count tabulation of the association between the SIPP interview outcome types and the SPD EAS interview outcome types by incentive amounts was included in this study as shown in Table 2. The information in Table 2 was mainly used for determining the success rates of locating the attrited SIPP sample households in the SPD EAS as discussed in Section V.

V.   Results

Based on the analysis of the experimental data discussed in Section IV, the results of the response rate analysis were included and described in Tables 1, 2, 3A, 3B, 4A and 4B. Provided below are the highlights of the results of the response rate analysis.

    Highlights

    For the sample households (cases) with total household income below 200% of the poverty
    threshold ( the entire sample Tables 1 and 2):

•  The overall success rate for designated cases was 34%.  The success rates for $0, $50, and $100 incentive amounts were 27%, 33%, and 41%, respectively, with the $0 rate versus $100 rate being the only statistically significant difference.
• After removing the ineligible cases from the base, the response rates become 37% for overall, and 29%, 37%, and 44% for the respective incentive groups.  Again only the 15% difference between the $0 rate and the $100 rate was found to be statistically significant.
• Roughly 65% of the assigned cases were located.  Roughly 92% of cases assigned  were eligible for the survey.  Of those located and eligible, 60% were interviewed.  By incentive amounts, the response rates breaks down from $0 to $100 as 51%, 60%, and 66%, respectively, with only the 15% difference between the $0 and $100 groups being statistically significant.
• Due to the small sample sized involved, there were no significant nor substantive differences were detected between early SIPP attritors (before wave 5) versus late SIPP attritors (at wave 5 or a later wave).  Namely, the response rates without any footnote mark or marks (put beside them) in Table 1.
• Incentives seem to have a more substantive impact on cases that were SIPP Type A nonrespondents (refusals, no one home, and temporarily absent) than on cases that were SIPP Type Ds (unlocated movers).  In the $100 group, the response rate for the SIPP Type As was 53% versus 34% for SIPP Type Ds, and they were statistically significantly different.   Note that incentives could be offered only if the cases (sample households) were located.
• There were 145 ‘hard' refusals and 26 ‘soft' refusals.  Within the ‘hard' and ‘soft' Type As groups, incentives have no statistically significant effects on response rates at the10% level of error due to the small sample sizes.  However, comparing the response rates of ‘soft' versus ‘hard' Type As, the ‘soft' have a significantly better response rate in the overall and in the $100 incentive groups.
• The cost per case was approximately $1000 because of the inefficiencies of having small workloads across 12 regional offices.

• The sample size was 132 and the overall response rate was 39%. The response rates for $0, $50, and $100 incentive amounts were 34%, 42%, and 42%, respectively. All these response rates were not statistically significantly different.

• The sample size was 241 and the overall response rate was 35%. The response rates for $0, $50, and $100 incentive amounts were 26%, 33%, and 44%, respectively. Only the $0 response rate versus the $100 response rate were  statistically significantly different.

• The sample size was 191 and the overall response rate was 39%. The response rates for $0, $50, and $100 incentive amounts were 33%, 42%, and 42%, respectively. All these response rates were not statistically significantly different.

• The sample size was 182 and the overall response rate was 35%. The response rates for $0, $50, and $100 incentive amounts were 25%, 31%, and 46%, respectively. Only the $0 response rate versus the $100 response rates were  statistically significantly different.

We began the SPD EAS with a target overall rate of success of 50% in the $50 incentive group of low-income households.  The experimental results show that we did not reach our target using the $50 incentive.  The overall results for the $100 incentive group were more encouraging.

One problem we still have in the fight against attrition is in locating these cases after losing track of them for several years.  Approximately, a third of the assigned sample for the SPD EAS was not located at all.  Although this number is large, it appears to demonstrate that our tracking abilities have improved over time since more than half the designated sample for the SPD EAS were SIPP Type Ds or unlocated movers.

Finally, roughly 40% of the located and eligible sample households refused to cooperate once we got to their door. By incentive amounts, the Type A rates break down from $0 to $100 as 50%, 39%, and 34%, respectively, with only the 16% difference between the $0 and $100 incentive groups being statistically significant.

The average cost per case (sample household) was approximately $1000 which is higher than the cost of the actual full scale survey.  This reduction in efficiency is due to distributing small workloads across the 12 regional offices (i.e., losing the edge of the economy of large scale).

Table 1: Response Rate Analysis of the SPD Exploratory Attrition Study (EAS) for Households below 200% of Poverty Threshold
 

Categories of the Response Rate Analysis		With and Without Incentive			Without Incentive ($0 Incentive)			$50 Incentive			$100 Incentive
		Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent
Assigned Sample 1		406	138	33.99	128	35	27.34	133	44	32.60	144	59	40.97*
Eligible Assigned Sample 2		373	137	36.73	121	35	28.93	120	44	36.67	132	58	43.94*
SIPP Type As 2		172	72	41.86	51	15	29.41	55	22	40.00	66	35	53.03*
	Before Wave 5	79	35	44.30	23	8	34.78	24	11	45.83	32	16	50.00
	On or After Wave 5	93	37	39.78	28	7	25.00	31	11	35.48	34	19	55.88*
	‘Hard’ Type As	145	56	38.62	46	13	28.26	46	18	39.13	53	25	47.17
	‘Soft’ Type As	26	16	61.54	5	2	40.00	9	4	44.44	12	10	83.33
SIPP Type Ds 2		201	65	32.34	70	20	28.57	65	22	33.85	67	23	34.33
	Before Wave 5	110	37	33.64	36	11	30.56	35	11	31.43	39	15	38.46
	On or After Wave 5	91	28	30.77	33	9	27.27	30	11	36.67	28	8	28.57

* The $100 Group’s response rate is statistically significantly different from the $0 Group’s but not statistically significantly different from the $50 Group’s.

¹ All cases having EAS outcomes of Interviewed or Nonresponse Type A or B or C or D.

² All cases having EAS outcomes of Interviewed or Nonresponse Type A or D.
 

Table 2: SPD Exploratory Attrition Study (EAS) - SIPP Outcome Types by EAS Outcome Types by Incentive Amounts
 

	With and Without Incentive			Without Incentive ($0 Incentive)			$50 Incentive			$100 Incentive
	EAS Type A	EAS Type D	EAS Interview	EAS Type A	EAS Type D	EAS Interview	EAS Type A	EAS Type D	EAS Interview	EAS Type A	EAS Type D	EAS Interview
SIPP Type As	65	34	72	23	13	15	23	10	22	19	11	35
‘Hard’ Type As	63	26	__pard 56	22	11	13	22	6	18	19	9	25
‘Soft’ Type As	2	8	16	1	2	2	1	4	4	0	2	10
SIPP Type Ds	28	109	65	11	39	20	6	37	22	11	33	23

Table 3A: Response Rate Analysis of the SPD Exploratory Attrition Study (EAS) for Households below 100% of the Poverty Threshold
 

Categories of the Response Rate Analysis		With and Without Incentive			Without Incentive ($0 Incentive)			$50 Incentive			$100 Incentive
		Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent
Eligible Assigned Sample		132	52	39.39	44	15	34.09	45	19	42.22	43	18	41.86
SIPP Type As 1		47	22	46.81	18	6	33.33	14	6	42.86	15	10	66.67
	Before Wave 5	22	10	45.45	11	4	36.36	7	4	57.14	4	2	50.00
	On or After Wave 5	24	12	50.00	7	2	28.57	7	2	28.57	11	8	72.73
	‘Hard’ Type As	37	14	37.84	18	6	33.33	12	5	41.67	7	3	42.86
	‘Soft’ Type As	10	8	80.00	0	0	NA	2	1	50.00	8	7	87.50
SIPP Type Ds 1		85	30	35.29	26	9	34.62	31	13	41.94	28	8	28.57
	Before Wave 5	51	16	31.37	17	5	29.41	19	7	36.84	15	4	26.67
	On or After Wave 5	34	14	41.18	9	4	44.44	12	6	50.00	13	4	30.77

^{1 All cases having EAS outcomes of Interviewed or Nonresponse Type A or D}.

Table 3B: Response Rate Analysis of the SPD Exploratory Attrition Study (EAS) for Households at or above 100% of the Poverty Threshold
 

Categories of the Response Rate Analysis		With and Without Incentive			Without Incentive ($0 Incentive)			$50 Incentive			$100 Incentive
		TotalHHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent
Eligible Assigned Sample		241	85	35.27	76	20	26.32	75	25	33.33	90	40	44.44*
SIPP Type As 1		125	50	40.00	33	9	27.27	41	16	39.02	51	25	49.02
	Before Wave 5	57	25	43.86	12	4	33.33	17	7	41.18	28	14	50.00
	On or After Wave 5	68	25	36.76	21	5	23.81	24	9	37.50	23	11	47.83
	‘Hard’ Type As	109	42	38.53	28	7	25.00	34	13	38.24	47	22	46.81
	‘Soft’ Type As	16	8	50.00	5	2	40.00	7	3	42.86	4	3	75.00
SIPP Type Ds 1		117	35	29.91	43	11	25.58	34	9	26.47	39	15	38.46
	Before Wave 5	59	21	35.59	19	6	31.58	16	4	25.00	24	11	45.83
	On or After Wave 5	57	14	24.56	24	5	20.83	18	5	27.78	15	4	26.67

* The $100 Group’s response rate is statistically significantly different from the $0 Group’s but not statistically significantly different from the $50 Group’s.

¹ All cases having EAS outcomes of Interviewed or Nonresponse Type A or D.
 

Table 4A: Response Rate Analysis of the SPD Exploratory Attrition Study (EAS) for Households below 150% of the Poverty Threshold
 

Categories of the Response Rate Analysis		With and Without Incentive			Without Incentive ($0 Incentive)			$50 Incentive			$100 Incentive
		Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent
Eligible Assigned Sample		191	74	38.74	61	20	32.79&	65	27	41.54	65	27	41.54
SIPP Type As 1		77	36	46.75	24	7	29.17	24	12	50.00	29	17	58.62
	Before Wave 5	37	20	54.05	13	5	38.46	12	8	66.67	12	7	58.33
	On or After Wave 5	40	16	40.00	11	2	18.18	12	4	33.33	17	10	58.82*
	‘Hard’ Type As	61	26	42.62	22	7	31.82	20	10	50.00	19	9	47.37
	‘Soft’ Type As	16	10	62.50	2	0	0.00&	4	2	50.00^	10	8	80.00*
SIPP Type Ds 1		114	38	33.33	37	13	35.14	41	15	36.59	36	10	27.78
	Before Wave 5	66	20	30.30	23	7	30.43	26	8	30.77	17	5	29.41
	On or After Wave 5	48	18	37.50	14	6	42.86	15	7	46.67	19	5	26.32

* The $100 Group’s response rate is statistically significantly different from the $0 Group’s.

^ The $50 Group’s response rate is statistically significantly different from the $0 Group’s.

& The response rate for the $0 Group below 150% poverty is statistically significantly different from the $0 Group at or above 150% poverty.

¹ All cases having EAS outcomes of Interviewed or Nonresponse Type A or D.
 

Table 4B: Response Rate Analysis of the SPD Exploratory Attrition Study (EAS) for Households at or above 150% of the Poverty Threshold
 

Categories of the Response Rate Analysis		With and Without Incentive			Without Incentive ($0 Incentive)			$50 Incentive			$100 Incentive
		Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent	Total HHs	Response HHs	Response Rates in Percent
Eligible Assigned Sample		182	63	34.62	59	15	25.42&	55	17	30.91	68	31	45.59*
SIPP Type As 1		95	36	37.89	27	8	29.63	31	10	32.26	37	18	48.65
	Before Wave 5	42	15	35.71	10	3	30.00	12	3	25.00	20	9	45.00
	On or After Wave 5	53	21	39.62	17	5	29.41	19	7	36.84	17	9	52.94
	‘Hard’ Type As	85	30	35.29	24	6	25.00	26	8	30.77	35	16	45.71
	‘Soft’ Type As	10	6	60.00	3	2	66.67&	5	2	40.00^	2	2	100.00
SIPP Type Ds 1		87	27	31.03	32	7	21.88	24	7	29.17	31	13	41.94
	Before Wave 5	44	17	38.64	13	4	30.77	9	3	33.33	22	10	45.45
	On or After Wave 5	43	10	23.26	19	3	15.79	15	4	26.67	9	3	33.33

* The $100 Group’s response rate is statistically significantly different from the $0 Group’s but not statistically significantly different from the $50 Group’s.

^ The $100 Group’s response rate is statistically significantly different from the $50 Group’s.

&The response rate for the $0 Group below 150% poverty is statistically significantly different from the $0 Group at or above 150% poverty^.

¹ All cases having EAS outcomes of Interviewed or Nonresponse Type A or D.

SPD Working Papers

SPD Publications

Contact: (dsd.survey.program.dyanmics@census.gov)
URL: http://www.spd.census.gov/SPD/

| Introduction to SPD |  Survey Design & Content |  Data Editing |  Finding SPD Info |  Sampling & Weighting | 
| Linking Files | Publications |  S&A  |  News&Notes |  Users' Guide |  Search  | 

Census 2000  |  Subjects A to Z  |  Search  |  Product Catalog  |  Data Access Tools  |  FOIA  |  Privacy · Policies  |  Contact Us  |  Home