THE SURVEY OF PROGRAM DYNAMICS?
Daniel H. Weinberg, Patricia A. Johnson, and the SPD Team
Prepared for the Census Advisory Committee of Professional Associations
P.L.104-193, the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) eliminated the Aid to Families with Dependent Children program. Part of that law directed the Census Bureau to "continue to collect data on the 1992 and 1993 panels of the Survey of Income and Program Participation [SIPP] as necessary to obtain such information as will enable interested persons to evaluate the impact of the amendments made by Title I of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 on a random national sample of recipients of assistance under State programs funded under this part and (as appropriate) other low income families, and in doing so, shall pay particular attention to the issues of out-of-wedlock birth, welfare dependency, the beginning and end of welfare spells, and shall obtain information about the status of children participating in such panels."
To carry out that directive, we are planning a Survey of Program Dynamics (SPD) to simultaneously measure important features of (1) the full range of welfare programs, including both programs that are being reformed and those that are unchanged, and (2) the full range of other important social, economic, demographic and family changes that will facilitate or limit the effectiveness of the reforms. Current plans are for data to be collected for each of the six years from 1996 through 2001, providing panel data for ten years (1992-2001) when combined with the 1992 SIPP data.
The survey has been designed with three fundamental sections -- (1) the "bridge" survey which will provide the link between the 1992 and 1993 panels of the SIPP and SPD (using the March 1997 Current Population Survey instrument); (2) the 1998 SPD which will collect annual retrospective data starting in 1998; and (3) the 1999 SPD Child Well-Being Module to be administered starting in 1999, though its content may vary from year to year.
There are three issues on which we would like advice from the Committee -- criteria for subsampling, approaches to weighting, and the character of the microdata files to be produced from the program.
WHAT ARE THE ISSUES IN IMPLEMENTING
On August 22, 1996 the President signed legislation passed by Congress and the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) was enacted as P.L.104-193. This is a comprehensive piece of legislation with far-reaching implications for many programs. The law eliminates the open-ended federal entitlement program of Aid to Families with Dependent Children (AFDC) and creates a new program called Temporary Assistance for Needy Families (TANF), which provides block grants for states to offer limited cash assistance. The legislation also makes far-reaching changes to child care, the Food Stamp Program, Supplemental Security Income (SSI) for children, benefits for legal immigrants, and the Child Support Enforcement program. It also includes modifications to child's nutrition programs and a reduction in the Social Services Block Grant and retains child welfare and child protection programs.
For particular agencies, many specific program evaluation needs can be served by a series of focused single-purpose surveys or experiments. But if the research community were to rely solely on highly focused data collection, there will inevitably be major gaps. While single-purpose approaches to data collection are useful, an omnibus data collection vehicle can provide the basis for an overall evaluation of how well welfare reforms are achieving the aims of the Administration and the Congress, and meeting the needs of the American people. This requires a survey that casts a wide net, one that simultaneously measures important features of (1) the full range of welfare programs, including both programs that are being reformed and those that are unchanged, and (2) the full range of other important social, economic, demographic and family changes that will facilitate or limit the effectiveness of the reforms. Further, such a survey should be in place before enactment of the reforms to allow adequate assessment of baseline circumstances.
Several years before the passage of the actual legislation, the Department of Health and Human Services (Office of the Assistant Secretary for Planning and Evaluation--ASPE) and the Department of Agriculture (Food and Consumer Services--FCS) invested substantial resources in the initial development of such a survey by the Census Bureau in the hope that they could fund and field such a survey to meet their needs to understand the effects of public policy changes on the population. In these planning activities (including consultations with the National Institute for Child Health and Human Development's Family and Child Well-being Research Network on children's issues) several design features have emerged as essential:
continue to collect data on the 1992 and 1993 panels of the Survey of Income and Program Participation [SIPP] as
necessary to obtain such information as will enable interested persons to evaluate the impact of the amendments made by
Title I of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 on a random national sample of
recipients of assistance under State programs funded under this part and (as appropriate) other low income families, and
in doing so, shall pay particular attention to the issues of out-of-wedlock birth, welfare dependency, the beginning and
end of welfare spells, and shall obtain information about the status of children participating in such panels.
To implement this directive, the Bureau established a team to carry out this survey effort, which we call the Survey of Program Dynamics (SPD).
As directed by the legislation, data already collected in the 1992 and 1993 SIPP panels will provide extensive baseline (background) information from which to determine the effects of welfare reform. SIPP is a longitudinal survey of households, each of which was interviewed at least nine times at four-month intervals, and followed if they moved. The SIPP collects more detailed data than any other national survey regarding program eligibility, access and participation, transfer income, and in-kind benefits. Regarding economic and demographic data, the 1992 and 1993 panels collected very detailed data on employment and job transitions, income, and family composition. By interviewing the same households in the SPD, analysts would then have data for the baseline pre-reform period, the reform implementation period, and for the medium-term post-reform period. These data are required to assess short-term and medium-term consequences and outcomes for families and individuals. The use of both panels will also double the size of certain groups of interest, subject of course to our ability to recontact households in the two panels and their willingness to participate. Because the funding provided is not sufficient to interview all households in both panels past 1997, we will have to subsample after 1997.
The Census Bureau also has worked closely with policy agencies to develop and field topical modules that enhance the value of the basic SIPP data. Modules of special interest here include those on (1) education and training, (2) marital, fertility, and migration histories, (3) family relationships within the home, (4) work schedules, child care, child support, support for non-household members, (5) medical expenses and utilization of health care services, and (6) child well-being.
SPD data collection will take place simultaneously with the fielding of the 1996 SIPP panel (begun in April 1996), providing opportunities for placement of a few of the same questions on the SIPP as are on the SPD, tremendously enhancing the sample size for cross-sectional analysis.
II. CURRENT STATUS AND PLANS
Current plans are for data to be collected for each of the six years from 1996 through 2001, providing panel data for ten years (1992-2001) when combined with the 1992 SIPP data. Our original plans were to have an instrument ready to field concurrent with welfare reform. Because the legislation was vetoed twice during 1995, plans were put on hold. Consequently, we were unable to pretest the SPD questionnaire and could not field the survey we had designed in 1997. Nevertheless, we felt it critical to fill the data gap between the end of the SIPP observations and the start of the basic SPD observations. Consequently, the survey has been designed with three fundamental sections -- (1) the "bridge" survey which will provide the link between the 1992 and 1993 panels of the SIPP and SPD; (2) the 1998 SPD which will use the core instrument already developed to collect annual retrospective data starting in 1998; and (3) the 1999 SPD Child Well-Bing Module to be administered starting in 1999, though its content may vary from year to year.
SPD " BRIDGE" SURVEY
It is very important to collect income and program participation data in spring 1997 for calendar year 1996 from as many of the 1992 and 1993 SIPP households as we can find. Data for 1996 will be collected in April-June 1997 by administering a modified version of the annual March 1997 Current Population Survey (CPS) demographic supplement, with a few new questions designed to collect summary 1995 data for the 1992 SIPP panel (who were last interviewed in January 1995). Eligible to be included in the SPD sample will be all SIPP "100-level" persons, that is, all those interviewed in the first wave of the 1992 and 1993 SIPP panels (and still being interviewed at the end of the panel). (We are also considering developing a procedure to find "100-level" persons who attrited before the end of the SIPP.)
Finding people who move is critical to the success of any longitudinal survey, particularly one as focused on the low-income population as SPD. Naturally, SIPP has developed many time-tested procedures that will help in that regard. But the automated questionnaire instrument itself must permit such tracking as well. Luckily, the CPS implemented a "mover module" in January 1997 to track people leaving formerly interviewed CPS households. This is crucial to the SPD especially given the time that has elapsed since the last interview. We are calling this survey the SPD "Bridge" Survey. We will also be testing the use of a $20 monetary incentive for low-income households in an attempt to reduce nonresponse to the Bridge survey; the Census Bureau demonstrated that such an incentive was successful in reducing nonresponse to wave 1 of the 1996 SIPP panel.
1997 PRETEST OF THE 1998 SPD
We are using 1997 to complete work on the 1998 SPD questionnaire. The University of California at Berkeley is authoring this instrument; UC-Berkeley is the developer of the CASES authoring language that is used for most computer-assisted Census Bureau surveys. We plan a pretest for September using about 500 retired 1996 CPS households in four of our regional office locations. From this test we will have a good idea of how well the instrument does in an operational environment. We also will test the use of a Self-Administered Adolescent Questionnaire using audio-cassettes to obtain information from youths 12 to 17 years old.
Using the fully developed CAPI instrument, annual data collection will occur once each year in March-May, with annual recall for the preceding calendar year. There will be a set of retrospective questions for all persons aged 15 and older that focuses on topics such as jobs, income, and program participation. Also included will be a section focusing on children in the household that addresses topics such as school status, activities at home, child care, health care, and child support. Average interview length would be about one hour per household. The adolescent survey will probably include issues such as family conflict, vocational goals, educational aspirations, and crime-related violence (this questionnaire is still being developed).
1999 SPD and later
Work has begun on identifying the topics for the 1999 child supplement.
We plan to include more child-oriented topics with particular foci on elements
of child assessment using clinically tested assessment scales.
III TECHNICAL ISSUES
Subsampling. It is clear we cannot interview all households in the 1992 and 1993 SIPP panels in 1998. How much we will subsample depends on the response rate to the 1997 SPD "Bridge" survey. Further, we have yet not decided which groups should be over-represented, but we expect to use the welfare reform law as a guide. For example, our initial thoughts focus on keeping all low-income households (e.g., households with incomes 150 percent of poverty or less) and households with children. If sample sizes of high-interest populations are too small, we have suggested that we consider adding "retired" March 1997 CPS households (most of whom also have a March 1996 interview) to the SPD sample. We will need to see the results of the 1997 bridge survey before deciding the final sampling strategy.
Weighting. We have not decided this issue yet either. Current thinking involves weighting the interviewed population to represent the April 1997 U.S. population and releasing a cross-section file plus developing a longitudinal file weighted to represent the 1993 population (the basis for the SIPP weights). Differential attrition of the low-income population is a serious concern in developing appropriate weights. The Appendix presents one proposal for weighting.
Database and product development. Users of the longitudinal data will have a hard time figuring out how to use data from three separate surveys (SIPP, CPS, and SPD) simultaneously in a longitudinal analysis. The microdata from the SIPP surveys are already available on-line at the Census Bureau's web site (www.census.gov) through the "Surveys On-Call" program; plans are underway to provide the data as well through FERRET (the Census Bureau's Federal Electronic Research and Review Extraction Tool, which already provides the CPS micro data to users). FERRET is the demographic component of the Demographic Access and Dissemination System. When the bridge survey data are processed, we will make them available in a CPS-compatible format on FERRET. The subsequent challenge is to create a longitudinal data set with annual data from the SIPP and the CPS in a format consistent with the way data will be provided from the 1998 SPD, so that users can develop familiarity with the data and be ready for the first wave of SPD.
Supplementary data. We have contracted with the University of Wisconsin to create a complementary data base of state and county welfare program characteristics that we could then match to the SPD data. (County-level matches would have to remain confidential and researchers would have to work on that matched data set at the Census Bureau to maintain respondents' confidentiality.) We may also be able to match other administrative data provided to us electronically (e.g., tax returns, welfare program records) to the survey data; again, these would need to be accessed only at Census Bureau secure locations.
1. What criteria should we use for subsample selection for the 1998 survey to maximize the usefulness of this survey for welfare reform evaluation yet not complicate selection and weighting too much? Examples of criteria that might be used include income, presence of children, past welfare program participation.
2. Are there any particular issues about weighting of which we need to be cognizant? How would you suggest we best deal with differential attrition of the low-income population?
3. Can you give us any guidance on what researchers will need in the way of microdata and when? First, we assume that sophisticated users can manipulate all three source files--is that a reasonable assumption? Second, while it is relatively easy for us to create a cross-section file of just the 1997 Bridge data, we wonder if such a file would be of use. We have had substantial attrition in the 1992 and 1993 SIPP files since they were originally fielded. Further, we know that the attrition is unevenly concentrated in the low-income portion of the universe. Poverty measures from such a file will be far lower than that produced by the March CPS even though the questions asked will be very similar. Data users may be tempted to compare these two measures and that would definitely not be appropriate. We do not plan cross-section files for 1998 and beyond. Would we be better advised not to produce such a file in 1997 either?
DATA PRODUCTS AND WEIGHTING PROPOSAL FOR THE SPD
There are three objectives that influenced development of the following proposal for SPD data products and weighting:
o Provide longitudinal data
to evaluate the effects of welfare reform.
Described below is a one-time data product that the Census Bureau will
release after the 1997 Bridge data is collected and
1997 SPD Bridge File
The Census Bureau will release a public use file that will include data from the 1992 and 1993 SIPP longitudinal public use files, the 1997 Bridge data, and codes to link the SIPP and Bridge data. It will be released as a research file with appropriate caveats. It will include the longitudinal and calendar year weights from the SIPP 1992 and 1993 data files. These weights will allow users to do panel, calendar year and multiple panel analyses. Adjustment factors to post-stratify up to a January 1992 or January 1993 cohort will be provided either on the file or as part of the documentation. The adjustments will be crude, but should suffice for preliminary research purposes. Researchers have the option to make additional weighting/imputation adjustments as they deem necessary for their specific analyses.
1998+ SPD Public Use Files
The Census Bureau will release two files each year after SPD data has been collected, processed, and weighted. The first will include the SIPP 1992 and 1993 panel data, the 1997 bridge data and the 1998+ SPD data as originally collected. The second will attempt to create a consistent set of annual measures for each year of data (a "common format" file), to simplify the analyst's chores. These are basically the same type of longitudinal product and weighting we have been using for the SIPP since its inception.
A longitudinal panel cohort and all calendar year cohorts will be identified and a weight provided for analyses. In general, a longitudinal cohort is defined as all the people interviewed in every interview within the time period of interest. For example, the panel cohort for the 1993-1998 SPD cohort will be all those people interviewed from January 1993 through the 1998 SPD interview (April-June 1998). Information from other interviewed people in any one month will be included in the family/household totals on the data file.
The control date will be as of the beginning of the cohort period (January 1993 for the example cited above as illustration). This creates a nationally representative longitudinal cohort for people interviewed at the beginning of the cohort period. We will also investigate and apply weighting adjustments to help compensate for differential nonresponse.