Tracking Devolution: The Survey of Program Dynamics
Robert A. Kominski
Daniel H. Weinberg
Paper presented at the
March 29, 1997
The views and conclusions expressd in this
Tracking Devolution: The Survey of Program Dynamics
Robert A. Kominski
The Census Bureau is currently in the field collecting data for a new study, the Survey of Program Dynamics. The project has been brought to life as a result of the passage of welfare reform legislation developed during the 1996 Congress. The goal of the survey is to track some of the long-lasting effects of the decentralization of the welfare system in the United States, and the social, economic and demographic impacts these changes will have on families, adults and children over time. The initial impetus for the survey did not arise from the welfare reform act, but in a desire to develop a vehicle for monitoring long-term changes in families. Preparatory work done during the past few years laid the groundwork for the Bureau to capitalize on the opportunity created by the passage of the welfare reform bill. In this paper, we discuss some of the background leading up to the development of the survey, the basic design as currently formulated, the main components of the survey design, some of the technical issues we have had to resolve thus far, and our current status.
The beginning of the Survey of Program Dynamics was located in a small group of Census Bureau researchers who felt that there was a need for a long-term survey vehicle to monitor social, demographic and economic change in the U.S. population. During the early 1990's the Census Bureau was actively engaged in the redesign of the Survey of Income and Program Participation (SIPP), the agency's primary vehicle for measuring longitudinal economic conditions in U.S. households. Among the issues being seriously discussed at that time was the appropriate length of the new longitudinal panel. Panels conducted since 1984 (the start of the SIPP program) had lasted, for the most part, about 32 months each, with data collected for each month of observation, and a new panel gegun at the start of each calendar year. Some researchers (both within and outside the Bureau) had argued that the 32-month period was not fully effective in monitoring the kinds of phenomena the survey was supposed to focus on, for example, job activities, spells of unemployment and program participation. The National Academy of Sciences, in its review of the SIPP program, had urged the Bureau to consider lengthening the longitudinal observation period. After much deliberation and research, the Bureau ultimately decided on a new non-overlapping panel design of 4 years observation, with the first new panel slated to begin in 1996.
Despite this decision, there was still a feeling among some researchers that even a 4-year period of observation would not be highly useful for studying many of the social and demographic changes associated with changing economic conditions households experience over time. In early 1993 a group of Bureau scientists began discussions about a possible ‘extended' SIPP panel which would follow respondents from a conventional SIPP panel for some additional amount of time. The result of the work of these individuals and their discussions with colleagues at other agencies, especially Health and Human Services, was that there was interest in the development of an ‘extended SIPP panel'. Interest was particularly strong regarding the long-term effects of economic conditions and changes on the children in these households. During 1993 and 1994, several federal agencies, Health and Human Services, and Food and Consumer Services in the Department of Agriculture, began to reinforce their interest with the appropriation of small amounts of money to the Census Bureau to try to develop a new long-term survey instrument. Other sources of interest, focused through NICHD's Family & Child Well-Being Research Network, a national collaboration of academic and other researchers working on children's issues, began providing background material for a wide array of issues that might be addressed in such a survey.
Throughout the development and planning process, a number of substantive topics emerged as important for the data collection activity:
o The survey should measure program eligibility
and participation for the full range of welfare programs.
By the fall of 1994, discussions had proceeded far enough that the rough goals of the Survey were becoming clear. Essentially, the design would focus on two primary elements:
1) Providing information on actual and potential program participants
over the period
2) Examining the consequences of program participation on the well-being
of recipients ,
At the beginning of 1995, the working team at the Bureau had assembled a large amount of questionnaire material for consideration as the content of the SPD. One large portion of this was essentially derivative of the content and topic coverage of the SIPP - since the SPD panel was to be an extension of a retired SIPP panel, this made obvious sense. The second assortment of materials were a series of questions submitted by various experts on children's research issues. The primary task before the team now was to try to bring this varied collection of materials into a single coherent document that met the survey goals. Throughout the early months of 1995, many decisions were made about content scope and design, working toward the tentative goal of a pretest in the spring of 1996 and implementation in the spring of 1997. During this time, it also became apparent to the steering committee for the SPD that the full proposed scope of question content for the survey could not be realistically administered. For this reason, the survey was broken into two parts. The ‘track 1' portion would be the annual portion of about an hour that would focus on issues such as employment, financial conditions and program activities, with some questions on children's statuses. The ‘track 2' questionnaire would be administered annually also, but would begin one year after ‘track 1' (because of needed additional development time). This part of the study would focus specifically on children, using more detailed psychological and other assessment measures. A meeting held with a group of external reviewers in mid-1995 confirmed the decision to separate and refine the two distinct questionnaires.
Throughout 1995, development work proceeded somewhat cautiously, because the study still had no real sponsor. During 1994 it had become apparent that possible welfare reform legislation might be the vehicle which would create the funding opportunity for the SPD, but with no clear legislation in sight, it was possible only to prepare contingency plans for conducting the survey. In 1995 it seemed possible that the Congress might develop some welfare reform legislation that the President would sign. We continued to use the small resources available to move forward and prepare. However, by late 1995, with no bill passed, and no additional source of funding available, the study was reduced to finalization of the track 1 CAPI instrument specifications and some cognitive pretesting of the adolescent self-administered questionnaire for track 1. The SPD began a ‘hiatus' that would last until mid-1996, when we received word that a new welfare reform bill had been introduced, one which had greater chances of becoming law than the previous incarnation. We began focussing on the problems of operationally taking on the survey, since the likelihood of funding had become much more elevated. In the fall of 1996 we received notification that the President had signed legislation passed by the Congress, and the Personal responsibility and Work Opportunity Reconciliation Act of 1996 was enacted as Public Law 104-193. This bill directs the Census Bureau to:
"continue to collect data
on the 1992 and 1993 panels of the Survey of Income and Program Participation
With passage of the public law, ten million dollars were annually appropriated to the Census Bureau for each of the fiscal years 1996-2001, and the Survey of Program Dynamics moved from being an idea to an authorized and funded program.
The basic design of the SPD uses the retired 1992 and 1993 SIPP longitudinal panels as the base sample. All households which were in the initial (wave 1) interview of these surveys, and which were still in sample at the panel's conclusion, are eligible households for inclusion in the first interview of the SPD. The use of both retired samples (approximately 20,000 households each) gives us a large initial sample for the SPD. Given likely difficulties in locating many households, and possible nonresponse from some households who felt their initial 2 ½ years in the SIPP are all they are willing to give, this very large sample should provide us a good resultant sample with which to continue in subsequent years. This should also allow us latitude to conduct optimized subsampling in the later years, particularly since funding levels will likely not allow us to continue with a full 40,000 household design,
The plan to continue data collection for both the 1992 and 1993 SIPP panels provides extensive baseline (background) information from which to determine the effects of welfare reform. Regarding program data, the 1992 and 1993 panels collected more detailed data than any other national survey with regard to program eligibility, access and participation, transfer income, and in-kind benefits. For economic and demographic data, the 1992 and 1993 panels collected very detailed data on employment and job transitions, income, and family composition.
From the very start of the SIPP, the Census Bureau also has worked closely with policy agencies to develop and field topical modules that enhance the value of these SIPP core data. Modules fielded in both the 1992 and 1993 panels of special interest here include those on (1) education and training histories, (2) marital, fertility, and migration histories, (3) migration history, allowing the identification of immigrants and citizenship status, (4) detailed family relationships within the home, (5) work schedules, child care, child support activities, and financial support for non-household members, (6) medical expenses and utilization of health care services, (7) work disability histories, (8) extended measures of child well-being, and, (9) measures of social support networks. Thus, the data collected as part of the original SIPP samples provides an incredibly rich soil in which to plant the seed of a new survey designed to track welfare reform as it takes effect. By interviewing the same households in the SPD, analysts will have data for the baseline pre-reform period, the reform implementation period, and for the medium-term post-reform period. All of these data are required to assess short-term and medium-term consequences and outcomes for families and individuals.
Current plans are for data to be collected for each of the seven years for 1996 through 2001, providing an ultimate panel of data for nine years (1992-2000) when combined with the 1992 SIPP panel, and eight years (1993-2000) when combined with the 1993 SIPP panel. Due to the delay in the start of the SPD, we have had to develop a mechanism for filling in the space between the end of SIPP observation, and the start of routine SPD observation. Thus, as currently configured, the survey has three fundamental pieces; (1) the ‘bridge' survey to be fielded in the spring of 1997 (April-June) only; and providing the link between SIPP and SPD; (2) the ‘track 1' survey, to be administered yearly beginning 1998 (for reference year 1997) and, (3) the ‘track 2' survey to be administered yearly starting 1999.
Creation of the ‘bridge' survey was forced by the hiatus of the program during 1996, when funding did not emerge, and virtually all planning and development work had to stop, and necessitated by the need to locate the retired SIPP panel households and collect information for them before too much time had passed. When it became clear in late 1995 that we would not have funds to continue the project, most work on the project, including CAPI development of the instrument, stopped. By mid-1996, when it appeared that passage of the welfare reform bill was more likely, we had to find a way to bring the project back on line quickly. The SIPP panels of 1992 and 1993 had ended their survey lives in late 1994 and 1995, respectively. We would not have enough development time to fully author and test the instrument to get it in the field for early 1997; however, in order to provide data continuity with the expired panel, we needed a mechanism to get data for 1996. We were also concerned that the longer we waited to reestablish contact with households, the more problems we would have with movers and tracking them down (about 17% of all persons in the U.S. move each year). This could be particularly a problem in SPD, since SIPP rules follow all members of a household, even if different members go different places. Consequently, it became important that we get some instrument into the field sometime during 1997.
The solution we have implemented is to use a modified version of the
annual March Current Population Survey demographic supplement to collect
information for 1996. By making only a relatively small set of changes
to the CPS instrument (already in CAPI form), we are able to bring the
SPD online in the field this spring. (The major change is the addition
of a few questions to get some summary calendar-year data for 1995 for
the 1992 SIPP panel- who were last interviewed in January 1995.)
While the use of the CPS supplement has not been effortless, it has required
far fewer resources than attempting to field a brand-new survey would
have needed. Use of the CPS supplement should mean less intensive
content training for the field staff, and allow them to focus on the more
critical problem (at this point) of tracking the respondents down.
We anticipate fielding the bridge survey in the period of April-June of
this year. The Office of Management and Budget has approved
an incentive payment of $20 for some households, as part of a test of
incentive payments to reduce nonresponse.
While the basic paper questionnaire for track 1 was developed before the hiatus period, it has undergone relatively little work since that time. We are using 1997 to conmplete CAPI development (authoring is being done at the University of California-Berekely, developers of the CASES language the Bureau uses) and to field an extensive field pretest of the instrument in September 1997. From this, we expect to have a good idea of the how well the instrument will perform in a real operations context, and what kind of problems need to be worked our before routine field implementation for the period of April-June 1998.
The second component of the survey -- ‘track 2' -- will be brought on-line during the 1999 interviews, and will be a part of the survey for the years 1999-2001. In this component we hope to address more child-oriented topics, with particular focus on elements of child assessment, quite likely using clincially-tested assessment scale devices. These may include tests (such as math or reading ability), or evaluative scales (such as psychological adjustment scales). Given the short duration of time (3 years), it is unlikely that we will be able to administer many of the measures more than one time; however, we are considering the possibility of having some recurring items in this part of the survey.
A variety of technical issues have occupied part of our time in preparing for SPD, and are likely to be of continuing interest to the project.
One of the first important tasks we will be faced with will be bringing the households of the 1992 and 1993 SIPP panels back into sample for SPD. We are assembling a set of procedures to gather as much information as possible to assist in this activity. We will begin, of course, with the final addresses used at the close of the 1992 and 1995 panels. Given our knowldge of mover rates from other surveys, we believe that at least 20% of the 1993 panel and at least 30% of the 1992 panel, will have moved one or more times. It is this last group which, of course, will be most troublesome. Generally, short-term multiple movers tend to be persons with less stable household situations and living conditions - in short, the kind of people we most want in sample for the SPD are also those to likely be most difficult to locate.
Just because individuals move, however, does not mean we will not be able to track and find them. SIPP currently uses tracking procedures on the shorter 4-month interview-to-interview cycle, and manages to keep sample loss relatively low. Many of the field representatives working on the SPD will be persons who currently work on SIPP. In some cases, we expect them also to be the same people who worked on the 1992 and 1993 panels. While the interview period for the 1997 bridge survey will officially be the period from April through June, it is quite likley that we will extend observation beyond this period in order to locate as many of the households as possible. While we have to expect that we will lose some households, we are optimistic that we will find and interview a sizable proportion of them.
As noted earlier, it is clear that we will not be able to interview all
households in the 1992 and 1993 SIPP panels beyond the 1997 interview.
How much we will have to subsample depends on the ultimate response rate
to the 1997 SPD ‘bridge' survey, and what focus we wish to make on specific
subpopulations. Further, no decisions have yet been made on which
groups should be overrepresented, but we expect to use the welfare reform
law as a guide. For example, our initial thoughts focus on keeping
all low-income households (e.g. households with incomes 150 percent of
poverty or less) and households with children. Since SIPP uses a clustered
interview design, we would probbaly also maintain the sample clusters
that these designated households are in. If, after this strategy,
sample sizes of high interest populations are too small, it has been suggested
that we consider adding "retired" March 1997 CPS households (most of whom
also have a March 1996 interview) to the SPD sample. We will need
to see the results of our 1997 ‘bridge' efforts before moving onto a final
sample designation for 1998 and beyond.
In this area we have also not made final decisions. Our current
thinking involves weighting the interviewed population to represent the
April 1997 U.S. population, rather than the 1992 or 1993 population (basis
for the SIPP weights). We are using as the initial SPD sample all
households that were in the first interview of their respective panels
(100-level households, in SIPP parlance) and who were also in sample at
the conclusion of the survey. This is done to insure a full set
of ‘baseline' information for the SPD households. We will interview
all persons in those households, including those who entered after the
first SIPP interview. However, if a non-original member of a SIPP
household leaves an SPD household, we will not follow them, because the
base weights for longitudinal purposes will need to be derived from the
original SIPP panel weights.
A variety of data products will ultimately come to fruition from the SPD effort. We recognize that users of the longitudinal data will have a hard time figuring out how to use data from three separate surveys (SIPP, CPS, and SPD) simultaneously in a longitudinal analysis. The microdata from the SIPP surveys are already available on-line at the Census Bureau's web site (www.census.gov) through the "Surveys On-Call" program; plans are underway to provide the data as well through FERRET (the Census Bureau's Federal Electronic Research and Retrieval system, which already provides the CPS microdata to users). As soon as the bridge survey data are processed, they will be made available in a CPS-compatible format on FERRET. The next challenge is to create a longitudinal data set with annual data from the SIPP and the CPS in a format consistent with the way data will be provided from the 1998 SPD, so that users can develop familiarity with the data and be ready for the first wave of SPD.
Beyond the basic longitudinal file that will grow over over time, the
various topical module supplements of the orignal SIPP panels will provide
added substantive detail. In addition, we are attempting to include
some some items to be used in the 1996 SIPP panel in the SPD as well.
This would provide a possible ‘enhanced' cross-sectional sample for some
topics using the combined SIPP-96/SPD data files. Another possibility
that has been discussed is the possible creation of a combined bridge-March
1997 CPS file. Since therse two surveys will have highly overlapping
content, this file could provide a very large national household sample
(approx 100,000 households) for detailed one-time cross-sectional analyses.
Like many projects, the current SPD is derived from an idea somewhat different from where it now stands poised. The dramatic changes posited for the social welfare system deem it necessary that tools be on hand to attempt to measure the possible large scale social, economic and demographic changes that could follow in the near future. The merger of the orignal vision of the SPD with the practical data needs and collection realities (especially timeliness and cost) we now face have produced a program that we believe will provide useful data and information for many years to come.