Measuring Welfare Reform, Preliminary Results
Arthur F. Jones Jr.
February 2, 2000
The role of social science lies not in the formation of social policy, but in the measurement of its results.
Senator Daniel Patrick Moynihan, 1969
*This paper reports the results of research and
analysis undertaken by Census Bureau staff. It has undergone a more limited
review than official Census Bureau publications. This report is released
to inform interested parties of research and encourage discussion.
MEASURING WELFARE REFORM, PRELIMINARY RESULTS
Arthur F. Jones Jr., Barbara
Downs, Stephanie Shipp, Census Bureau
Background: The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) is the most significant piece of welfare legislation since Aid to Families with Dependent Children (AFDC) established in 1935 as part of Franklin Roosevelt's New Deal. One of the main features of the new welfare reform law is the end of the federal government's 61-year old entitlement program, AFDC, which is replaced by a block grant program to states called Temporary Assistance for Needy Families (TANF). (1) PRWORA also establishes time limits for benefits and requires states, not the federal government, to be responsible for managing and administering public assistance payments. In addition, it requires welfare recipients to participate in community service programs or skills enhancement activities while receiving benefits and allows states to provide additional services to help break the cycle of welfare dependency.
How does PRWORA affect state planning? (2)
Shifting the burden from the federal government to state governments may be the most significant feature of welfare reform. In accordance with PRWORA, each state must design and implement a plan of action to be eligible for federal funds under TANF. Under the new block grant system, eligible states receive a fixed dollar amount of funds for welfare programs. The federal government provides a budget of $16.5 billion annually for TANF through Fiscal Year 2002. The amount of each state's block grant is based on the amounts received by the state in 1994 and 1995, two years that represent historically high levels of welfare spending. For states to qualify for maximum federal funds under TANF, they must maintain a significant portion of their historical financial commitment to public assistance outlays. These maintenance of effort (MOE) funds are, much like TANF, fixed over time and do not vary with caseload or services provided by the state.
Given that federal funds are allocated based on historically high levels of need, many states are eligible for more funds under TANF than under previous welfare provisions. In fact, many states are currently experiencing budgetary surpluses of public assistance funds. As companies in most regions of the country continue to create jobs at impressive rates, driving unemployment rates to near 30-year lows, states are left additional budgetary resources. These resources allow states to provide additional support services for the needy as well as setting up contingency funds in case of an economic downturn. According to a recent the General Accounting Office report, (3) many states have allocated additional expenditures on support services geared toward self-sufficiency rather than increasing benefit levels.
The decision of whether to expand services through additional spending or to place unused money into a contingency fund is an important one, which also highlights the significance of state fiscal planning. Recent research suggests that public assistance expenditures are very sensitive to changes in business conditions. McGranahan (4) found that a 1-percent rise in the unemployment rate increases state assistance expenditures by $52 per capita. This is not an immediate concern: the nation's economy is performing well, but its ability to maintain such a high level of performance is never a sure thing. Even increasing the amount of job and skills training to welfare recipients will not stem the tide of increasing welfare recipient cases during an economic downturn. Former welfare recipients are likely to be the first released as companies trim payrolls-- "last hired, first fired." The longer the economic expansion lasts, however, the longer welfare recipients may hold their jobs, thus reducing the likelihood that they will be trimmed from the payrolls during slack market conditions. However, this group generally has high job turnover which increases their likelihood of being laid off during slow periods.
Under the PRWORA, states have two primary options for dealing with the likelihood of increasing caseloads during an economic downturn. First, the federal government may make available to states a Contingency Fund that provides limited matching funds. This requires that states increase outlays to become eligible for additional federal resources. The other option available to states is the Loan Fund. This fund allows states meeting certain economic hardship criteria to borrow federal funds to handle increasing caseloads. Although these funds are available, the likelihood that a state would qualify, let alone use the funds, is small. Qualifying states must have a jobless rate in excess of 6.5 percent and a corresponding 10 percent increase in the unemployment rate during one of the last two years. States may also qualify if they experience a 3-month increase in their food stamp caseload of 10 percent in excess of the corresponding 1994 or 1995 fiscal year. (5) States must also meet higher levels of spending to qualify for these funds.
How does PRWORA affect families?
While the short-term effect on PRWORA may be greatest for state policy makers, there are long-term implications for people and families who are dependent on public assistance. The subject of economic well being of people and families is always controversial, particularly when it concerns public policy. PRWORA, in effect, removes the federal government from the direct process of meeting the needs of the nation's needy families. The argument for such a move is that empowering state governments will allow them to tailor programs to their population, given a particular state's economic and demographic composition. Under PRWORA, states are required to limit the time a family may receive welfare benefits. In some cases, states have set limits of fewer than five years. (6) In place of long-term financial assistance, states are encouraging self-sufficiency through the use of job training, job placement, and child care and transportation assistance, as well as incentives that make employment a financially viable alternative to welfare. States also have the freedom to exempt up to 20 percent of their public assistance caseloads from the new welfare rules. Families may be granted exemptions due to the age of parent (or caretaker), disability, caring for a disabled person, caring for a young child, general hardship (including barriers to employment), high local unemployment, or being the victim of domestic violence. (7)
Welfare recipients are also required to fulfill work requirements to receive benefits. Some states require immediate compliance, while others allow 30, 60, or 90 days of receiving benefits before the work requirements begin. The work requirements can be met in a variety of ways, such as skills training, community service, work experience, and job-related and other training. As is the case with time limits, states may exempt up to 20 percent of welfare recipients from work requirements. States are required to reduce the amount of public assistance to non-exempt families that do not comply with work requirements within a specified period. (8) Under PRWORA, states may provide services to ease the transition from welfare to the paid-labor force. Consequently, most states have made available a number of new programs and services that help families become self-sufficient over the long term. Before welfare reform, one of the major objections to public assistance programs was that they did little to allow families the opportunity to gain their economic independence. Under the AFDC system, families with an adult in the paid labor force faced substantial reductions in benefit amounts. Although PRWORA did not address the specific topic of earning disregards (that is, the amount of family earnings not counted against benefit levels), it did give each state the option to adopt its own disregard policy. Most states have drafted generous disregard policies, which are designed to ease the move from welfare to work. These disregards are more generous than those that existed under the previous AFDC system.
Survey of Program Dynamics--Measuring Welfare Reform (9) PRWORA also directs the Census Bureau to undertake the task of collecting data that permits researchers to measure the effects of welfare reform. To accomplish this, the Census Bureau is conducting the Survey of Program Dynamics (SPD). The SPD represents the most substantial effort toward creating a nationally-representative longitudinal (10) data set that can be used to examine how welfare reform has affected adults and children who were previously dependent on federal entitlement programs. (11)
At the outset, the three primary goals for SPD are:
1. provide information on spells of actual and potential program participation over a ten-year period;
2. examine the causes of program participation and its long-term consequences on the well-being of recipients, their families, and their children; and
3. monitor the possible long-term changes for individuals that result from implementing welfare reform.
To measure the characteristics of the population over a ten-year period, the SPD follows households from the 1992 and 1993 panels of the Survey of Income and Program Participation (SIPP). SIPP is the Census Bureau's longitudinal vehicle for measuring changes in the social and economic status of the United States' population. The data from the 1992-93 SIPP provide analysts with up to 3 years of pre-welfare reform data. These data provide baseline estimates against which to measure the effect of welfare reform. The SPD, which will collect data for the calendar years, 1996 through 2001, will provide 5 years of post-welfare reform data. Using these combined SIPP and SPD data, analysts will measure PRWORA's effectiveness, and its effect on the families and children. The combined SIPP/SPD data will allow for the evaluation of the same people before, during, and after welfare reform.
Data for the first phase of SPD, known as the SPD Bridge, were collected in the spring of 1997. This Bridge survey served two purposes. First, it allowed the Census Bureau to reestablish contact with sample households from the 1992-93 panels of SIPP. Second, it collected data on the 1996 calendar year, which occurred just prior to PRWORA's implementation. The SPD Bridge was administered using a modified instrument from the March Demographic Supplement of the Current Population Survey (CPS). (12) The use of the March CPS instrument allowed the Census Bureau to collect information in a relatively cost-effective manner. In addition, it facilitated the use of the March CPS processing system, which minimized processing resources.
The SPD Bridge sample was drawn from the SIPP 1992 and 1993 panels. These included sample people included in the initial SIPP interview (otherwise known as original, or "100-level" sample people) and interviewed in the last wave of the 1992 or 1993 panels. This yielded a sample of approximately 30,000 households. In addition, the survey collects data on all people residing in a household occupied by an original sample person. Efforts have been taken to track movers within the sample (that is, original sample people who have moved since the last SIPP interview).
Of the 30,000 households responding to the SPD Bridge, a subsample of approximately 20,000 households are to be interviewed in the 1998 through 2002 surveys (a subsample of Bridge interviewed households were selected because the SPD budget only permitted the interviewing of 20,000 households). Low income households and households with children were selected with certainty or near certainty. (13)
SPD Bridge Content.
The use of the March CPS instrument for the SPD Bridge has a number of important implications. While the SIPP and subsequent SPD instruments provide a great deal of detail on income sources and amounts, assets and eligibility, and on program participation (including spells of program participation), the SPD Bridge provides basic data on income, earnings, program participation, and health insurance. It did not collect data on spells of program participation or eligibility for means-tested government programs. It also did not include detailed questions covering new state public assistance programs. The employment and earnings data collected in the SPD Bridge is identical to that collected by the March CPS. Like the March CPS, data collected in the SPD Bridge is for the previous calendar year. Data from this survey reflect characteristics of the sample for the calendar year 1996. Employment and earnings are collected for up to four jobs during the past calendar year for all household members over the age of 15. Weeks worked and usual hours worked at longest job are also collected.
The SPD Bridge also includes a core of income sources and amounts. There are 18 income concepts collected in the SPD Bridge; Table 1 contains each income concept and a brief description of the data collected. In addition to information collected on income sources and amounts, the SPD Bridge includes a health insurance module. These data include health insurance coverage, source of coverage (that is, private versus government), and whether children were covered by health insurance.
Limitations of the SPD Bridge Core Items.
Collecting the income items in the SPD Bridge instrument poses some challenges to longitudinal research, particularly when used in conjunction with the SIPP 1992 and 1993 longitudinal files.
First is the structure of the data files. The SIPP longitudinal files present core data in monthly format, which the user must convert to annual estimates to use with SPD data.
Second, although the frequency of SIPP data collection aids respondent recall (SIPP interviews are conducted once every four months), the difference in data collection presents problems associated with household and family dynamics. While SIPP family and household characteristics may change from one interview to the next, SPD Bridge estimates are fixed based on the month of interview, in this case April or May of 1997. (14)
The SPD Bridge provides less detail than the SIPP on program participation and dynamics in a given calendar year. For example, spells of program participation were not collected for the SPD Bridge.
The first two items (file structure and frequency) are partially reconciled using annualized SIPP data files. (15) The Urban Institute created annual, March CPS "look-alike" SIPP files from the 1992-1993 SIPP longitudinal data file. Quarterly SIPP data are annualized and recoded to look like CPS data. These files represent the base the Census Bureau will use to build its core longitudinal data set for SPD.
Beyond the SPD Bridge Survey.
In 1998, the second phase of SPD began, including the implementation of a new SPD instrument. The Census Bureau's interdivisional SPD work group developed the instrument in 1995 and 1996, with the support of the Department of Health and Human Services (DHHS) and the U.S. Department of Agriculture (USDA). The University of California at Berkeley programmed the instrument for computer-assisted interviewing using the CASES computer language. The new SPD instrument underwent a field pre-test during October 1997, and entered the field in June of 1998.
The new SPD instrument contains a number of changes and enhancements. These additions allowed the Census Bureau to collect more information about government programs that were affected, or created, due to welfare reform. A series of questions designed to collect specific information about participation in means-tested government programs are among the improvements to the 1998 SPD instrument. The new instrument also collects data on spells of program participation and information for determining eligibility for means-tested government programs.
The programs covered are:
Another of PRWORA's goals is to address the problem of moving welfare recipients into the paid-labor force through skills enhancement. The 1998 SPD addresses this issue by including a module on educational enrollment and work training. This series of questions regarding training and educational activities is asked about all adults in the household. If members of the household have reported receiving income from a means-tested program, they are asked an additional series of questions. These additional questions are designed to determine the length and scope of the training, as well as whether or not these activities were required to maintain benefits eligibility. Another brief series of questions are asked about these adults to determine whether the training helped them find a job. In addition, there are modules asking child related questions on children's school environment, enrichment activities, mother's work schedule, child care, behavior, and family routines. Additional measures of child well being are collected in 1999 and 2002; these include questions about parent-child interactions: outings with children, sports, hobbies and games, reading to children, etc. The SPD includes a substantial amount of data on child outcomes because reforms may have positive or negative consequences for children.
Adolescent Self-Administered Questionnaire (SAQ).
The 1998 SPD included a separate module, the Adolescent Self-Administered Questionnaire (SAQ), which was completed by 3,259 adolescents aged 12-17. In 2001, the adolescent self-administered questionnaire (SAQ) will be administered for a second time. Changes in a family's economic situation, whether good or bad, can have an effect on family functioning. Many believe that adolescents are most at risk for becoming welfare recipients, so the adolescent SAQ will allow researchers to study their characteristics.
Some possible scenarios follow. If a previously unemployed parent begins working, a greater share of household tasks may fall on the children. This could lead to youths having less time for schoolwork or extracurricular activities, which may have a negative effect on their school performance or attitudes toward schooling. A negative shift in family finances may also lead to additional stress in the household, which can negatively affect children and adolescents. There may be more parental conflict regarding the allocation of household funds, and parents may have fewer economic resources to provide extracurricular and enrichment activities for their children.
However, the additional income provided by a newly working parent can offset some stress the family may feel. A parent with more income may have higher self-esteem, which may be reflected in better parent-child interactions. More positive interactions could in turn result in a child engaging in more "parent-pleasing" behavior, e.g., greater involvement in school activities and less problematic behavior. More positive interactions could also be related to increased parental supervision, resulting in fewer incidences of youths engaging in risky behavior.
To collect information regarding the activities and well-being of adolescents after the 1996 Welfare Reform Act, the SPD Adolescent SAQ asked youths 107 questions about their family routines and relationships, academic involvement, social activities, alcohol and drug use, and romantic relationships. The combination of the 1998 and 2001 SAQ data will allow for an examination of trends in adolescent well-being at two points in time.
The SAQ data can be linked to SPD household data, giving researchers even more information to examine family well-being. The family and household data from the SAQ provide a context within which adolescent behavior and feelings can be interpreted.
Residential History Calendar
In 2000, the SPD will include a Children's Residential History Calendar (RHC) designed to collect complete childhood residential histories of all children of SPD respondents and their household members. The data collected will record the histories of children through their 17th year.
The SPD RHC measures the number and timing of moves that children make. Research has demonstrated that children who move frequently do not fare as well educationally as their counterparts with more stable residential situations. (16) As mobility is associated with socio-economic status, it is important to understand how changes in family economic situations affect children's residential changes.
Additionally, changes in children's living situations will be measured. Specifically, the SPD measures separations of more than 3 months from either biological parent, and periods of more than 3 months when children shared a residence with other adults (regardless of whether the children's biological parents were present). This information will be used to study how changes in children's living situations affect their well-being. Questions to be answered with this data include:
This package of information, along with the detailed family economic histories collected in the SIPP/SPD will make the SPD a unique and comprehensive source of data that can be used to study not only the effect of moves on children's educational and social/behavioral outcomes, but the effect of types of residential changes on those same outcomes. Additionally, all of these data can be examined in relation to the 1996 PRWORA, providing analysts with an opportunity to thoroughly study family and child well-being over a long time period.
SPD Public Use Files.
The first fully edited longitudinal SPD file will consist of SIPP data from the 1992 and 1993 panels, data from the 1997 SPD Bridge, and data from the 1998 SPD. The data file will thus present data for the 1992-1997 period. The first longitudinal file is scheduled for release in 2000. Subsequent files will be produced by adding additional years of data until the entire 1992-2001 file is completed.
Additional data collected each year in the SPD will be appended to the SPD longitudinal file, as will the data whose longitudinal aspect covers only two periods. For example, the adolescent self-administered questionnaire was administered in 1998 and will be administered again in 2001. This will allow analysts to study change between two periods.
The Census Bureau is also releasing a series of unedited calendar-year files; missing and incomplete answers are not imputed. Each file contains a unique person identifier that allows advanced researchers to identify original SPD sample people and link back to previous SIPP and SPD panels.
Preliminary Data from the SIPP and SPD Bridge
In February 1999, the Census Bureau released the first data collected from the SPD sample. The data are from the 1997 SPD Bridge and released primarily as experimental data in the form of a public-use data file. (17) As a matter of convenience to data users, and as an exercise to ensure the reasonability of the data, the Census Bureau produced tabulations comparing data collected from both SIPP and the SPD Bridge. The tabulations are for people and household members who completed interviews in the 1992 and 1993 SIPP panels and the 1997 SPD Bridge.
Using data from the 1992 and 1993 SIPP longitudinal files and the 1997 SPD Bridge, simple counts were computed on income recipiency for employment income, income from government transfer programs (both means and non-means tested) and retirement income. In addition, the tables include a section on health insurance coverage. SIPP is a person-based survey where people are classified by household and family type. Hence, tabulations are of people by household and family characteristics.
The tables are designed to present a profile of income recipiency for the SIPP/SPD sample over time. In order to accomplish this task, we sub-sampled those individuals who were interviewed in the 1992-1993 SIPP panels and the 1997 SPD Bridge. Subsampling in this manner allows one to examine income and program participation rates for a fixed sample over a 4-year period (1993-1996) for 3 years of data (1993, 1994, and 1996).
People are tallied by household types: all households, married-couple households, other family households, and non-family households. For married-couple and other family households the tabulations include total households as well as households with and without children. Other family households include single parent households. (18)
The SPD Bridge file is part of a long-term longitudinal database still to be completed. Therefore, given that we select only people in both SIPP and SPD for tabulations, an experimental SPD longitudinal weight was developed for the SPD Bridge file. The weighting scheme used in this paper is based on the experimental SPD weight. Tabulations are presented as normalized-weighted counts. The weights are derived by dividing each individual's weight by the average sample weight.
The results of the normalized weighting procedure resemble unweighted counts. They do, however, preserve the weighted relationship between variables. That is, the proportional distribution is the same whether normalized or straight cross-sectional weighting procedures are used. These results are not national estimates.
SPD Bridge Results. (19)
The results of the SIPP/SPD Bridge tabulations presented in this paper are not national estimates of program participation, and are presented as a guide for future research. In this section, descriptive statistics are presented from the SPD Bridge survey with comparisons to 1997 March CPS estimates and administrative sources. This is done to ensure that estimates from the SPD Bridge are reasonable, given that the instrument was used on both samples. Additionally, the March CPS results, as cross-sectional estimates, are not subject to the potential attrition bias in the SPD and SIPP estimates.
Descriptive statistics from the SPD Bridge Compared to the CPS. (20)
Prior to its release, the SPD Bridge file was rigorously reviewed using a number of diagnostic statistics to compare it to March CPS estimates. (21) Figure 1 illustrates the income distribution given selected income limits for the calendar year 1996; the data are from both the March CPS and the SPD Bridge for 1997. The figure is presented in terms of the weighted percent distribution for each sample using both the March supplemental weight from the CPS and the SPD weight from the Bridge. The distribution for both the SPD Bridge and March CPS indicate a very similar pattern of income dispersion across both surveys for 1996. The one distinction between the two distributions is that the March CPS has a higher percentage of households with income below $25,000 (35.8 percent) compared to the SPD Bridge (34.1 percent). This difference is statistically significant.
Table 2 shows another Bridge to CPS comparison, displaying selected descriptive statistics for all households and households participating in one or more means-tested government programs. The average household income constructed from the two samples is very similar. The average income from the SPD Bridge ($47,381) is not statistically different from the CPS ($47,123). (Both figures are measured in 1996 dollars.) The average age of householder for total households in both cases is approximately 49 years of age in the SPD Bridge which is statistically significant from 48.4 years of age in the CPS.
Overall, household means-tested program participation rates are statistically consistent across the two surveys. (22) Among the households in the SPD Bridge, 16.2 percent received one or more government transfer programs. However there are small, but statistically significant differences in some programs, e.g. TANF, Food Stamps, and energy assistance. Among the programs that are affected by welfare reform, free school lunches and food stamps are the most frequently received transfers: 8.7 and 7.6 percent respectively of Bridge households reported income from these sources.
The first examination of the characteristics of the SPD sample is a series of tabulations that match original sample people back to the SIPP 1992-93 panels. Although these results are not longitudinal in the sense that they do not isolate individuals or their cohorts over time, they examine the income and program participation of the same fixed SPD sample for the years 1993, 1994, and 1996.
Table 3 contains employment-income recipiency rates for original SPD sample people living in all households and by married-couple households with and without children and by other families with and without children. (23) These data serve as another diagnostic check. The receipt of employment income among all people is 57 percent in all three years, 1993, 1994, and 1996. People in married-couple households with children had a lower rate of receipt from this income source--53.1 percent in 1993 and 56.1 percent in 1996, (the increase is statistically significant). Of people living in other family households, where one or more children were present, 38.4 percent received job income in 1993 and 41.2 percent in 1996, (these percents are not statistically significant).
Married-couple households, with no children, had a high rate of job income receipt--66.3 percent in 1993 and 62.2 percent in 1996, a statistically significant decrease. Those people living in other family households, where no children were present, had the highest rate of job income recipiency: 67.1 percent in 1993 and 62.6 percent in 1996, a statistically significant decrease.
The data in Table 4 shows the rate of program participation of SPD sample people. The data show that participation in AFDC or TANF (commonly known as public assistance) has decreased since 1993 and 1994. The Bridge data presented in the third column indicate that the percentage of individuals, in all households, receiving TANF decreased from 1.9 percent in 1994 to 1.3 percent in 1996, a statistically significant decline.
These trends are consistent with Administration for Children and Families data, which show that states reported historically high caseloads during the 1994-1995 period; after which welfare caseloads declined (see Figure 2). (24) The trend in declining welfare participation since 1993 is also depicted using data collected from the March 1997 CPS (data are for 1996) for households in Figure 3, which shows the percentage of households reporting public assistance (AFDC/TANF) between 1989 and 1998. Public assistance participation as measured by households receiving AFDC (or TANF post-1996) has declined steadily since its 1993 peak.
The data show that the percentage of SPD respondents who reported receipt of noncash transfer benefits decreased from 24.7 percent in 1993 to 18.9 percent in 1996. The means-tested noncash transfer benefits programs with the largest decrease was receipt of food stamps, which decreased from 13.3 percent to 8.6 percent, and free or reduced school lunch, which decreased from 17.4 percent to 13.4 percent among SPD respondents --both statistically significant decreases.
The decline in Food Stamp participation is consistent with other data sources.
Figure 4 shows household participation in the food stamp program has declined post-1993, as evidenced by the March CPS and Department of Agriculture data.
Table 4 (continued) also shows program participation rates of people in other family households, both with and without children. Other family households with children have the highest rate of means-tested government program participation; 15.1 percent of people in these households reported receiving some form of cash assistance in 1993; in 1996 this figure fell to 12.5 percent -- a statistically significant drop in recipiency when compared to 1993.
Non-cash means-tested transfers are a much more common form of assistance among people living in other family households with at least one child present. In 1993, 69 percent of people living in other family households received some form on non-cash transfer program, which declined to 60 percent in 1996, a statistically significant drop in recipiency when compared to 1993. The most common programs were free school lunches and food stamps, where 58 and 48 percent, respectively, reported receiving benefits in 1993. In 1996, the receipt of free school lunches and food stamps declined to 51 and 34 percent, respectively, both are statistically significant declines.
Results from Adolescent Self Administered Questionnaire (SAQ).
Preliminary analyses of the unweighted SAQ data indicate approximately a quarter of youths aged 14-17 had engaged in sexual intercourse, while 40 percent of 12-17 year-olds had smoked cigarettes, 18.5 percent smoked marijuana, and 61 percent had a drink of alcohol. (25) The adolescents completing the SPD also report fairly positive relationships with their mothers -- they are evenly split across a 5-point scale (poor, moderate, good, very good, excellent) measuring affective closeness between the youth and his or her mother, with 60 percent also reporting that their interactions with their mothers are generally positive. (26)
Of the 5,579 SAQ-eligible adolescents, 58.4 percent, or 3,259, completed the questionnaire, resulting in a non-response rate of 42 percent. Examining the characteristics of the respondents has demonstrated that the adolescents completing the SAQ exhibit patterns of exposure to sex, smoking, drinking, and marijuana usage that are similar in direction to those of the national population. (27) The SAQ non-response also appears to be random with respect to several demographic variables. Thus, these data appear to be of suitable quality to be used for analysis of adolescent and family well-being, but with a strong caution that a significant proportion of children missing key background indicators may bias the survey in an unknown direction. (28)
The Youth Risk Behavior Survey (YRBS), conducted by the Centers for Disease Control and Prevention, is a national survey of high school youths designed to gauge the prevalence of behaviors which are associated with both short- and long-term health outcomes. The survey is administered in schools and asks young adults about, among other things, smoking, drinking, problem behaviors (e.g., fighting or carrying weapons to school), and sexual behavior. The YRBS has been administered every other year since 1991, which allows researchers to examine trends in the prevalence of these health risk behaviors and the implications of those trends for later public health. Comparisons of unweighted SPD-SAQ frequencies to distributions from the YRBS, presented in Table 5, suggest that SPD SAQ estimates exhibit somewhat comparable patterns to other national estimates of U.S. youth behavior.
Comparing Response Rates for the SPD, PSID, and NLS-Y.
The Survey of Program Dynamics is a longitudinal survey. The usefulness of data from any longitudinal or panel study that follows and interviews the same respondents over a period of years depends on the assumption that the data represent the relevant populations; nonresponse by members of the sample is a potential source of bias that can undermine the quality of estimates derived using longitudinal data. This section compares overall response rates between the initial interview and the most recent interview for three major national surveys: the Survey of Program Dynamics (SPD), conducted by the U.S. Census Bureau; the Panel Study of Income Dynamics (PSID), conducted by the Survey Research Center at the University of Michigan; and the National Longitudinal Survey of Youth (NLS-Y), conducted by the Center for
Human Resource Research at the Ohio State University for the Bureau of Labor Statistics. (29)
Table 6 presents the current overall response rates for specified survey periods, both including and excluding deceased persons from the base. The response rate for the entire survey period between initial sample selection and the 1997 Bridge survey is 57 percent for the SPD, slightly less than the rate of 62 percent for the NLS-Y, but substantially higher than the approximate rate of 39 percent for the PSID. (30) At present, the SPD response rate is comparable to the PSID and NLS-Y.
Two issues merit attention, however. First, response and attrition over time tends to be highly selective of persons who are concentrated among lower socioeconomic status individuals. This has been shown in analyses of the PSID, (31) the NLS-Y, (32) and the SPD and SIPP. (33) These analyses did not find evidence that sample nonresponse or attrition leads to seriously distorted estimates pertaining to a variety of topics. Although the research addressing this issue is much more extensive for the PSID and the NLS-Y than for the SPD and the SIPP, the similarities across surveys in the nature of nonresponse and attrition suggest that estimates from the SPD are in general likely to be no more biased than estimates from the PSID and NLS-Y; that is, they are likely to be of acceptably high quality.
Second, although the current cumulative response rate for the SPD lies between the corresponding rates for the PSID and NLS-Y, the latter surveys have been in the field much longer (26 to 28 interviews and 17 interviews, respectively, compared to 10 to 11 interviews for the SPD), and they have involved larger numbers of field administrations. Many PSID and NLS-Y interviews occurred, however, in the 1970s and 1980s when response rates were generally higher for all surveys than has been the case in the 1990s. In the SPD, the response rate between the last SIPP interview (the 1992 and 1993 panels) and the first SPD interview (the SPD bridge) was only 79 percent, and preliminary estimates suggest that the response rate between the first SPD interview and the 1998 SPD interview was about 83.4 percent overall, or 85.2 percent excluding both deaths and the newly-institutionalized population from the denominator.
Other longitudinal surveys (such as the PSID and NLS-Y) have gone back to early panel nonrespondents and successfully brought them back into sample. It was proposed that an exploratory study be done to test the feasibility of this process for SPD.
Compared to the SIPP, the SPD is a very long longitudinal survey. SPD covers a 10- year period versus 4 years for the 1996 SIPP panel and 3 years for other SIPP panels. Respondents from the 1992-93 SIPP panels were told that they had completed their participation in the survey, but we are returning to obtain additional interviews as required by the PRWORA legislation. Moreover, these people have refused us many times before, thus making them hard-core nonrespondents. Because of the complexities and cost of programming a computer-assisted personal interview (CAPI) instrument for a small sample, a revised 1993 SIPP panel wave 9 paper questionnaire and control card was used to interview these households instead of the SPD CAPI instrument. Three incentive amounts were tested ($0, $50, and $100) to see if the size of the incentive affected response rates. Table 7 shows the results.
The Exploratory Attrition Study sample consisted of the 358 randomly selected low-income (below 200 percent poverty threshold) cases that became nonrespondents in the 1992 and 1993 SIPP Panels and 48 cases spawned (34) after SIPP but before the SPD Exploratory Study interview, for a total of 406 cases. Of these 406 cases, 373 are eligible cases (either interviewed, refusals, no one home, temporarily absent, or unlocated movers). The remaining 33 cases are Type B or C. (Type Bs include vacant units, units under construction, or units occupied by people whose usual residence is elsewhere. Type Cs include demolished units, units converted to a business, or units that have been moved,e.g. a trailer). All the comparisons are based on the statistical hypothesis tests at the 90 percent significance level.
The preliminary analysis shows that the overall response rate for eligible cases was 37 percent. Response rates for eligible cases by incentive amounts shows that the $100 group (44 percent) had a significantly different response rate compared to the $0 group (29 percent). However, the response rate for the $50 group (37 percent) was not significantly different from the $0 group. The overall response rate for those below the poverty threshold is 39 percent, which is not significantly different than the response rate of 35 percent for those above the poverty threshold.
Incentives have a larger effect on cases that were SIPP Type A nonrespondents (refusals, no one home, and temporarily absent) than on cases that were SIPP Type Ds (unlocated movers). In the $100 group, the response rate for the SIPP Type As (54 percent) was significantly different from that for Type Ds (35 percent). Incentives could be offered only if the cases were located.
The data presented in this paper provides a first step toward understanding the effect of welfare reform on the nation's population, particularly for those participating in means-tested government programs. The presentation of the data in this paper is meant to ensure that the estimates derived from the SPD Bridge survey are consistent and representative of the population. The results show recipiency patterns that are consistent with overall economic trends as well as evidence provided in the March CPS and other administrative data sources. Analysts, however, should be cautious when drawing conclusions on the causal effect of welfare reform; further data is needed before this is done. The first SPD longitudinal public-use file is scheduled for release in 2000. The Bridge data, however, serve as a valuable tool for assessing the changing characteristics of the SIPP/SPD cohort between 1993 and 1996.
1. This paper reports the results of research and analysis undertaken by Census Bureau staff. It has undergone a more limited review than official Census Bureau publications. This report is released to inform interested parties of research and encourage discussion.
2. This section is based on U.S. General Accounting Office (GAO). Welfare Reform: Early Fiscal Effects of the TANF Block Grant. Report to the Chairman, Subcommittee on Human Resources, Committee on Ways and Means, House of Representatives. GAO/AIMD-98-137. August 1998.
Welfare Reform: Are States Restructuring to Reduce Welfare Dependence. Report to the Chairman, Subcommittee on Human Resources, Committee on Ways and Means, House of Representatives.GAO/HEHS-98-109. June 1998.
3. U.S. General Accounting Office (GAO). Welfare Reform: Early Fiscal Effects of the TANF Block Grant. Report to the Chairman, Subcommittee on Human Resources, Committee on Ways and Means, House of Representatives. GAO/AIMD-98-137. August 1998.
7. L. Jerome Gallagher, Megan Gallagher, Kevin Perese, Susan Schreiber, Keith Watson, One Year After Federal Welfare Reform: A Description of State Temporary Assistance for Needy Families (TANF) Decisions as of October 1997. The Urban Institute. Assessing the New Federalism. Occasional Paper no. 6, 1998.
9. This section draws from Daniel Weinberg, Patricia Doyle, Arthur Jones, Jr., and Stephanie Shipp, "Measuring the Impact of Welfare Reform with the Survey of Program Dynamics." Social Statistics Proceedings, American Statistical Association meetings, Dallas, Texas, August 1998.
11. Some might ask why we are not using the 1996 SIPP panel to examine the effect of welfare reform. Only some of the interesting questions can be answered using the 1996 panel of the Survey of Income and Program Participation (SIPP). The SIPP will provide nearly four years of longitudinal data for almost 37,000 households beginning in April 1996. Welfare reform legislation took effect on October 1, 1996, only six months after the 1996 SIPP panel started. One could argue that this pre-reform information is sufficient for many analyses, and the SIPP does try to collect retrospective program participation information. However, other analyses need a longer pre-reform period to accurately measure some initial condition; this need is particularly acute for this research because many (if not most) states had already begun to make changes under federal program waivers well before the beginning of the SIPP panel. A welfare reform module was collected for wave 8 of the 1996 SIPP panel. Data are being collected between August and November 1998. See Charles Nelson and Pat Doyle, "Recommendations for Measuring Income and Program Participation in the Post Welfare Reform Era," Government
Statistical Section, Joint Statistical Meetings, Baltimore, MD, August 1999. More details about the SIPP can be found on <http.://www.sipp.census.gov/sipp/>.
12. Modifications to the March CPS instrument from 1997 were relatively minor. In addition to the core items from the March CPS, the SPD Bridge included a limited number of household questions to collect additional information on public assistance, food stamps and Medicaid benefits during the 1995 calendar year for those in the 1992 SIPP sample. These items are carried as unedited fields on the Bridge public use file.
14. Typically, data are collected in a relatively short span of time. Given the nature of the SPD Bridge, however, a greater period was used to allow field representatives to track down and close out difficult cases. Therefore, the SPD interview period spanned the months of April and May of 1997.
16. Gloria A. Simpson and Mary Glenn Fowler. "Geographic Mobility and Children's Emotional/Behavioral Adjustment and School Functioning." Pediatrics 93(2):303-309. 1994. C. Jack Tucker, Jonathan Marx, and Larry Long, "'Moving On': Residential Mobility and Children's School Lives." Sociology of Education 71:111-129, 1998.
David Wood, Neal Halfon, Debra Scarlata, Paul Newacheck, and Sharon Nessim, "Impact of Family Relocation on Children's Growth, Development, School Function, and Behavior." Journal of the American Medical Association 270:1334-1338, 1993.
18. A broad-based category called other family households was used when computing the combined SIPP/SPD Bridge tabulations. This household concept includes households with at least one family present. A family consists of two or more people living together related by blood marriage or adoption. This group also includes single parent households.
19. The results contained herein are from Census Bureau cross-tabulations of SIPP and the SPD Bridge. The unabridged tabulations are available on the Internet at <http://www.sipp.census.gov/spd/pubsmain.htm>.
21. Data from the SPD Bridge were collected and processed in a manner consistent with March CPS processing. Data from the calendar year 1996 were available from the March CPS and facilitated a convenient method for comparisons.
22. A household is selected for this group if they reported receiving one or more of the following income sources: Public Assistance, also known as AFDC or TANF; Supplemental Security Income: Food Stamps; Housing Assistance, through section 8 or if the household is located in a housing project; Energy assistance; or if any children reported receiving free lunch through the federal government's school lunch program.
23. Job or employment income is an all-inclusive definition of income received through work activities. A person is said to receive job income if they reported earnings from longest job, self-employment earnings, working on a farm, farm self-employment, or wages and salary.
25. Loretta E. Bass and Barbara Downs, "What Can the SPD Adolescent SAQ Tell Us About the Well-being of Adolescents in the Aftermath of the 1996 Welfare Reform Act?" Paper presented at the Annual Meeting of the Population Association of America, New York, NY, March 24-27, 1999.
28. Barbara Downs and Loretta E. Bass, "The Survey of Program Dynamics: A New Source of Data to Explore the Effects of the 1996 Welfare Reform Act on Adolescents." Paper presented at the American Sociological Meetings, August 1999.
29. Donald J. Hernandez, "Survey of Program Dynamics (SPD), Panel Study of Income Dynamics (PSID), and the National Longitudinal Survey of Youth (NLS-Y): Sample Retention, the Accuracy of Change Estimates, and Implications for Assessing Welfare Reform," unpublished manuscript, November 1998.
30. These response rates are mortality-adjusted cumulative response rates. Although each of these rates would be slightly higher if mortality were taken into account for the period between sample selection and the first interview, the general pattern of differences across the surveys would be unchanged for rates that were fully adjusted for mortality.
31. John Fitzgerald, Peter Gottschalk, and Robert Moffit, "An Analysis of Sample Attrition in Panel Data: The Michigan Panel Study of Income Dynamics," The Journal of Human Resources XXXIII: 251-299, 1997.
John Fitzgerald, Peter Gottschalk, and Robert Moffit, "An Analysis of the Impact of Sample attrition on the Second Generation of Respondents in the Michigan Panel Study of Income Dynamics," The Journal of Human Resources XXXIII: 300-344.
Gerard J. van den Berg and Maarten Lindeboom, "Attrition in Panel Survey Data and the Estimation of Multi-State labor Market Models," The Journal of Human Resources XXXIII: 458-478, 1997.
Jeffrey E. Zabel, "An Analysis of Attrition in the Panel Study of Income Dynamics and the Survey of Income and Program Participation with an Application to a Model of Labor Market Behavior," The Journal of Human Resources XXXIII: 479-506, 1997.
James P. Ziliak and Thomas J. Kniesner, "The Importance of Sample Attrition in Life Cycle Labor Supply Estimation," The Journal of Human Resources XXXIII: 507-530, 1997.
33. Donald J.Hernandez, "Survey of Program Dynamics (SPD), Panel Study of Income Dynamics (PSID), and the National Longitudinal Survey of Youth (NLS-Y): Sample Retention, the Accuracy of Change Estimates, and Implications for Assessing Welfare Reform."
Enrique Lamas, Jan Tin, and Judith Eargle, "The Effect of Attrition on Income and Poverty Estimates from the Survey of Income and Program Participation (SIPP)," U.S. Bureau of the Census Working, SIPP Working Paper 9402, 1994.