Measuring the Impact of Welfare Reform: Issues in Designing the Survey of Program Dynamics Questionnaire
Jennifer C. Hess and Jennifer M. Rothgeb
Abstract: The Personal Responsibility and Work Opportunity Reconciliation Act of
1996, more commonly known as the Welfare Reform Act, charged the U.S. Bureau of the Census to evaluate the impact of the law. Toward this end, the Census Bureau developed the Survey of Program Dynamics (SPD). The SPD is composed of two distinct parts: one is an interviewer-administered automated instrument to be answered by an adult respondent and the second is an adolescent self-administered questionnaire. In this paper, we describe challenges we faced in developing, designing and testing these two new survey instruments. Development issues include defining the content of this omnibus survey to meet the needs of the legislation and limiting the scope so as not to overburden respondents and exceed budgetary constraints. Design issues include incorporating both household- and person-level questions to improve the efficiency of collecting data, and administering the questions for the adolescent questionnaire with an audio-cassette player (with headphones) to ensure privacy for the adolescent respondent when answering potentially sensitive questions on various behaviors and practices. Testing issues include conducting cognitive interviews from the paper version of the automated adult questionnaire because of a compressed schedule for pretesting the instrument, and conducting cognitive interviews on an instrument designed to be administered by cassette player with adolescent respondents. We will describe the different challenges we faced and discuss how each was resolved.
Keywords: welfare reform, cognitive interviewing, questionnaire design
President Clinton signed The Personal Responsibility and Work Opportunity Reconciliation Act of 1996, more commonly know as the Welfare Reform Act, on August 22, 1996. One section of the Act charged the U.S. Bureau of the Census:
Toward this end, the Census Bureau developed the Survey of Program Dynamics (SPD). With current funding, the SPD will extend the 1992/93 SIPP panels through 2001 resulting in 10-years of longitudinal data. This paper describes the challenges we faced in developing, designing, and testing the SPD survey instruments. (See Weinberg et al. in this volume for background information about the SPD.)
II. Development of the Survey Instrument
The SPD is comprised of two parts. The first part is called the "core" instrument and includes questions about adults and children. The adult questions, with a few minor exceptions, are asked of all household members ages 15 and over. The core questionnaire was designed for computer-assisted personal interviewing (CAPI.) The second part is a separate self-administered questionnaire (SAQ) for adolescents 12-17 years of age. Provided below is a brief description of the content areas that are included in the various components of the SPD.
A. Adult Questions
1. Employment, earnings
One of the primary goals of SPD is to chart the economic well-being of families over time in order to evaluate the impact of welfare reform. We collect whether adults are currently working and a detailed account of work-related activities in the past calendar year, including weeks they worked, weeks on layoff, and weeks spent looking for work. We collect detailed employment data for up to four jobs in the previous calendar year.
2. Income Sources, Amounts, and Eligibility
In addition to earnings, the SPD collects data on a comprehensive list of other income sources, including cash and non-cash transfer programs. Data for these items include the type of income received, who received it, months received (if appropriate), and amount received for the previous calendar year. In addition to income, we ask about assets and debts. These items provide information on the economic well-being of persons and households, eligibility for welfare programs, receipt of welfare and other cash and non-cash assistance, and duration of welfare spells.
3. Educational enrollment and work training
The adult questions on educational enrollment will track the progress of adults toward receiving high school or high school equivalency degrees as well as college and graduate degrees. The work training questions will focus on whether adults received any work training designed to help them look for a job or to train them for a new job. Both educational enrollment and work training are important activities to monitor since participation in these activities is tied to eligibility for receiving welfare benefits in some states.
4. Disability, health care utilization, and health insurance
Questions on disability, health care utilization and health insurance are condensed versions of similar series included as topical modules in SIPP. These questions are included to measure changes in the US health care system and how the changes affect accessibility to government health insurance, such as Medicaid and Medicare, as well as private and employer-provided insurance, utilization of health services, and health-related outcomes.
5. Food security
This series of food security questions is a shortened version of the USDA-sponsored Food Security Supplement to the Current Population Survey (CPS) and is intended to measure the subjective experience of hunger. The questions are used as a scale to measure the severity of hunger in a household. Direct changes in the Food Stamp program account for nearly half the total Federal cost savings under the legislation. Additionally, the food stamp benefits of legal immigrants and able-bodied persons ages 18-50 years old without dependents were affected almost immediately by changes in the legislation.
6. Marital relationship and conflict and adult depression
Another objective of welfare reform is to encourage marital and family stability. Questions on marital relationship and conflict and adult depression provide indicators of marital happiness and of overall stress that can contribute to marital and family harmony or instability. Changes in program participation and employment can have fairly large and immediate consequences for marital and family stability. These questions are asked of the respondent only and are included following the child-related questions. Because of the personal nature of these questions, the Field Representative will turn the laptop computer toward the respondent and allow the respondent to answer these questions by himself/herself.
B. Child-related Questions
1. School enrollment and enrichment activities
The school enrollment questions track children's progress through and out of school over time. As household and family conditions change, student progress may be impeded or facilitated. Questions on extracurricular enrichment activities add to the overall portrait of the child's development. The extent to which parents have the financial resources or the time to devote to such activities may be strongly influenced by their program participation and employment in the labor market. These questions provide the basis to study how welfare reforms affect key child outcomes by influencing children's exposure to enriching activities.
2. Disability and health care utilization
These questions are similar to those asked about adults and are described in Section II, Item 4 above.
3. Child care
A key objective of welfare reform is to encourage single mothers to enter or re-enter the labor force. Children of these mothers will need to be taken care of during the time the mother is at work. The amount of time children are in child care and the type of care they receive, as well as the stability of care arrangements has been linked with child well-being. If the demand of child care outstrips the supply, or if child care is too costly, there is the possibility that greater numbers of children will be left to care for themselves.
We will ask about all arrangements used since January of the previous calendar year, until the date of the interview. We will know which type of arrangement was used and which months that arrangement was used. This will allow analysts to match the child care data to the employment data for the preceding calendar year. Detailed questions about hours used per week, cost and whether the arrangement is subsidized will be asked only for arrangements currently used.
4. Child support and contact with absent parent
Improved enforcement of child support agreements has been highlighted as a cornerstone of welfare reform. Questions on child support will allow researchers to examine the nature of the awards and whether the awards are being followed or enforced. Child support payments are also an important factor in determining the economic status of children living in single-parent families. Another objective of welfare reform is to encourage closer family ties and greater responsibility of parents for their children. Absent parents may participate in and contribute to their children's well-being by providing economic resources or by spending time with them, or both. Questions on contact with absent parents measure the amount of time the non-residential parent spends with their children.
C. Adolescent Self-Administered Questionnaire
The Census Bureau, Child Trends, Inc., and the National Institute of Child Health and Human Development's Child Well-being Research Network collaborated to develop the content of the adolescent self-administered questionnaire (SAQ). Adolescence is a time when youths develop the skills and characteristics that increase or decrease the risk of intergenerational dependency. We thought it was important to interview adolescents about their own behaviors because adolescents are often more knowledgeable about their own activities and perceptions than their parents are and collecting data directly from the adolescent will likely improve measurement of these concepts. Provided below is a list of the content areas included in the adolescent questionnaire.
2. Parent-child relationships
3. Parental monitoring
4. Contact with absent parents
5. School engagement
6. Minor problem behaviors and substance abuse
7. Knowledge of and attitude towards welfare regulations
8. Dating, early sexual initiation, contraception, and child bearing
III. Questionnaire Design Issues
During the development of the SPD, we confronted various questionnaire design issues as the draft questionnaire was reviewed by subject matter and survey methodology experts. Some of the design decisions required major revisions of the questionnaire.
A. Adult questionnaire
The questions on income sources, income amounts, assets and debts were initially designed similar to the March Income Supplement to the CPS. In that survey, all questions about one income source (e.g. who received it and the annual amount received) are asked before asking about the next income source. In order to reduce item nonresponse due to a conditioning effect, we abandoned this design in favor of the design used in the SIPP. In SIPP, we collect an inventory of income sources first. After compiling the different types of income received by all members of the household (e.g. unemployment compensation, Social Security, public assistance), we then ask the amount received from each income source identified. This was done to reduce the likelihood that respondents would stop reporting income sources because they don't want to answer all the follow-up questions.
In a departure from SIPP, which collects this information person-by-person to encourage self reporting, SPD uses household-level screening questions for each income source (e.g. Did anyone in this household receive any unemployment compensation payments at any time during 1996?) and a single household respondent. This was done to increase efficiency and reduce the amount of time spent collecting this type of information. Increased usage of proxy reporting in SPD will likely lead to some under reporting of income sources. To minimize this type of under reporting, we included extensive use of flashcards during the collection of the income source data. Flashcards listing specific types of income associated with a broader income category are shown to respondents as the Field Representative reads the household-level screening question for that particular income source (e.g. "This is a list of different sources of retirement income. Did anyone in this household receive any pension or retirement income from a previous employer or union, or any other type of retirement income during 1996?"). The theory is that the specific terms included on the flashcard may be more familiar to the respondent than the broad income category included in the question and may improve respondent reporting of income sources, part icularly those of other household members. In addition to the use of flashcards, we explicitly encourage respondents to use records to report their earnings and income data.
Another design issue is the collection of income amounts. To assess the impact of time limits associated with welfare reform, we needed to collect both the months a particular income source was received as well as the amount received. In SIPP, which collects data quarterly, amounts are collected for each month of the reference period. In the March CPS Income Supplement, respondents are requested to report an annual amount. Based on research conducted during the redesign of the CPS labor force questions, we opted for a design that allowed respondents to report the income source in the manner that was easiest for them. The computer then calculates an annual amount that the Field Representative confirms with the respondent. This method was shown to reduce item non-response to earnings questions in the CPS.
B. Child-related questions
As with many demographic household surveys, any household member age 15 or over is eligible to be the household respondent for the SPD; however, for the child-related questions we decided to be more restrictive. Census Bureau experts on children's issues indicated that mothers tend to know more about their children than fathers. They recommended asking the child-related questions of the "designated parent." In the SPD, the designated parent is defined as the mother in two-parent families, the resident parent in single parent families, and as the "person most knowledgeable about the child and his/her activities" in households without a parent. If the mother is not available, we will interview the father. If neither parent is available, we will schedule a call back to talk to the mother. These procedures will increase costs and may also increase item non-response if the Field Representative is unable to collect the data at a later date. However, researchers believe that the benefits associated with improved data quality outweigh the costs and risks.
C. Adolescent Self-Administered Questionnaire
The adolescent SAQ contains potentially sensitive questions on problem behaviors, alcohol and drug use, sexual activity and contraception. Protecting the privacy of adolescents was paramount in designing this part of the survey. The questionnaire format and procedures mirror those used in the 1992 Youth Behavior Survey (YBS), which asked similar types of questions. Adolescents who are home at the time the Field Representative visits the household will be administered the survey by using an audio-cassette player and will fill out an answer booklet while listening to the tape. The answer booklet contains the answers only and not the questions. Upon completion, the adolescent is instructed to place the answer booklet in the envelope provided and seal it before returning it to the Field Representative. We also developed a separate booklet that contains the survey questions only. This booklet will be shown to parents who request to see the questionnaire. For privacy reasons, the questions are in a different order than those on the tape.
Based on results from the YBS, we estimate that half to two-thirds of the adolescents will not be home at the time of the original interview. We will not make callbacks to administer the adolescent SAQ in person. Instead the Field Representative is instructed to conduct the interview by phone. To protect the privacy of the adolescent during telephone administration, we modified the questionnaire to ensure that answers provided would not reveal the content of the question asked. For example, the following question was included on the cassette tape version of the questionnaire regarding the last time the respondent had sex:
"What method did you or your partner use? Please choose all that apply."
_____ No method
_____ Birth control pills
_____ Foam, jelly or cream
_____ Cervical cap
_____ Suppository or insert
_____ Female condom, vaginal pouch
_____ IUD, coil, loop
_____ Depo-Provera, injectables
_____ Morning after pills
_____ Rhythm or safe period
_____ Withdrawal, pulling out
_____ Other method
_____ Not sure
Telephone administration of this question may compromise the adolescent's privacy if he/she answers by giving the name of the method. Therefore, for telephone administration, we modified this question so that the respondent could provide a yes/no response:
IV. Questionnaire Testing
A. Adult and Child-related Questions
The adult and child-related questions in SPD were designed for a CAPI environment. Originally plans called for cognitively testing sections of the automated instrument as they became available. However, we had to work under a compressed time schedule, which meant that cognitive testing and instrument automation occurred simultaneously rather than consecutively. Rather than eliminate the cognitive testing, we decided to cognitively test those sections of the questionnaire that could be conducted, albeit somewhat difficultly, on paper. These included all adult- and child-related sections with the exception of the employment, income sources, income amounts, and eligibility questions. The complex skip patterns in these series of questions made conducting an interview on paper impossible.
Testing using a paper instrument proved quite useful. We were able to identify individual questions and series of questions that caused problems for respondents. Problems identified included confusing and unclear reference periods, terms and concepts not well understood by respondents, and items that were too difficult for respondents to answer accurately. Revisions were made to specific items as well as entire series.
B. Adolescent SAQ
We conducted cognitive interviews with adolescents ages 12-17 using the version of the SAQ designed to be administered by an audio-cassette player. The objectives of the test included evaluating question understanding, task difficulty, and question sensitivity. To address the first two of these objectives, we conducted interviewer-administered interviews and instructed respondents to "think-aloud" as they answered the questions. Although this method of administration does not mirror the field administration by an audio-cassette player, we believed that using a retrospective technique (with an audio-cassette instrument) would jeopardize our ability to adequately evaluate question understanding and task difficulty.
Three researchers at the Census Bureau's Center for Survey Methods Research conducted the interviews. To ensure comparability across surveys, we developed a protocol beforehand that included additional probing questions to be used at the interviewer's discretion if the respondent did not convey the information while thinking aloud or didn't convey the information after general probes such as, "Could you tell me more about that?" At the end of the protocol we included a few debriefing questions regarding question difficulty and question sensitivity.
Provided below we've identified some of the areas that caused the most problems for adolescents and the revisions that were made to the questionnaire.
1. Respondents tended to ignore reference periods when they were included in the questions, such as "During the past 30 days,...," We revised the questionnaire to include all reference periods in the response options.
2. Respondents tended to interpret lists of examples too narrowly rather than as examples of a broader class of similar activities or events. They would report only about those activities included in the list. We recommended being very cautious of including such lists. In some cases, we deleted the list. In other cases, we revised the list to include items we believed best reflected the concept of interest.
3. Respondents had great difficulty reporting their contact with their absent parent in terms of a "typical month." They tended to report the last time the event happened if it was infrequent, or over report, by guessing, if the event occurred frequently. We revised these questions to ask "how often" the event happens and included categorical response categories ranging from "never" to "everyday or almost everyday."
4. The series of questions on attitudes toward welfare included a response scale ranging from "strongly agree" to "strongly disagree" with a middle category of "I'm in the middle." We found that respondents used this middle category for two purposes: 1) to indicate that they both agreed and disagreed with the statement and 2) to indicate that they didn't know or didn't have an opinion about the statement. We revised these questions to include a specific "don't know" category.
5. The questions on relationship with fathers (in the series on parent-child relationships) referred to the "man who is most like a father to you." A couple of respondents who live in single-parent families with their mothers answered these questions about their mothers since they believe that their mothers are filling the role of both mother and father within their household. We changed the answer category from "there is no one like a father to me" to "I don't live with a biological, adoptive, step, or other father figure."
6. Although there was great concern that the questions about delinquent behaviors, alcohol and drug use, dating and sexual activity would be highly sensitive, only one respondent said he/she was uncomfortable answering one of the sex questions. Some respondents indicated they would be more comfortable using the procedure that will actually be used for the survey (answering the questions privately by listening to a cassette recorder and marking an answer sheet), rather than responding to an interviewer as was done in cognitive testing.
7. The cognitive interviews last from 60 to 90 minutes. We were concerned that the length of the interview and the tedious task of thinking aloud might prove too difficult for adolescents, who are generally portrayed as non-communicative and unable to focus for an extended period of time. Our experience proved contrary to expectations. Adolescents were quite capable of articulating their thoughts in a think-aloud setting and quite able to focus throughout the lengthy interview. Based on our experience, we found a greater need to probe during these interviews than is typically done during cognitive interviews with adult respondents.
V. Plans for Evaluating the Pretest
A. Adult and Child-related Questions
We used several methods to evaluate the pretest questionnaire, interviewing materials, manuals, and procedures. Census Bureau staff observed interviews in all four Regional Offices participating in the pretest and completed an Interviewing Observation Form for each household they observe. The form was quite detailed and covered areas of concern such as difficulty administering the adolescent questionnaire at the same time as the adult questionnaire, adolescents ability to use the audio-cassette recorder to answer the SAQ, flashcard usage, disruptiveness of changing respondents for the child-related questions, and difficulty with specific questions or series of questions (e.g. confusion with the reference period, terms or concepts that were not understood, questions that required extensive probing, etc.).
All Field Representatives were requested to tape two complete interviews, with permission of the respondent, to be used for subsequent behavior coding. Behavior coding is the systematic coding of interviewer and respondent interactions. Due to the limited time we have to analyze the pretest data (less than six weeks) and the length of the survey (approximately 60 minutes), coding each question contained in the 90 tapes we hope to get is not possible. We developed a coding scheme that is less systematic and more qualitative in nature than those typically used, but hope that it will still yield sufficient information to identify problematic items and provide some information on possible solutions for fixing the items.
Representatives from CSMR and the Census Bureau's Field Division facilitated debriefings sessions with all Field Representatives participating in the pretest (similar to a focus group). Each session contained 8 to 10 Field Representatives. Topics covered included those contained in the observer form described above, as well as record usage, screen layout of the computerized instrument, problems with the instrument (rostering, demographics, function keys, etc.), manuals, training, case management, and the length of the interview. Prior to interviewing, Field Representative's were informed of these debriefing sessions and given a diary to record any problems or observations they have. The diaries were divided into sections based on the content areas outlined in the protocol. Field Representative's were instructed to complete the relevant sections of the diaries on a flow basis so that important information is not forgotten. The diaries helped to keep the discussion focused during the debriefing sessions and make them as productive as possible.
In addition to the evaluation techniques mentioned above, the pretest will also provide data on the length of the survey. We have budgeted for a survey that averages 60 minute per household. We developed a plan to ask selected sections of the questionnaire every two years rather than every year as originally planned and chose sections that we thought would be least likely to change dramatically from year to year (adult and child disability and health care utilization, child enrichment activities, and contact with absent parent). This plan will be implemented if the interview exceeds the targeted 60-minute household average. Subject matter experts preferred this solution because they believed they had already whittled down their series to the bare bones and couldn't adequately measure the concepts with fewer items. The instrument authors preferred this solution, as well, since skipping over an entire self-contained section requires a minimal amount of programming, whereas deleting specific questions may affect skip patterns throughout the instrument and require extensive testing of the revised instrument.
B. Adolescent SAQ
We included a series of respondent debriefing questions at the end of the adolescent questionnaire. The adolescent questionnaire contains several series of questions with identical response categories. Owing to concerns about literacy, especially among younger adolescents, we were unsure whether we needed to read all response options for every question. We recorded two different versions of the tape: one in which the answer categories were read for every question, and a second in which the answer categories were read only the first time a series of questions with the same categories was asked. We included debriefing questions at the end of the survey to asses the pace of the tape, whether there was adequate time to mark the answer sheet, and preference for the reading of the answer categories. In addition we asked about privacy concerns if the questions would have been included in the answer booklet (alleviating the need for the audio cassette recorder), the adolescent's ability to concentrate throughout the 30-minute interview, the respondent's level of interest in the survey, and his/her level of comfort answering selected series of potentially sensitive questions.
Child Trends, Inc. will conduct analyses to assess the internal consistency of scales included in the adolescent SAQ, such as those measuring positive relationships with parents, parental monitoring, and school engagement. They will also analyze the frequency of responses such as "don't know," "not applicable," and "no response". This will allow us to identify questions that the respondents had trouble understanding or felt uncomfortable answering. In addition, they will examine whether respondents failed to finish filling out the questionnaire, which may indicate that the questionnaire is too lengthy for the respondent's attention span.
VI. Future Plans
Pretest evaluation will be completed in Fall 1997 and a revised draft questionnaire completed by mid-December 1997. Production SPD will be implemented in Spring 1998.