This the second of two reports on the development and cognitive testing of modified messages on American Community Survey (ACS) envelopes, letters, and a flyer in 2009 for use in 2010 during the massive 2010 Census advertising campaign and mobilization. The aim of the overall project was to develop new messages to mitigate, or at least reduce, an anticipated decline, similar to that seen during Census 2000, in ACS mail return rates for households in 2010 that would receive mailings to complete both the ACS and the census questionnaires.

This report presents results of two phases of development and cognitive testing of experimental messages on envelopes and letters in the ACS prenotice and initial and replacement questionnaire packages. The messages in the letters informed respondents that the ACS is separate from the census, is used for different purposes, and that response to both the ACS and the census is mandatory. In Phase 1, we also developed and tested new envelopes with the survey name, "THE American Community Survey," in a new green text box centered above the address window to distinguish the ACS materials from 2010 Census materials, for use during 2010 when some households would receive both the ACS and the census materials.

We aimed to learn if the envelopes and letters were effective in distinguishing the ACS from the census questionnaires and in conveying the message that respondents were required to complete both. We asked respondents how likely they would be to complete and return both forms to assess initial reactions on the effectiveness of the revised materials in promoting response. During this initial testing, we noted that a number of cognitive respondents had not heard of the ACS, some thought the ACS materials might not be from the government and some said they might not be likely to open and complete it if they received it in the mail.

These initial results suggested this approach of separating the ACS from the census was not working as well as had been hoped in potentially increasing ACS response. As a result, we decided to stop testing midstream and redesign the envelopes to link them clearly to the Census Bureau to legitimize this survey as a government-sponsored data collection. We did this by revising the envelope - €”adding into the toned-down green text box the words, "€œU.S. Census Bureau" above "THE American Community Survey" - €”and then conducted a second round of testing with a different set of respondents. Phase 1 in this report refers to the initial round with the first set of materials and Phase 2 refers to the second round with testing of the revised envelopes. Phase 3 refers to the later development and testing of a flyer for use during the final personal visit operation (Schwede and Sorokin 2009a).

The addition of "€œU.S. Census Bureau" above the survey name did appear to increase the legitimacy of the American Community Survey for some Phase 2 respondents who had not previously heard of this survey. All but one of the respondents said they would be very likely or somewhat likely to complete and mail an ACS form if they got one in the mail while they were also receiving census forms to complete. This was confirmed when vii half of our respondents mentioned that if the agency name had not been included in the text box, they might have been less likely to open the envelope and complete the survey.

The green color had mixed results. Results suggested it was more noticeable than the 2010 Census form with which it was tested, but less official than that form. While most respondents liked the green, there were mixed results on whether the green, or a third color in general, would make the questionnaire more effective or not. The letters worked well and we did not make changes to them for Phase 2.

One notable finding involved associations between the way that respondents took the five items out of the ACS envelope (from the front with the questionnaire on top or not) and their likeliness to find and read the cover letter in the questionnaire package. During Phase 1 observations, we noticed two patterns in how respondents removed the five inserts from the questionnaire packages. First, about two-thirds of the respondents opened the package from the front of the envelope, pulling out the stack with the questionnaire on top, while the remaining third opened the envelope from the back flap to see the back of the return envelope on top. Those who opened the questionnaire package from the front during the testing focused on the questionnaire on top and were less likely to find and read the experimental letter among the other inserts than those who started from the back of the envelope and moved down the stack of inserts to see the letter before getting to the questionnaire. There appeared to be a substantial difference between the two groups in likelihood to find and read the letter with the key messages we were trying to convey to respondents.

While this sample was very small and purposive and caution should be taken in drawing conclusions from these results, the same patterns and associations were observed again with different respondents in both Phases 2 and 3 of this project (Schwede and Sorokin 2010). Consistent results like this across three rounds of cognitive testing suggested that there might be a problem in finding the letter among the five ACS inserts when respondents pull the questionnaire out on top of the stack as they remove them from the envelope. This suggested that we consider reordering the five inserts in the envelope to make the letters more likely to be seen and read. At the request of the American Community Survey Office, the ACS processors tried to rearrange the materials in the envelope but this caused machine jams so the five inserts remained in the same order in 2010.

A later split-panel test of the experimental letters and envelopes was conducted within the American Community Survey production data collection from January to November, 2010 during the 2010 Census advertising environment (Chesnut and Davis, 2011). Two experimental envelopes were tested: one with the green text box background developed in this project and a new hybrid version with the same wording within the box, but no green color (the "€œno-color" version). The experimental envelope format with "U.S. Census Bureau" and "THE American Community Survey" enclosed within a text box above the window in the ACS questionnaire package letters achieved significantly higher mail response rates than the standard ACS envelopes in 2010 used as the control. The new hybrid no-color version had a greater increase in the mail response rate than the viii green version did, but both had significantly higher response rates than the standard ACS control envelope. The experimental letters also increased participation significantly. The best combination was the no-color experimental white envelopes paired with the experimental letters, which significantly raised response rates in 2010 before, during, and after the peak of census operations, which were from March to May, 2010 (Chesnut and Davis 2011: 8) During the census and post-census periods of the split-panel test in 2010, the majority of the ACS production sample had been allocated to the new letter and no-color version, so the mail response rate gains applied to the majority of the actual ACS production sample. From the results of the test, it appeared that the ACS in 2010 interdisciplinary working group’s goal of at least mitigating the effects of the census environment on ACS response in 2010 had been achieved.