2022 FedCASIC Virtual Conference

Addressing Data Collection Management Challenges: Ensuring Data Collection Software Meets FISMA Security Standards

Meeting Federal FISMA standards to both ensure that data collection instruments are secure and to assure the government agency that the data collected on their behalf remains secure has become critical as emphasis on data security continues to build. An examination of recent projects fielded by RTI indicates that new projects are facing more stringent requirements for documentation, and that legacy projects are now coming under the purview of federal agency security teams and are being evaluated and required to complete the Security Authorization process to receive an Authority to Operate (ATO). To help ensure that projects and surveys hosted at RTI are secure and meet documentation standards we develop to collect data are able to meet FISMA standards, RTI developed a repository and website for digitized security documentation with high usability that can be searched through metadata tagging. The documentation helps in evaluating business opportunities, building proposal text and budgets, and creating document packages for ATO assessments. This presentation demonstrates the web interface and document repository and its usefulness in addressing federal data collection projects.

Incorporating Speech Analytics Into A Telephone Survey Quality Program

Speech analytics is the branch of data science that deals with computerized processing of recordings and subsequent transcriptions to extract information. (Mishra and Sharma, 2016). Call centers, particularly of the large inbound customer service type, have increasingly been using speech analytics programs to enhance or replace their agent monitoring systems. Beginning in 2020 NORC at the University of Chicago invested in one of these systems for the Telephone Surveys and Support Operations department to explore how its application could be applied to Survey Research, particularly in the realms of quality assurance and gaining cooperation. NORC addressed several challenges in its application, from basic questions like, what the best data to feed in is, to more complex ones such as how to get maximum value from short duration calls when these systems are designed for longer customer service interactions. We will discuss the promise of the software, and how we are incorporating it into our quality program, specifically looking at what it does and does not replace from the existing human call review processes. We'll also address its reception by interviewers and quality monitors.

Digital Transformation Of Sample Frame Construction: RTI's New eListing App

In area probability household surveys, field staff use maps to enumerate and verify sample frames, locate sample dwelling units, and apply frame supplementation procedures. Traditionally, this has been a paper and pencil process, suffering from drawbacks of high shipping costs, costs of producing paper maps, labor costs, and data entry costs. Recent advances in mobile technology, adoption of GPS by mobile equipment manufacturers, modern GIS capabilities, and proliferation of mobile data networks have created exciting new possibilities. Recently, RTI has developed new systems to support fully digital listing. The first version of RTI's eListing app was built for Android. The most recent version of the eListing app targets the latest family of Microsoft Windows Surface touchscreen tablets. To completely digitize the workflow, complementary server-side systems and APIs were built to interface with eListing apps running on both platforms. This presentation describes the new eListing app for Windows. We present the end-to-end eListing workflow, screenshots showing the new app in action, and our use of the ESRI ArcGIS API for Windows to provide advanced mapping capabilities.

Leveraging The Mapillary Platform To Perform In-Office Listing

In-person listing has long been the gold standard of survey frame generation. As costs have increased and access to areas has diminished, alternatives to in-person listing have become more desirable. This has been brought into even sharper focus with the current global Covid pandemic. In-office listing using remotely accessed imagery provides a viable alternative to inperson listing. Vertical imagery is generally available through various free imagery services, while horizonal (street view) imagery can be licensed from commercial sources. However, imagery provided by free services and commercial sources suffer from geographic collection date disparities since they tend to update areas of higher growth more frequently. The Mapillary imagery hosting and display platform provides an alternative way to perform in-office listing that allows users to upload their own imagery. This imagery is captured when and where it is needed making it consistent across study areas. RTI International is currently customizing the Mapillary user interface to support the creation and review of survey frames, as well a provide a flexible platform for a wide variety of asset collection and management.

Top Three Challenges Organizations Are Encountering In Technology And Survey Computing

Panelists will identify the top challenges facing their organizations today given the changing survey technology, data systems, and programming environments. Projects today often include innovative survey technologies, the use of specialized programming customizations, incorporate administrative and extant data sources, and the integration of different devices and technologies to support data collection. The panelists will discuss the ways that their organizations are dealing with the environmental changes that they have identified, and offer examples and lessons learned in addressing these challenges.

Multimode Development And Data Quality Assessment In The Medical Expenditure Panel Survey (MEPS)

The Medical Expenditure Panel Survey (MEPS) is the nation's primary source of medical expenditures, utilization and insurance coverage. On March 17, 2020 MEPS switched from face-to-face to telephone interviewing because of the Covid-19 pandemic. MEPS continues to rely heavily on both modes, while adding video interviewing and exploring the use of web. The first part of this presentation spotlights design changes made amidst the Covid-19 pandemic to ensure MEPS could continue to collect data, critically important to measure the changes in health care use during and after the pandemic. We also discuss efforts to build a more resilient study design for the future through an increasingly multimode design. The second part of this presentation describes analyses assessing the effect of switching to telephone interviewing on the data collected by the survey in 2020. These analyses take advantage of the longitudinal design of MEPS to isolate changes in survey mode from changes due to the Covid-19 pandemic. We look at survey processes that affect data quality, such as respondent use of records during the interview, and key survey estimates, health care utilization and health insurance.

Administering In-Person Show Cards In The Telephone Mode

The Medical Expenditure Panel Survey (MEPS) was designed for the in-person mode, with interviewers providing hard copy show cards to respondents for items with long or technical response sets. Due to Covid-19, MEPS switched to the telephone mode, which meant interviewers could no longer provide physical show cards to respondents. Instead, respondents could access them online or the interviewer could read the show card text aloud. Because show cards are not designed for the telephone mode, reading show cards aloud is likely difficult for interviewers and respondents. In this presentation, we analyze interviewers' behaviors in reading show cards aloud in the telephone mode. We assess if they are more likely to read the text placed at the beginning as opposed to the end of the show card; how often they read the show card text verbatim; if their behaviors in administering the show card depend on the amount of text; and if their behaviors change over time. As MEPS continues to be a multimode data collection, this analysis may highlight opportunities for revising telephone procedures, enhancing training, or redesigning complex questions.

Impact Of Variable Selection On Mode Effect Adjustment For A Longitudinal Study

Mode effect occurs when estimates differ across modes the data is collected. Mode effect is a major concern in mixed mode surveys and research is needed to evaluate approaches to adjust for mode effect statistically. Several statistical approaches exist to adjust mode effect, all of which require fitting models with a set of variables in addition to the mode indicator. However, not many studies focus on the effect of variable selection on mode effect adjustment. This paper evaluates the impact of variable selection in a regression modeling approach to adjust mode effects using a synthetic dataset generated from a longitudinal survey that used two modes - telephone and audio computer-assisted self-interviewing. Three types of variables will be examined as potential model predictors.

Testing For A Mode Effect When Collecting Data On Rape And Sexual Assault: A Comparison Of Self-Administered Versus Interviewer Administered Surveys

This presentation will provide results of a study that tests for the effects of survey mode for the collection of rape and sexual assault victimization (RSA) among a general population of women age 18-49. The study was funded by the Bureau of Justice Statistics and was designed to improve the measurement of RSA on the National Crime Victimization Survey. Collecting reports of RSA on a survey is extremely sensitive. Based on this, one would expect there to be a significant difference between self- and interviewer administered modes. The presentation will compare estimates from a survey administered with audio computer assisted self interview (ACASI) to a parallel random digit dial telephone survey of females age 18-49. The questionnaires administered were designed using very similar survey questions. Contrary to the published literature on mode effects, this study did not find a significant difference in RSA prevalence rates between modes. In addition to the estimates of prevalence, we will also compare differences with respect to the characteristics of the incidents that were reported by respondents.

New Facets In Data Sharing, With GSA's Office Of Customer Experience

GSA's Office of Customer Experience presses against the traditional boundaries of data sharing in order to achieve and communicate an understanding of customer experience that provides both breadth and depth. To do this, we use traditional surveys, but also strategically curate a well-attended community of practice through which we share data, as well as leveraging tools like Github within the agency to bring teams together into a shared data space. This latitudinal approach allows OCE and our partners to view and interact with a variety of dataset types that traditionally stand alone, or are highly silo-ed, due to platform and practitioner mismatches. In our talk, we will walk attendees through two of the methods by which we share datasets.

Using Technology To Modernize Data Presentations In The National Crime Victimization Survey

The Bureau of Justice Statistics (BJS) released the NCVS Victimization Analysis Tool (NVAT) based on data from BJS' National Crime Victimization Survey (NCVS) in 2012. Since then, the NVAT has become a widely used tool for accessing NCVS data. Though the NVAT provided a user-friendly way to work with decades of NCVS data, given recent advancements in technology, the tabular format of the tool had become dated. In late 2019, BJS analysts partnered with a team of professionals at RTI International through a cooperative agreement to create a more modern and visually engaging dashboard. This presentation will describe the collaborative process pursued to conceptualize, develop, and test the tool so as to better engage current users of the NVAT tool and to expand the reach of the NCVS data. In addition, this presentation will cover the various internal agency efforts required to eventually deploy the dashboard in 2021 as well as best practices and lessons learned throughout the process.

Helping Federal Clients Move From Table-Based Reporting To Dynamic Dashboards: A Case Study With The National Crime Victimization Survey

The National Crime Victimization Survey (NCVS) led by the Bureau of Justice Statistics (BJS) is the primary source of information on criminal victimization in the United States. This presentation will cover BJS' progression from table-based reporting of NCVS data using the National Victimization Analysis Tool (NVAT) to a JavaScript-based custom web application (N-DASH). N-DASH allows users to investigate crime victimization data related to personal and property crime victimization, by select victim, household, and incident characteristics. It includes an array of visualizations for an overview of national crime victimization, and a dashboard-style interface for users to generate custom visualizations to address specific research questions. I will describe the iterative dashboard development process undertaken by BJS and RTI, including: 1) Data processing and ingestion, creation of custom visualizations and interfaces to suit communication needs of the data, 2) designing for appeal to target audiences, including user testing, 3) use of an Open Source, RTI-developed, 'Harness' plugin for dashboard state management, and 4) the benefits of close collaboration and communications throughout the development process.

Discovering The New And Improved data.census.gov

Data collected from over 130 surveys and programs help the U.S. Census Bureau provide a fresh portrait of America's people and housing, but these data are only meaningful to the extent that users can discover it. With the end goal in mind, the Census Bureau has recently upgraded their primary dissemination platform, data.census.gov, with improved navigation. The truly enhanced navigation makes the platform more versatile, and easier-to-use than ever before. In this session, we will provide a short demonstration of recent upgrades to data.census.gov based on user feedback and our agile development process.

Assessing The Quality Of Administrative Data For National Estimation Of Crime Statistics: The National Incident-Based Reporting System

Federal agencies are increasing their collection of detailed administrative data as an alternative to survey data for some federal statistics. The FBI's Uniform Crime Reporting (UCR) Program has always been an administrative data collection - with participation from nearly 18,000 law enforcement agencies - collecting aggregate, summary-level crime counts from agencies. In 2021, the UCR Program transitioned to collecting data through the National Incident-Based Reporting System (NIBRS), which requires law enforcement agencies to submit a much more detailed set of information about each crime reported to the police than the prior system. The conversion to NIBRS allows for a much richer set of estimates to be produced. However, with a more complex set of information collected, a greater set of data quality issues arise and need to be addressed prior to producing estimates for use as official statistics. We detail how BJS, FBI, and RTI are identifying different types of data quality issues such as outliers, partial responders, and unanticipated values. The assessments presented here are likely applicable to other federal agencies using or considering using administrative data.

Considering The Use Of Alternative Data In A New National Longitudinal Youth Survey

BLS has begun to plan for a new National Longitudinal Survey of Youth (NLSY). Because the environment for data collection has changed substantially in the 25 years that have passed since BLS last began a new cohort, BLS is impelled to consider incorporating alternative data sources in the new NLSY. The goals of this paper are threefold: to discuss the potential uses and challenges of adding alternative data to a new NLSY; to develop an approach for applying the FCSM Data Quality Framework to evaluate the suitability of potential alternative data sources; and to then use the approach to classify potential data sources by suitability. Because so many possibilities for incorporating alternative data into a new NLSY exist, the program needs to determine which alternative data to pursue, based on the many dimensions of data quality. In addition, the cost implications of acquiring and managing each data source must be considered. NLS has contracted with NORC to compile a list of alternative data sources that could be used with a new NLSY. NORC will evaluate the potential data sources, so that the NLS program can classify potential alternative data sources as suitable or not.

Online Travel Booking Site As A Source For Airfare And Lodging Price Indexes

Online travel booking sites aggregate purchasing options from multiple service providers, so that consumers can search, compare, and purchase travel services. The BLS Consumer Price Index (CPI) currently uses these sites to manually collect prices for small portions of its samples for travel-related services. CPI is researching the possibility of using price data from a travel booking site collected by webscraping as the source of much larger portions of its samples for travel-related services, to determine whether collecting a greater number of prices will improve measurements of price change for these services. Research has been carried out for airfare and lodging. Over two years of price indexes have been calculated using price data collected from the web and using thousands of more prices than CPI typically collects each month for these indexes, greatly reducing the instances of non-response imputation. The research price indexes are based on a new methodology that incorporates the average period-to-period price change of many travel reservations per sample observation instead of relying on the period-to-period price change of one travel reservation per sample observation.

The New Non-Employer Business Demographics Statistics: Responding To 20th-Century Survey-Based Statistics Challenges

The new annual Nonemployer Statistics by Demographics or NES-D is the Census Bureau's response to the challenges of 20th-century survey-based statistics while addressing 21st-century needs for more frequent high-quality data, at lower cost and no additional respondent burden. NES-D is not a survey; it is a series that uses existing administrative and census records (AR) to provide demographics for the universe of nonemployer firms by geography, industry, receipt size class and legal form of organization (LFO). The nonemployer universe is extracted from Census' Business Register, and demographics are obtained from the decennial census, the American Community Survey, SSA Numident, and VA's data. To link demographics to the business owners, we use Census' Protected Identification Key (PIK). PIKs are anonymized individual identifiers used for linkage across data sources. Firm ownership is assigned to a given demographic group if the owners in that group collectively own more than 50%. NES-D replaces the nonemployer component of the quinquennial Survey of Business Owners. Coupled with the new Annual Business Survey, providing demographics for employer firms, Census now offers business demographics via a blended-data approach that combines AR- and survey-derived estimates. NES-D will be enhanced with characteristics relevant to nonemployers, such as those related to the gig economy.

Collecting Detailed Covid-19 Vaccine History Among Medicare Beneficiaries

The Medicare Current Beneficiary Survey (MCBS) is a longitudinal survey of a nationally representative sample of the Medicare population sponsored by the Centers for Medicare & Medicaid Services (CMS) and administered by NORC at the University of Chicago. The MCBS has been at the forefront of COVID-19 vaccine data collection, closing important policy gaps by providing more timely data than administrative sources. Notably, MCBS vaccination data can be supplemented by other MCBS data on key factors like socio-demographics and chronic conditions. The MCBS began collecting COVID-19 vaccination data in early 2021 after the first vaccine received emergency use authorization. In response to rapidly changing vaccine availability and booster guidance, MCBS COVID-19 data collection was redesigned in Winter 2022 to add flexibility around policy changes and improve data quality using a vaccine roster. For each dose, the roster collects the date, manufacturer, and location received, enabling policymakers to answer questions about vaccination progress for sub-groups. We discuss key considerations in the design and implementation and preliminary findings from the field and data review.

Implications Of Using Proxy Respondents For In-Person Versus Phone Interviews

Many federal surveys use proxy respondents ('proxies') when sampled individuals are unable to participate on their own. This may increase coverage and flexibility, but also could introduce operational challenges. The Medicare Current Beneficiary Survey (MCBS), a nationally representative, longitudinal survey of the Medicare population, is conducted by the Centers for Medicare & Medicaid Services (CMS) through a contract with NORC at the University of Chicago. For beneficiaries living in the community, the interview is typically conducted with beneficiaries directly, but proxies are often needed for beneficiaries with physical or mental disabilities or who are deceased. When the MCBS transitioned from in-person to phone interviewing due to the Covid-19 pandemic, it seemed plausible that the mode change would affect when proxies were needed. This study investigates changes in the overall proportion of interviews conducted with proxies, characteristics associated with proxy interviews, and differences in data collection effort between beneficiary and proxy interviews in rounds of data collection completed before and after the mode transition.

Redesigning The Collection Of Home Health Care Data In The Medicare Current Beneficiary Survey (MCBS)

Home health care for Medicare beneficiaries is of growing interest to Medicare policy. According to the U.S. Department of Health and Human Services (2016), Medicare paid $18.1 billion for home health care for beneficiaries with postacute or longterm skilled care needs. The Medicare Current Beneficiary Survey (MCBS), a nationally representative survey of the Medicare population conducted by the Centers for Medicare & Medicaid Services (CMS) and administered by NORC at the University of Chicago, is well-positioned to collect robust home health data. CMS and NORC redesigned the MCBS Home Health Questionnaire (HHQ) to provide accurate home health data, informed by health care research, policy, and the ability to link survey data to administrative claims. This presentation discusses how home health differs from other care, why the MCBS is positioned to collect these data, evaluating the existing vs. preferred data collection design, and key features of new HHQ. The redesigned HHQ collects data in a way that closely aligns with home health delivery, making it possible for MCBS data to better support analyses on impact of payment and delivery system reforms and home health care use.

The Impact Of Single- And Mixed-Mode Designs On Establishment Survey Participation, Nonresponse Bias, And Costs

The IAB Job Vacancy Survey (IAB-JVS) is a voluntary nationally-representative establishment survey that quantifies the size and structure of job vacancies in Germany. Since 2011, it has been carried out using a concurrent mixed-mode design, with establishments receiving paper questionnaires and the option of online completion. However, this mode design is facing increasing costs and declining response rates. To counteract these trends, a more pronounced push-to-web strategy offers a promising alternative. However, a change of survey mode might affect response rates bias, nonresponse bias and costs. To test an implementation of an alternative mode design, a large-scale experiment comparing four self-administered mode designs was conducted with 155,000 establishments in the 4th quarter of the 2020 IAB-JVS.

Challenges And Approaches Related To Data Governance

This panel will discuss approaches and challenges related to data governance. Because surveys both collect data, and use existing datasets, data governance requirements can have broad impact on survey management and operations. This panel will provide the opportunity for the audience to gain an understanding of data governance requirements, challenges in meeting those requirements, and enablers to being compliant without negatively impacting survey goals. Some of the components of data governance are: Data architecture, Data quality, Data modeling and design, Data storage and operations, Data security, Data integration and interoperability, Data classification and GDPR compliance, and Meta-data.

Choose Your Own (Data Science) Adventure

Statistical agencies are increasingly leveraging data science to increase efficiency, create innovative products, and reach broader audiences. Across the Census Bureau, there are diverse teams doing data science work. While there is commonality in the problems we tackle and methods we use, our teams often look different. They are composed of federal employees and contractors with varied skills, degrees, and experience. In this panel, we highlight several approaches to assembling and developing teams that do data science work across Census to help others who are forming or upskilling their teams. We focus on four teams with unique perspectives.

Automating Quality Assessment Of Survey Interviews Using Natural Language Processing And Machine Learning

RTI manually reviews completed telephone interviews to assess interviewer performance. Each interview is assigned an overall score indicating whether the interview met expectations, as well as several sub-scores that evaluate different components of the interview. Using recordings from completed projects as training data, we have developed a machine learning algorithm to predict the overall score. The model uses text-based features derived from interview transcripts created using speech-to-text algorithms. By helping identify interviews that do not meet expectations and therefore merit indepth evaluation, our approach reduces the time needed for evaluation, lowering telephone data collection costs. The algorithm is designed to be project agnostic, meaning that minimal effort is required to apply the model to new projects and interviews. Future uses of this approach include providing reviewers with information on individual sub-scores and evaluation of interviews in real time.

Implementing An Efficient Data Quality Follow-Up (DQFU) Procedures In Large Establishment Surveys

RTI has developed surveys for many federal studies and very often analyzes the collected data to ensure high data quality. Possible issues surrounding data quality issues that arise during these checks include missing required values, responses outside of an allowable range, and responses that are inconsistent given other responses. Although web surveys can facilitate high quality data through logic checks, some clients are reluctant to use hard checks or allow for extensive error messaging during the survey. Thus, to improve the quality of collected data while honoring client preferences, RTI has, for multiple studies, implemented a Data Quality Follow-Up (DQFU) process. This process entails an interviewer calling or emailing the respondent with the intent to collect new responses for the variables in question. In our presentation, we will describe the technical components that RTI developed to facilitate this follow-up. These tools and processes are critical to ensure high quality data when clients are reluctant or prohibit the use of hard checks within a web survey environment.

Telling The Story Of The Covid-19 Pandemic Through Economic Survey Data Collection

When the COVID-19 Pandemic started in March 2020, The U.S. Census Bureau had over a dozen organizational (business and public sector) surveys in the field. The massive disruption to our respondents' workplaces meant the collection strategies we used no longer fit the needs of the respondent community. In many cases, respondents became dislocated from the records and files necessary to report when their workplaces were closed for safety reasons. Likewise, Census invested heavily in the creation and maintenance of up-to-date physical mailing addresses and office phone numbers for our respondents. Mailings and phone operations now reached empty desks and offices. The presenter will discuss the operational and methodological challenges of soliciting response during the pandemic. We will use paradata metrics from our mail and phone operations to illustrate the pandemic's impact to our respondents, the challenges of conducting organizational surveys to empty workplaces, and how the Census Bureau adjusted strategies to meet the 'new reality'. The presentation will cover the shift in collection modes to accommodate teleworking employees from several industries and highlight differences between business and public sector respondents.

Predicting Census Survey Response Rates Via Additive Regression With Interactions

An earlier crowdsourcing competition revealed that an ensemble of regression trees led to the best performance in predicting survey response rates; however, the corresponding models could not be adopted for the intended application due to limited interpretability. In this paper, we propose an additive regression with interactions to predict, with high accuracy, response rates in surveys. To facilitate interpretation we focus on parsimonious models via regularization, as well as hierarchically structured variants that provide enhanced interpretability. Our proposed method on the US Census Planning Database yields high-quality predictive models that permit actionable interpretability, for different segments of the population, without losing in predictive performance to state-of-the-art black-box machine learning methods based on gradient boosting and feedforward neural networks.

Incorporating Machine Learning Into A New Hazardous Materials Survey: Lessons Learned From Multiple Methods

At the request of the Pipeline and Hazardous Materials Safety Administration (PHMSA), Census Bureau researchers conducted several phases of research in order to bring a new hazardous material (HAZMAT) survey to fruition. With a goal of bridging the data gap regarding detailed shipping practices of HAZMAT, researchers employed three methodologies; exploratory, cognitive and usability testing to explore companies' record keeping practices as they relate to HAZMAT. These methodologies informed the evolution of the survey instrument, and the mechanisms used to address difficulties with responding. Exploratory testing first revealed that respondents often do not have easy access to the codes associated with each hazardous material, and thus in future rounds of cognitive and usability testing, machine learning was tested to better assist respondents by automatically filtering through the federal database of HAZMAT codes. This presentation will explore the methodologies researchers employed to support the creation of a new survey, focusing on how we implemented the machine learning functionality and what we learned from respondents regarding how to make that process easiest for them.

Usability Testing For Survey Research

Usability is a critical aspect of survey software systems because poor user experience can lead users to misunderstand the user interface or the meaning of a question which will lead to incorrect or incomplete responses. One way to improve usability is through implementing usability testing methods. This presentation will provide a brief overview of usability testing methods and the key areas of usability testing - the product, the users of the product, users' goals, and the metrics of evaluation. We will demonstrate these using some examples from recent surveys and also discuss what these key areas mean when evaluating the usability of surveys, and the importance of usability testing as a pretesting method.

Using Online Panels To Recruit And Survey Diverse Voices/Audiences: Best Practices And Lessons Learned

The Bureau of Justice Statistics is in the process of redesigning aspects of the National Crime Victimization Survey (NCVS) methodology and instrument. Thoughtfully and responsibly redesigning such a large and complex survey necessitates extensive research and testing. Because the NCVS focuses on specific crime victimization experiences, many of which are quite rare, finding sufficient, appropriate, and diverse research subjects for testing and cognitive interviewing can be challenging. This presentation focuses on three separate NCVS redesign and testing efforts conducted in 2020 that required the involvement of large and diverse samples of respondents with particular criminal victimization experiences, namely juvenile crime victims and their parents, victims of hate crime, and victims of identity theft. We describe the processes and lessons learned from conducting three efforts that used multiple designs, modes, and online platforms to identify, screen, cognitively interview, and survey large numbers of respondents over a several month period. We will cover the implications of using hybrid designs (probability and non-probability) and different platforms.

Using Technology To Conduct High Volume Cognitive Interviewing To Test New Content For The 2022 U.S. Economic Census

Every five years, the U.S. Census Bureau conducts the Economic Census, which provides extensive statistics about businesses that are essential to understanding the American economy. Given the number of topics (18), the number of questions (74) and the need to get feedback from so many different types of very specific businesses (size, industry classification, etc.), the process for setting up and conducting cognitive interviews was very complicated. We utilized a digital protocol and instrument with multiple paths to allow respondents to review various groups of questions. We conducted a total of 230 interviews over 3 rounds of testing in a 4-month time frame. Because interviewers entered notes and quotes in real time into the electronic protocol while conducting the interview, we could export these data and quickly discern trends in question performance. This presentation will provide an overview of the components of iterative quick turnaround cognitive testing, focusing on the technologies that were used for respondent recruitment, protocol development, and qualitative data analysis. We finish with best practices for high volume cognitive testing.

Blockdates.jl: A Context Aware Fuzzy Date Matching Solution

The date is often a critical piece of information for safety data analysis. It provides context and is necessary for measurement of event frequency and time-based trends. In some data sources, such as narrative information about an event or subject, the date is provided in various non-standardized formats. The Bureau of Transportation Statistics uses data provided in narrative, free-text format to validate and supplement reported safety event data. We developed the open-source software package BlockDates using the Julia programming language to allow the extraction of fuzzy-matched dates from a block of text. The tool leverages contextual information and draws on external date data to find the best date matches. For each identified date, multiple interpretations are proposed and scored to find the best fit. The output includes several record-level variables that help explain the result and prioritize error detection. In a sample of 59,314 narrative records that include dates, the tool returned positive scores for 96.5% of records, meaning high confidence the selected date is valid. Of those with no matching date, 77.9% were recognized correctly as having no viable match.

Bringing Efficiencies To Criminal Justice Manual Coding Through Machine Learning

Criminal justice research can often require conversion of open-ended, free-text offense descriptions into overall charge categories to aid analysis. For example, the free-text offense of 'eluding a police vehicle' would be coded to a charge category of 'Obstruction - Law Enforcement'. Since free-text offense descriptions aren't standardized and often need to be categorized in large volumes, this can result in a manual and time intensive process for researchers. Using publicly available national data to train a machine learning model, we present a web application allowing for the bulk conversion of offense text stored in common formats (e.g., XLSX, CSV) into offense categories used in criminal justice. This results in the reduction of an hourslong coding task to minutes with an overall accuracy of 93%.

O*NET And ORS Tasks Classification Project - Topic Modeling

The Occupational Requirements Survey (ORS) collects information on requirements related to the critical tasks of a job. Field economists for this survey collect the critical tasks of a job as open text. This text is not fit to publish. These critical tasks are essential to the primary purpose of the job and are collected in the ORS survey process. Our goal is to classify tasks to further the extensive research into task data and publish task data for public consumption. O*NET contains only occupation and not job level data but could be leveraged as a taxonomy for classifying tasks. O*NET could be a way to classify ORS task and act as a taxonomy. Topic modeling is a type of statistical model for discovering the abstract 'topics' that occur in a collection of documents. In our case, the documents would be O*NET generalized work activities. We use topic modeling to classify the data and then we evaluate the results. We create LDA models to describe and fit the data. We hope to present to you all our findings of measuring models of O*NET data to compare to ORS and hopefully lead to publishing task data for public consumption.

Using Natural Language Processing To Help Develop A Frame Of Energy Suppliers

The frame for the Residential Energy Consumption Survey (RECS) Energy Supplier Survey (ESS) is developed by identifying the correct names of energy suppliers reported by respondents in the RECS household survey. In the 2020 RECS, 19,000 respondents provided nearly 30,000 entries of electricity, natural gas, fuel oil, and propane supplier names in open text fields of web and paper questionnaires. A single energy supplier might be reported in multiple ways, due to abbreviations, nicknames, misspellings, and typos. In prior RECS ESS cycles, human coders manually compared these name variations against a reference list of known energy suppliers to clean up and deduplicate entries. This was a labor-intensive and timeconsuming task. To increase efficiency in the 2020 RECS, Westat turned to natural language processing (NLP). This presentation describes our use of NLP to match energy supplier names reported in the household survey to those on our reference list of suppliers. We discuss how we developed the matching algorithm using an iterative design to improve outcomes, our quality control review on the output, key results, and recommendations for using NLP matching in future ESS cycles.

Multimode Contact And Data Collection PIN Management For A Seamless Respondent Experience

Today's large-scale survey research sample recruitment and data collection environment requires increasingly complex multimode contact and instrument designs to achieve response rates historically realized in single-mode designs and presents challenges for respondent burden. We will discuss a multimode PIN management design capable of tracking contact modes and messages, linking the contacts to survey statuses, and incorporating PIN management into a harmonized multimode instrument that facilitates a smooth respondent experience across modes. This PIN design allows for the assessment of contact modes and messages and their success in achieving completed responses for subgroups of interest as well as enhanced opportunities for 'real-time' field monitoring and reporting. These paradata can inform future contact protocol design and tailored subgroup contact attempts, with the goal of improving response rates in those groups. The design also allows respondents to start a survey instrument in one mode and finish in another mode without loss of previously collected data, thereby reducing respondent burden and providing the respondent with a seamless multimode survey experience.

Adding jQuery, CSS, And HTML Programming To Voxco For Better User Experience

RTI programmed the Juveniles in Residential Placement (CJRP) and the Juvenile Residential Facility Census (JRFC) establishment surveys for a pilot study for the Office of Juvenile Justice and Delinquency Programs. The goal was to implement and evaluate improvements to existing surveys. RTI selected Voxco survey software. RTI applied jQuery, CSS, and html to Voxco survey software toward enhancing the user experience for respondents. We will display and discuss several enhancements, including: 1) Adding multiple buttons to a question to create a dashboard, so that a respondent could easily access different sections of the survey, 2) Moving the location of elements on a page and editing size, text, and color, 3) Creating custom tables, some with un-editable fields, for displaying key entries from each path through a loop, 4) Programming, in the context of someone potentially going through a loop hundreds of times, to dynamically show key data from the last 10 passes through.

Measuring The User Experience Of Software Products

QUE Index was developed in the Office of Information and Technology of the Veterans Affairs in 2021 as a quick means to measure the user experience with software products. It uses three rating questions in a four-question survey to collect user feedback and calculates a single number ranging from 20 to 100 as the product's rating, where 20 is the lowest possible rating and 100 is the highest. The fourth question in the survey is an open-ended feedback for collecting customer comments. The three product attributes measured with this methodology, quality, usability, and efficiency, are the most important factors impacting the user experience positively and negatively. Users experience the quality of a product by the responsive way it functions; they experience the usability of a product by the ease in which they are able to learn and use it; and they experience the efficiency of a product if it helps them perform their work more efficiently. Each of these three attributes are rated on a 5-point Likert scale, which then is converted to a single number by taking the weighted average of all three attributes together.

Kill That Coronavirus! Usability Testing The Disinfectants For Coronavirus Website

In March 2020, cleaning product aisles in grocery stores everywhere suddenly went empty. Everyone wanted to know, how do I kill that coronavirus? News stations blasted the Environmental Protection Agency's coronavirus website, a place where people could search for disinfectant products approved to kill the coronavirus. The user feedback coming in on EPA's new and highly trafficked website was not good. We reviewed the feedback and decided our next step would be to run user tests. Could they find the EPA-approved disinfectants for killing coronavirus on our site? Or were they leaving frustrated and empty-handed? We will share our journey with running user tests remotely and look at how this user feedback drove improvements to our website. We will show how we used Teams technology to observe the user's screen remotely as they navigated through the site.

Fielding Accessible Web Surveys

Federal government agencies are constantly fielding surveys to gather useful data from survey respondents. Collecting this data is crucial in obtaining information and making informed decisions. Collecting high-quality survey data that are reliable, replicable, and aligned with the accessibility requirements can be challenging, particularly for organizations that have not conducted Section 508 compliant surveys before (especially using the latest WCAG 2.0 Section 508 requirements which are more specific and detailed). This presentation will provide some examples and best practices intended to help with the planning, management, and documentation necessary to successfully implement high-quality Section 508 compliant surveys.

Hot Decks And Cold Values: A Solution To The Missing Data Problem?

Hot-deck imputations are commonly used for replacing missing data at U.S. Census Bureau. However, their performance is inherently limited by a trade-off: the higher the number of selected predictive features, the better matched are donors to recipients but the lower is the probability of finding a donor. We propose a strategy to minimize the said trade-off. We use recursive feature elimination to select the most predictive features for employment status by looping over multiple estimation models such as random forests, logistic regressions, and Bernoulli naï classifiers, and using a grid search over different number of features to select. We choose a model and number of features that produce the highest precision for setting up our hot-deck matrix. If a cell has no donors, we populate cells with "mode of modes" values from sequentially dropping a covariate until a mode is obtained, taking the mode of the different combination of the remaining covariate values, and taking the mode of those modes. We show how to automate this process using metadata, Python, and SAS. This systematic approach ensures higher data quality by removing ad hoc human selection of cold-deck values from the data allocation process.

Operationally Linking Multiple Real-Time Data Sources To Develop Efficient Stopping Rules

This presentation will explore using data science to allow Census to make effective stopping rules around continued data collection with the shared goals of reducing respondent burden, saving money on non-response follow-up, and maintaining or improving data quality. Such an approach would leverage real-time survey response data, paradata, operational cost, and administrative data sources. We will focus on recent technology innovations that allow advanced real-time integration of disparate data sources to simulate alternatives that enable real-time, data-driven operational decision-making. We outline a modern approach to rigorous security and access control, while still allowing high interoperability with existing data lake and operational control systems. The optimal solution will use Artificial Intelligence and Machine Learning to discover new efficiencies over time and will provide transparent data lineage and change management in an end-to-end architecture.

Using Machine Learning To Assess Interviewer And Question Performance

Computer Assisted Recorded Interviewing (CARI) has long been used by field management to monitor interviewer performance and to assess the performance of questionnaire items in large-scale nationally representative surveys such as the Medical Expenditure Panel Survey (MEPS) and the Population Assessment of Tobacco and Health (PATH) Study. Conventionally, a human coder needs to first listen to the audio recording of the interactions between the interviewer and the respondent, and then evaluate and code features of the question-and-answer sequence using a pre-specified coding scheme. Such coding tends to be labor intensive and time consuming. Due to resource constraints, often a small proportion of completed interviews or a selected group of questionnaire items can be evaluated in a timely manner. In this study, we will describe a pipeline developed at Westat that heavily draws on the use of machine learning. We will demonstrate how to use the pipeline to detect potential interviewer falsification, identify interviewers with undesirable behaviors, and detect problematic questionnaire items. We will also discuss the time and cost implications of using the pipeline.

Using The Quick Start Toolkit (QST) At The Census Bureau

The Census Bureau's Business Process Management (BPM) Program has transformed the communication of survey and administrative processes by encouraging and supporting consistent graphic process documentation. Business process models, graphical representations of workflow required to reach program goals, lie at the heart of business process management. Models enable stakeholders to understand process logic and responsibilities, resulting in more effective program services. To draw business process models, the Census Bureau uses the Quick Start Toolkit (QST), built on MS Visio and MS Excel. Released in November 2020, the third generation QST provides many new features that reduce modeling time and improve content quality, including an automated 'health check' against modeling best practices. The QST has been used to document multiple survey processes including, but not limited to, Consumer Expenditures - Diary, National Survey of College Graduates, and National Teacher Principal Survey. See a survey process model and watch as issues found by the QST automated 'health check' are corrected.

Is This Information Correct? Assessing The Burden And Data Quality Tradeoffs Of Using Extant Data

Providing survey respondents with pre-filled data, either from extant data sources or from previous rounds of a longitudinal survey, can reduce respondent burden. However, this may reduce data quality if respondents choose to satisfice or otherwise provide low-quality data. For the National Teacher and Principal Survey (NTPS), schools are asked to provide a roster of eligible teachers that form the sampling frame for a Teacher Questionnaire. Schools in the 2020-21 collection were randomly assigned to receive either a blank roster or a pre-filled list of teachers, built from commercial data sources, and asked to make any appropriate corrections or updates. This presentation quantifies tradeoffs in respondent burden, data quality, and the downstream impacts of this form of dependent interviewing. For example, we compare roster response rates, the numbers of rostered staff and eligible rostered staff, and the overall impact of this tradeoff when surveying sampled teachers where schools are asked to verify pre-filled, extant data.

Integrating Diagnostic Tools Within Blaise 5: Examples From The National Study Of Mental Health

The National Study of Mental Health (NSMH) was conducted in households as well as prisons, homeless shelters and psychiatric hospitals across the U.S. with the goal of estimating the prevalence of serious mental health and substance use disorders among adults in the U.S.. A series of multi-modal instruments were developed by RTI in Blaise 5 that included the integration of 3rd party mental health screening and diagnostic tools. The integration of external applications was a bit simpler in Blaise 4, compared to Blaise 5, as Blaise 4 was solely a desktop application. The process in Blaise 5 is quite different as 3rd party applications (CAT-MH by ATT and NetSCID by Telesage) must be capable of being launched from a desktop application and/or a web-based instrument. In this presentation, we will discuss our experience in developing the solutions for integrating 3rd-party software in Blaise 5 for multi-mode data collection and provide examples from this complex mental health study.

Using A Predictive Search Database For Address Collection In Federal Surveys

Predictive search databases (PSDs) can autocomplete respondent addresses as they are typed to match entries in the USPS's Delivery Sequence File (DSF), potentially increasing the quality of address data and reducing the need for post-survey data cleaning. However, the data quality and cost implications of using a PSD for collecting addresses is currently uninvestigated. In this presentation, we compare the quality and cost implications of collecting address data with and without a PSD. We examine three indicators across two federal postsecondary studies.

Using Linked Data Sources To Predict Occupations With Respirator Or Mask Use Based On Establishment Characteristics

A national survey of respirator use and practices will provide crucial data to inform NIOSH's Respirator Approval Program priorities. Last conducted in 2001, lessons learned indicated that to increase data validity and reliability, respondents should be directed to a single occupation within their establishment for which occupation-level data is sought. To prepare for a second national survey, NIOSH and BLS collaborated to develop a predictive algorithm to be embedded in the electronic survey and used to direct respondents to an occupation for which data will be requested, thereby increasing the likelihood of identifying an occupation that is present at the respondent's establishment and for which respirator or mask use is practiced. Drawing on 1) anonymized establishment-occupation combinations observed in the Occupational Employment and Wage Statistics program and the Survey of Occupational Injuries and Illnesses and 2) personal protective equipment use data generated from the Occupational Information Network database, the algorithm uses establishment characteristics to identify occupations that are observed at similar establishments and likely to use respirators or masks.

Order From Chaos: Using Natural Language Processing To Find Patterns In Unstructured Text Survey Responses

This presentation demonstrates our use of Natural Language Processing (NLP) within our Artificial Intelligence (AI) engine to analyze unstructured text. Two research studies are described. In the first study, we utilized AI technology to review openended survey data to determine automatically which comments contained information that would be considered an alert to be escalated and addressed urgently. Those comments were then automatically assigned a code for further review. Using the AI engine resulted in a labor reduction of over 80% (compared to the manual comment coding process), with a high level of quality and reliability. A secondary benefit was reducing the elapsed time from survey completion to alert comment reporting. In the second study, we looked at the relationship between open-ended survey comment themes and Net Promoter Scores (NPS), with the intent to improve overall customer satisfaction. Using NLP text analytics, we discovered emergent themes that help explain overall scores and evaluated which themes (both new and existing) significantly predicted NPS.

Sentiment Analysis Insights from Language of Reports (SAILOR)

Federal agencies need the ability to quickly gain insight from survey free text responses. SAILOR extracts text from .pdf survey results, outputs an excel file, and utilizes Sentimentr with a customized dictionary to analyze sentiment and emotion in survey responses. Sentimentr has several advantages over alternatives, such as a high degree of customization and emotion labels. Tailoring the default lexicon dictionary to the survey context improves results. For example, text responses like 'yes' or 'no' by default result in positive and negative polarity. However, the polarity depends on the context of the question, so these terms need to be neutralized. Similar changes need to be made with the default emotion dictionary. Reviewing the results of the emotion extraction under the default allowed us to remove word associations that did not work well in a DoD context, such as 'intelligence' being labeled as 'trust' when the respondents were using the word as a noun. Customizing the dictionaries to be specific to our input data results in more accurate analysis of the free text. Similar techniques can be used by other agencies to adapt sentiment analysis systems to their domain.

Using Sentiment And Thematic Analysis Of Open Text To Capture The 2020 Census User Experience

In 2020 the public had the option to respond to the decennial Census online. The U.S. Census Bureau then re-contacted a sample of respondents and asked them to complete a short online survey to measure satisfaction and user experience. Using an open text question at the end of the survey, respondents could also share more about their experience filling out the 2020 Census. We conducted manual coding of these text responses, in addition to using Qualtrics Text iQ tool for sentiment analysis, to gain a more nuanced understanding of the respondents' thoughts and concerns. Our thematic analysis combined inductive and deductive approaches to coding, with themes that cover usability issues observed throughout the decade (e.g. separating race and Hispanic origin), feedback from other 2020 Census operations (e.g. duplicate enumeration), the 2020 Census (e.g. citizenship question), and the 2020 climate (e.g. politics). This talk will present the results from these qualitative analyses, including their relationship to quantitative measures of satisfaction. We will also address the benefits and drawbacks of Qualtrics' overall and topic-level sentiment analysis compared to manual coding.

Analyzing 2020 Census Audio Recordings: A Machine Learning Approach

The 2020 Census was primarily conducted online, but there was limited paper, in-person, and phone reporting. For assistance, members of the public could call a toll-free number. Callers initially interacted with Interactive Voice Recognition software, which handled routine questions. For more complicated questions or enumeration requests, callers were routed to a live Customer Service Representative (CSR) at one of 11 call centers located throughout the country. CSRs answered callers' questions and, in some cases, collected census responses. With callers' permission, the calls were recorded, resulting in millions of audio recordings we are using to improve future data collections. In this presentation, we discuss our experience using machine learning methods to analyze 2020 Census audio recordings of caller-CSR interactions. Because the recordings alone would not be able to answer our research questions, the first step was transcribing the audio to text. It was a lengthy process to navigate the features of these caller-CSR interactions. With reasonable transcriptions, we conducted sentiment analysis and topic modeling to better understand the nature of these conversations.

Experiences With Home-Based Interviewer Mediated And Respondent Self-Collected Physical Measure And Specimen Collection

Health surveys are adding biomeasures, specimen collection, and genomics to self-reported data to increase utility. Several studies have considered the motivators, barriers, and willingness of individuals to participate in surveys with biomeasures and specimen collection. Participants are more willing to have less intrusive biomeasures collected such as height, weight, waist circumference, and saliva than intrusive measures such as a blood draw collected in their home. Incorporating these measures into a study may expand the data available for research, broaden the base of researchers using the data, and encourage more complex analyses. This session will include presentations on four national studies that are utilizing novel approaches to collect biomeasures. We will discuss the differences between sampling strategies, staffing for in-home examinations versus self-collection with bio-kits, impact on respondent burden, organizational logistics, equipment selection tradeoffs, cooperation rates, loss to follow-up, and timing of biomeasure and specimen collection within the overall study protocol.

Covid-19's Effects On RDD Sample Productivity: Shock And On-Going Effects

Covid-19 has had a complex impact on survey productivity, and presents a type of 'natural experiment' through which we can learn about a pandemic's effect on our industry. During Covid's first year, we observed possible shock effects (i.e., large increases in March 2020) on RDD contact response, and refusal conversions, but had difficulty identifying ongoing effects. By the end of 2020, these sample productivity metrics had largely returned to expected levels, although large between-surveys variability remained. The current presentation extends our past Covid effects evaluation by adding sample productivity from 2021 for the same 12 surveys evaluated previously. In addition, it adds statistical tests of shock and on-going effects (i.e., tests of differences in productivity rates between months or groups of months, and tests of between-survey variability). Findings are discussed in the context of Covid-19's impact on data collection and the general difficulty of disentangling effects from natural experiments. Results should help survey organizations understand Covid effects and plan for future situations, large and small.

The Impact Of Prenotification Mailings On Response Rates In A RDD Sample

Prenotification letters have the potential to increase response in random digit dialing telephone surveys by alerting respondents to the impending call and increasing the likelihood they pick up. Addresses are obtained by matching sampled telephone numbers addresses from various commercial databases. A criticism of prenotification letters is that they only reach a nonrandom subset of sampled numbers potentially biasing results. The value of prenotification letters for cell phone samples has limitations due to address matching. This study reports on the tracking of pre-notification letters mailed for both the TX and WA BRFSS surveys. We explore if prenotification letters increase overall response rates to the survey. We examine the cost benefit trade-off of sending prenotification letters. We also compare the demographic differences between completed surveys from the matched and nonmatched addresses. The findings of this research are applicable to the many federal research studies which use prenotification letters as part of their data collections. The findings of this state level data can be transferrable to improve methodologies for data collection on the federal level as well.

Engaging Establishment Survey Respondents Using Electronic Communication Methods

As part of the agency's data collection modernization efforts, the National Agricultural Statistics Service (NASS) is incorporating electronic options, such as email and text messaging, to provide survey reminders with web links to facilitate online access. Over the past year, the agency has worked to develop the NASS Enterprise Messaging Outreach (NEMO) system to operationalize the requirements and methods for using electronic communications. Our presentation will discuss the research studies conducted and lessons learned using electronic communications to engage respondents. To understand the effectiveness of our email campaigns, we conducted several studies focused on factors that may impact respondent behavior such as varying the email subject and the number of emails sent within treatment groups. We evaluated analytics (e.g., open/click rates) to better understand respondent engagement with the emails and the impact on response. Lastly, we included a feedback questionnaire in the web instrument to gather respondents' reactions to the reminders. Our findings will help us improve the efficacy of future electronic campaigns to engage respondents during survey data collection.

Identifying And Mitigating Nonresponse Bias In School Surveys During Covid-19: The 2021 Adolescent Behaviors And Experiences Survey (ABES) Case

The Adolescent Behaviors and Experiences Survey (ABES) was a CDC-sponsored national student survey that assessed risk behaviors and experiences during the Covid-19 pandemic. The ABES was administered online to a nationally representative sample of high school students in the spring of 2021 when Covid-19-related interruptions to the instructional environment were present nationwide. In that fluid environment, nonresponse was a concern for the validity of survey data. This study discusses potential nonresponse biases and the strategies used to minimize their impact. We compared participating and non-participating schools' characteristics to evaluate potential nonresponse bias. Our multivariate analyses (logistic regression models) found that participating schools were more likely to be from the Midwest region, less urban areas, poorer, and provided hybrid or 100% virtual instruction. We created nonresponse weight adjustment cells based on the four significant predictors to minimize bias. The analyses demonstrate that a national school-based survey could provide valid and timely data on risk behaviors even in the middle of a pandemic.

Designing A Machine Learning Pipeline To Complement Survey Analysis

The presenters will demonstrate how to identify, evaluate, and apply current approaches in machine learning for their suitability in supporting survey analysis, with an emphasis on machine learning models for audio and text data. Recent advances in machine learning enable quantitative analysis of free text in survey responses, behavioral metadata from survey completion, audio from survey call centers, and video from focus groups. While the latest advances are exciting, identifying what approaches are available from public and private research, knowing what is the current state-of-the-art, implementing these approaches in practice, and evaluating the success of these approaches can be complex. This job is made easier, though, through a basic understanding of the kinds of approaches that are available, and common resources and frameworks used by researchers and practitioners in the field. This presentation will aim to provide that basic introduction, sharing ideas essential to identifying and evaluating the available tools, and will provide an example of these ideas applied in practice using audio and text data.

Choice Matters: Testing Multiple Video Platforms To Collect Survey Data

Computer Assisted Video Interviews (CAVI) may increase respondent burden with extra steps to log onto the video platform, reducing willingness to participate (Schober, et al., 2020; Larsen, et al., 2021). To reduce this burden, different video platforms have been offered to respondents (Hanson, 2021; Schober, et al. 2020). However, it has not been addressed how different platforms' features work, nor how this may affect the interviewer or respondent experience. In October, 2021, Westat began testing CAVI as a potential methodology for the Medical Expenditure Panel Survey-Household Component (MEPS-HC). Specifically, we set out to explore the feasibility of offering a video interview for interviews in households in later panel rounds that might otherwise be conducted by phone, testing features such as screen sharing and the use of showcards, the ability to expand the size of the shared content, and other features. We will present findings from mock interviews tested on multiple platforms, with a focus on the different features of each, and how the implementation of CAVI may impact interviewers and respondents. The potential impact of these on MEPS data quality will be discussed.

Do Humor And Relatability Predict Interviewer Performance?

Interviewers' personality and interpersonal skills can increase their cooperation success. Humor and relatability are two under-researched dimensions in random digit dial computer-assisted telephone interview (CATI) surveys. This talk addresses whether subjective assessments of interviewers' sense of humor and relatability correlate with important performance metrics, such as contact, refusals, refusal conversions, and cooperation? Interviewer quality assurance (QA) monitors were asked to nominate interviewers who they thought had the best senses of humor or were the most relatable via two simple questions emailed to them. Four groups of interviewers were created: 1) nominated funny only, 2) nominated relatable only, 3) nominated both funny and relatable, and 4) not nominated. Number of nominations each interviewer received was also used as a measure of the strength of these dimensions (i.e., are they widely known among QA staff as funny or relatable?). Contact, refusal, refusal conversion, and cooperation rates were compared between each group. The findings are discussed in the context of nonresponse avoidance, and interviewer training and management.

Electronic Advancements in Complex Multimode Collection: Using Digital Signatures to Access Medical Records on the Medical Expenditure Panel Survey

Advances in smartphone distribution, secure remote vendor services and signature-capture capabilities provide increased alternatives to large-scale hard-copy data collection. These advances give survey researchers more efficient data collection options. The presentation examines this evolution in survey research, focusing on recent advances for medical release forms. The Agency for Healthcare Research and Quality regularly collects Authorization Forms (AFs) from household members in the Medical Expenditure Panel Survey to access their medical and billing records. Historically, these AFs have been collected on paper, scanned, and sent to medical providers electronically. Starting in 2022, respondents are encouraged to sign forms electronically within CAPI via touchscreen. Respondents unavailable at the time of interview can use DocuSign after the interview to sign the document, using either an uploaded signature file or a verified digital signature. Hard copies are offered to respondents who request them. The presentation will describe experiences with the technology as well as the efficiency and effectiveness of this transition as it may more generally apply to other studies.

Overview Of Telephone Interviewing For National Health Interview Survey, April - December 2020

In March 2020, the National Health Interview Survey transitioned to telephone-only interviewing due to the Covid-19 pandemic. The survey remained telephone-only until July 2020 when Regional Offices returned to some in-person interviewing. A vendor provided telephone numbers for approximately 60% of the survey sample each month beginning in April 2020. This research provides an overview of the frequency of first contacts and completed interviews over the phone in March - December 2020, including differences by Regional Office. Field representatives documented difficulties in reaching respondents by telephone as well as strategies used to find correct telephone numbers. The vendor-provided telephone numbers were compared to phone numbers collected from respondents during the interview for a given sample unit. Only 21.9% of sample units with phone number(s) provided by the vendor later gave a number during the interview that matched. This varied by phone type, with 58% of home phone numbers matching a number from the vendor vs only 12% of cell phone numbers. Cell phone numbers, however, were the most common phone type given by a respondent during the interview (72% of numbers).