Recent Experience Testing Prepay Survey Tokens with Marginalized Populations

• In Event: Threats to Validity in Impact Evaluation Research

Abstract

When surveying individuals who face barriers to accessing services, the risk of biased impact estimates increases with low overall survey response rates. This risk is also a threat if there are large differences in response rates between key research groups. Tokens, such as gift cards, are a long-established, effective method of increasing survey participation and supporting retention in the case of a longitudinal data collection. A prepay token that accompanies the survey invitation and paired with a postpay token can be a particularly effective application of incentives for surveys (Singer and Ye 2013, Mercer et al. 2015). The prepay token may encourage participation by lending credibility to the promise of a postpay token for survey completion and to the study more generally. In addition, a prepay token can foster a sense of reciprocity. Behavioral science literature suggests that the norm of reciprocity requires that we repay in kind what another has done for us; in this context, giving a potential survey respondent a small prepay token should increase their desire to reciprocate by completing the survey. 

In this paper, we discuss recent findings from prepay token experiments across multiple recent studies with populations with complex challenges, such as physical and mental health conditions, criminal justice system involvement, or limited formal work skills and experience. We designed these experiments to answer specific research questions related to their potential study benefits.   

Research question 1:  For a large, randomized impact study of employment interventions, does a prepay token of appreciation, combined with a postpaid one at survey completion: 

• Increase response rates compared to only the postpaid one?   


• Decrease response rate differentials across randomized study groups, relative to only the postpaid one?

Research question 2: For a national survey of people receiving disability benefits, does a prepay token: 

• Reduce fielding time? 


• Increase the likelihood that sample members call in to complete an interview before outbound dialing begins? 


• Increase completion rates for specific subgroups with historically lower response rates on this survey, including men and younger age groups?   

The findings across these experiments are mixed and some results are unexpected. We find that prepay tokens improve the response rate on the national disability survey by a small amount but do not show any effect within the impact study focused on employment programs. However, one program within the impact study focuses on a criminal justice population. This program experienced a negative effect of the prepay token on the response rate potentially related to disproportionate housing instability among this program’s participants. Within the national disability survey, we found a large, positive impact of prepay tokens on completion rates among men on the disability survey. We do not find any effect of prepay tokens on response rate differentials for randomized study groups.

Author

• Ryan Callahan, Mathematica
  Presenting Author