Validity Evidence for Likert/Rating Scale Instruments

• In Event: Practical Guidance and Best Practices for Gathering Validity Evidence Based on Assessment Type

Abstract

The next paper will consider validity evidence in the context of instruments employing Likert response scales. Many social science and educational researchers rely on the Likert-type item format and Likert scales (which create a summed or average scores), when constructing and measuring non-cognitive attributes (Bandalos, 2018; Harpe, 2015). Specific to these Likert-style or rating scale responses is the appropriateness of the number of response categories, which is of consequence to scoring decisions and validity evidence for the scores. Wolfe and Smith (2007a) note that “adults can rarely reliably [sic] distinguish more than seven levels of a rating scale” and that between three and five categories tend to work best with Rasch analyses. While Likert (1932) indicated that 5 categories were appropriate for agree-disagree scales, Dawes (2008) argued that comparable results were found for 7- to-10 point scales, and Revilla, Saris, and Krosnick (2014) found that a 5 point scale for agree-disagree resulted in higher quality than a 7 point scale.
This particular aspect of validity evidence for scores isn’t explicitly mentioned in the 2014 Standards. However, several analyses have previously been proposed to evaluate the performance of rating scale categories, the results of which could serve as validity evidence. The split-ballot multitrait method approach (SB-MTMM; Saris, Satorra, and Coenders, 2004), used in combination with structural equation modeling to obtain reliability, validity and quality coefficients, has been illustrated as a randomized experiment to allow comparison of the performance of the same items with a varying number of response categories (Revilla, Saris, and Krosnick, 2014). Within a Rasch framework, Linacre (2004) outlined eight criteria for evaluating rating scale effectiveness based on a single administration of an instrument, which includes the potential for recoding responses to collapse underutilized categories and reanalyzing the characteristics of the shorter rating scale.
The context for this presentation is the middle grades and high school Self-Efficacy to Teach Statistics (SETS-MG, SETS-HS) instruments. The SETS items measure statistics teaching efficacy as it relates to specific statistics topics in the Guidelines for Assessment and Instruction in Statistics Education Pre-K-12 report (GAISE; Franklin et al., 2007) and the grades 6-8 and high school standards for data analysis in the Common Core State Standards for Mathematics (CCSSM; National Governors Association Center for Best Practice & Council of Chief State School Officers, 2010). For example, item 35 refers to interpreting slope and intercept of a regression line in the context of the problem. The items employ a 6-point rating scale, ranging from “Not at all confident” to “Completely Confident”. Using Linacre’s (2004) criteria, the performance of the 6-point rating scale was assessed for the SETS-MG using a sample of 309 pre-service middle grades and for the SETS-HS using a sample of 290 pre-service secondary teachers.

Author

• Leigh M. Harrell-Williams, The University of Memphis