Analysis of a Next-Generation Quantitative Literacy Assessment: Results From a Pilot Test

• In Event: Developing and Evaluating the Psychometric Quality of Next-Generation Student Learning Outcomes Assessments

Abstract

Theoretical Framework and Purpose

Quantitative literacy (QL) has been defined as the ability to interpret and communicate numbers and mathematical information throughout everyday life (e.g., OECD, 2012; Steen, 2001). Unlike traditional mathematics, QL is a “habit of mind” (Rhodes, 2010; Steen, 2001), focusing on certainty and data from the empirical world (Steen, 2001). The importance of QL has been recognized by both the higher education (AAC&U, 2011) and workforce communities (Hart Research Associates, 2013). Previous research has shown that many students are underprepared to use quantitative skills in the workforce (McKinsey and Company, 2013). Therefore, there is a critical need to take action to delineate the various components underlying QL, and create quality assessments to identify students’ strengths and weaknesses. Using the assessment theoretical framework developed by Roohr, Graf, and Liu (2014), we developed a next-generation QL assessment and piloted the assessment with 1,560 undergraduate students across 23 institutions in the United States. The purpose of this study was to analyze the pilot data to provide validity evidence for the use of the assessment at higher education institutions. We focused on the following research questions:
1. Does the QL assessment measure the dimensions specified in the theoretical framework?
2. Does the QL assessment produce reliable scores for institution-level reporting?
3. Are there significant cross-sectional learning gains in scores between freshmen and seniors?
4. Are there significant relationships between college-level variables and QL scores?

Methods and Preliminary Results

Pilot data were collected between March and April 2015. Six forms of a newly developed QL assessment were administered to all students. This assessment was developed based on two domains: problem-solving skills and mathematical content. Problem-solving skills included four subdomains: (a) communication, (b) interpretation, (c) modeling, and (d) strategic knowledge and reasoning. Mathematical content area also included four subdomains: (a) number and operations, (b) algebra, (c) geometry and measurement, and (d) statistics and probability.

Preliminary findings using a confirmatory factor analysis showed that the one-factor model best fit the data, meaning that the assessment was undimensional and measured one overall construct of QL. In terms of institution-level reliability across test forms, total score reliability was .99, and subscore reliabilities were all above .95 (see Table 1). For cross-sectional learning gains, results showed that after controlling for college admissions score, seniors performed significantly higher as compared to freshmen (5.4 points on a 0-100 scale). QL scores also showed significant relationships with high school GPA and the number of quantitative courses taken in college.

Importance of Study

Results from this study provide validity evidence for the use of QL scores from a next-generation assessment at higher education institutions, and provide support for moving forward with an operational assessment. The approach used to evaluate this QL assessment can be used by institutions planning to develop their own SLO assessment measuring QL. This approach also has great potential to inform institutions when they make decisions about SLO assessments.

Authors

• Katrina Crotts Roohr, Educational Testing Service

• HyeSun Lee, University of Nebraska - Lincoln

• Jun Xu, Educational Testing Service

• Ou Lydia Liu, ETS