Exploring Students' Spatial Thinking Strategies With Mixed-Methods Validity Techniques

• In Event: Advancing Methods for Evaluating Construct Validity in STEM Education Research

Abstract

We are engaged in the development of a spatial reasoning battery for students ages 7-14, a population for which there are few measures of spatial reasoning that have strong validity evidence. Spatial reasoning is the ability to mentally visualize, rotate, transform, represent, and recognize symbolic information. These skills are closely related to success in STEM careers in adulthood (Wai et al., 2010), but have also been connected to early mathematics learning (Woolcott, 2022). Differences in spatial reasoning have also been implicated in understanding why girls are less likely to pursue some STEM domains. The goal of this paper is to share a range of validity research methods we are using to build a strong validity argument and establish fairness evidence for a new measure of spatial reasoning.

Framework
A variety of qualitative and quantitative methods are available for developing the construct validity argument for a new assessment tool, including think aloud protocols, eye-tracking during problem solving, and psychometric analyses of items (Embretson, 2016; Embretson & Gorin, 2001). This paper particularly focuses on data from think aloud studies and eye-tracking that can provide valuable insight into examinees’ cognitive processes with a relatively low sample size. Think aloud, in particular, is valuable for uncovering differences in response processes that may be due to bias in item development as well as differences in solution strategies.

Methods
Leighton (2017) laid out the principles of think-aloud interviews as part of a construct validity argument. Participants from the target population are asked to solve representative problems while verbally reporting the processes they use to solve the test questions. While there are limitations to self-report, it provides insight into cognitive processes.

Eye-tracking has made great strides recently with the advent of more precise eye-tracking and statistical analyses that capture the movement of the examinees’ eyes while solving test problems. Although very complex models of eye fixations and traces are possible, in this paper we provide results from a qualitative analysis of eye traces, comparing the pattern of eye gaze for participants with high and low spatial ability.

Data
We have collected eye-tracking and think aloud interview data from 20 children ages 7 to 14. We are also collecting data from reference spatial tests for convergent validity.

Results
We are currently analyzing the think aloud data, which results in qualitative interview data that must be coded systematically. Our initial findings suggest that eye-tracking combined with think aloud provides unique insights into the cognitive processes that children use in solving the spatial items with great potential for exploring test fairness.

Significance
Beyond the field of measurement, validity theory and methods have evolved little in the last few decades, despite the increasing reliance on assessment data to inform education, particularly personalized learning. This paper will provide an example of how well-established validity methods from the measurement field can directly inform the work of researchers focused on STEM learning outcomes.

Authors

• Joni M. Lakin, University of Alabama
  Presenting Author

• Kaiwen Man, University of Alabama
  Non-Presenting Author

• Qingzhou Shi, University of Alabama
  Non-Presenting Author

• Jacob Feiler, University of Alabama
  Non-Presenting Author