Practical Measurement in Action: Designing a New Measure of Community College Student Motivation and Engagement

• In Event: Bridging Research and Practice With “Practical Measurement”: The Case of Improving Developmental Mathematics in Community Colleges

Abstract

The goal of this presentation is to explain the design process of a “practical” community college student motivation and engagement instrument. One challenge in developing the measure was the potential translation problem between academic language and real-person language in regards to interpreting the theoretical constructs being measured. Second, many of the survey items that were collected from the theoretical literature tended to be lengthy, ambiguous, and not easily interpretable. Third, most survey items privileged internal consistency reliability over validity. Fourth, most surveys tended to rely on agree / disagree scales, which have well-known but often ignored methodological flaws (e.g., Saris, Revilla, Krosnick & Shaeffer, 2010). Using qualitative interviews and best-practices survey design, all of these difficulties were overcome. In doing so, we challenge several myths about optimal survey design. Most specifically, we argue that, with strong theory and pre-testing, long, arduous and ambiguous measures can be reduced to clear and efficiently-administered surveys that resonate with practitioners, without substantially sacrificing validity.

Our survey is unique in that it started with a driver diagram of key problems facing students and causing the problematic outcome (see Figure 1). All potential drivers were based on interviews with faculty, students, and other community college practitioners in addition to an extensive review of the theoretical literature. Then we located measures of the drivers from the literature and via interviews with community college faculty. Hence, the construction was derived from both intuitive and a scientific theory underpinnings that grew both out of the research literature and interviews with students/ faculty. We conducted cognitive pre-testing on the items with students and re-wrote items to be construct-specific instead of using agree/disagree options.

Methods:
We initially took 20-50 item scales and reduced them to 1, 2, 3 or 4 item scales by privileging that were theoretically, most essential and empirically, the highest-loading. These items were then re-written to match community college students’ construals of the phenomena.

Data sources:
The items for the first primary driver were based on constructs related to students’ perceptions of interest and relevance, long-term aims, and choice, which has been shown to promote intrinsic motivation for learning. The items for the second primary driver were based on constructs related to students’ self-reported use of time and goal-management, knowing how to distill information, calculate their grade/performance in a course, understand course expectations, know how to ask for help, and there use of self-regulated learning and strategies for coping with stress or anxiety. Items for the third primary driver were based on students’ perceptions of having a growth mindset, attributions, and a positive and realistic academic identity. The items for the fourth primary driver were based on students sense of belonging for contending with racial stereotypes and perceptions of high standards and assurance.

Figure 1: Driver Diagram for Student Motivation and Engagement

Author

• Laura Graciela Costello, SEDL