How Do Individual or Group Guidelines Impact Calibration Accuracy and Achievement of High School Students in a U.S. Governance Course?

• In Event: 28.023 - 21st Century Calibration: Judging Performance in Digital and Classroom Contexts

Abstract

Objectives
This study is a partial replication of previous research (Bol, Hacker, Walck, & Nunnery, 2012) conducted to determine whether guidelines in group or individual settings impacted calibration accuracy and achievement of high school biology students. We found that calibration with guidelines, in both individual and group settings, resulted in better calibration accuracy and achievement, as compared to a control group that did not receive guidelines.
In this study, we focused on these same variables but in a high school US Governance course, comparing scores on multiple choice versus short answer items, over the course of 5 unit exams.

Theoretical Framework
Group settings may provide an ideal context for the fostering of metacognitive skills. Group activities are easily incorporated into the classroom and not only benefit the student metacognitively, but allow the student to learn from his or her peers. Research suggests that group work can promote metacognitive skills and judgments (Hadwin & Jarvella, 2011).
The use of guidelines during calibration can enhance metacognitive skills and learning by prompting students to focus their cognitive processes on the task at hand and evaluate their mastery of the material. Previous studies have shown that guidelines, prompts, or questions can improve metacognitive skills (Azevedo, 2018).

Methods
A quasi experimental design was conducted. Two 12th grade classes were randomly assigned to either the group (n=21) or individual (n=20) guideline conditions. The calibration activities occurred in the class period before the exams. After about 40 minutes of review, students were given 10 minutes to respond to guideline questions and make their predictions of scores on both the multiple choice and short answer items. Guidelines asked them to identify what content they did and did not understand well.
Data sources were calibration accuracy and achievement. Calibration accuracy was determined by comparing predicted and postdicted scores with the actual test score. Absolute values were used in analyses, where lower scores represented better accuracy. Achievement was students’ scores on multiple choice and short answer items on the five unit exams.

Results
There were no significant differences between conditions on the unit exam before treatment. After, ANCOVA results (F 1, 38 = 4.57, p. = .03, ƞ2 = .12) revealed that students in the individual condition had significantly higher scores on the 5th exam (M = 74%) than those in the group condition (M = 67%). A similar trend was found for open-ended ended items, but the difference was not significant.
Overall, students had good calibration accuracy and were underconfident. There were no significant differences by condition. Mean predictions on multiple choice items were 71% compared to actual scores of 76%, and on short answer items these mean percentages were 70% versus 75%, respectively. Postdictions were better, especially on short answer items (72% versus 75%).

Scholarly Significance
The results partially replicated Bol et al.’s findings (2012). Guidelines were helpful for students in the individual condition on multiple choice items. The effectiveness of guidelines and other interventions differed depending on item type and achievement level (Bol & Hacker, 2001; 2012).

Authors

• Linda Bol, Old Dominion University

• Christopher M. Fischer, Old Dominion University

• Melissa Quackenbush, Old Dominion University