Effects of First Impressions and Instructional Quality on Student Engagement: A Continuous Evaluation

• In Event: A Microscope for Lessons: Learning From the DARMA (Dual Axis Rating and Media Annotation) Method for Continuous Evaluation of Educational Experience

Abstract

Previous research has shown that student course evaluations can be predicted by from 30-second clips from the first day of college classes (Ambady & Rosenthal, 1993). There are two plausible interpretations of that result – 1) First impressions may provide valid evidence for instructional quality, or 2) First impressions may color student experience of instruction regardless of actual lesson quality. Previous research in our lab (Authors, 2016) used scripted lessons to control the relation between first impression and instructional quality, and showed that effects of first impression were much smaller than those of overall instructional quality in determining both student learning and evaluations.
In the current study we used these materials to validate a new method for collecting continuous student evaluations (DARMA - Girard & Wright, 2017). DARMA uses a joystick to collect continuous responses on multiple dimensions. Students watched a videotaped lecture and used the joystick to evaluate the quality of the instructor and their engagement with the lesson.
Coupling the DARMA method with scripted lectures allowed us to consider the following questions:
1) Does DARMA reveal differences between good and bad first impressions and lectures?
2) Does DARMA reveal transient effects of first impression that go away across the course of a lesson?
3) Does DARMA reveal any new phenomena that did not appear in the prior study?
Methods
Participants were 197 undergraduate students (103 male, 94 female) enrolled in an introductory psychology course at a large Midwestern University in the United States. Students watched one of four scripted lessons that cross good/bad first impressions with good/bad instruction, and participants were randomly assigned to one of the four conditions (2X2 experimental design (First Impression x Instruction)). There were 27-30 students in each condition.
Results/Discussion
Figure 1 shows results for students who saw the Good instruction videos. Comparing the top and lower panels shows the large but transient effect of the First impression the instructor made. Figure 2 shows results for students who saw the Bad instruction videos. As expected, the effects of First impression were the same as for the other groups, but the effects were again very transient. Table 1 shows results of a two-way ANOVA of overall effects of first impression and instructional quality. First impression was significantly related to later ratings of Anxiety, while Instruction was related to all final ratings as well as to performance on a posttest of student learning.
Instructional quality also had a significant main effect on participants’ instructor evaluations. Compared to participants in the bad instruction condition, participants in the good instruction condition rated the instructor significantly higher on all of the perceived teacher personality.
Examination of the DARMA results revealed systematic changes in evaluation that surprised the instructors. For example, a graph showing PISA results from many countries, even though well explained, led to a massive drop in engagement that took some time to resolve. This result supports the view that DARMA may help us discover aspects of student experience of education that are otherwise hard to identify.

Authors

• Inah Park, University of Michigan

• Yizhen Huang, University of Michigan - Ann Arbor

• Xin Sun, University of Michigan - Ann Arbor