Continuous Evaluation of Educational Experiences: Introduction to Dual Axis Rating and Media Annotation

• In Event: A Microscope for Lessons: Learning From the DARMA (Dual Axis Rating and Media Annotation) Method for Continuous Evaluation of Educational Experience

Abstract

Students (or other observers) are often asked to evaluate instruction. The validity of such measures remains controversial. Some reviews (e.g., Cohen, 1980; Marsh & Hocevar, 1984; Marsh, 2007) supporti their validity and others (e.g., Uttl, White, & Gonzalez, 2016) argue that effects are at best small and at worst may reflect publication biases. Even if one concedes the utility of course evaluations, the time scale for such evaluations is typically an entire lesson or even an entire course. However useful they may be for identifying good or poor teachers, they do not provide much information that would support the improvement of teaching itself.

In order to look at how teaching moves affect student engagement and learning, we need a paradigm that collects evidence at a more microscopic timescale. This session will describe the uses and limitations of a method for gathering continuous evaluation using readily available equipment (a joystick and a Windows PC equipped with Matlab).

DARMA (Girard & Wright, 2017) allows respondents to use a joystick to provide continuous ratings along multiple dimensions (2 in our work) of videotaped lessons. We will set up a demonstration that will allow participants to experience how to set up and use this method. Figure 1 shows an example DARMA application screen; Figure 2 shows some sample results

There are numerous potential uses of this method to understand the relation between instructor moves and student experience, some of which will be presented in other papers in this session. We could, for example, use this method to identify parts of a lesson that are confusing or otherwise need improvement, something that is otherwise difficult to identify.

Although we believe that momentary student evaluations provide useful information about student experience, the relation between such experience and overall learning may not always be a simple one. It may be the case, for example, that a lesson that elicits struggle and confusion at a strategic point may nonetheless promote engagement and ultimate learning. The DARMA method provides a way of studying these questions.

Our initial focus has been on college students, although we will be interested in participants views about extending this to younger students. Despite the use of adult students, our use of this method has shown that even experienced instructors are often surprised at how students experience the lessons they teach. Expert knowledge can at times interfere with the ability to understand novices (the so-called “Curse of Expertise”); by bringing student experience to the surface, DARMA may provide a way of overcoming this limitation.

More generally, we hope that a more microscopic focus on classroom experience will provide a way of changing our focus from evaluating teachers or lessons to a focus on the educational processes that underlie teaching and learning. That is potentially a more tractable unit for improving education as well as for understanding how the actions of teachers connect with the experiences of their students.

Author

• Kevin F. Miller, University of Michigan - Ann Arbor