Time Series Analysis of Dual Axis Rating and Media Annotation Data

• In Event: A Microscope for Lessons: Learning From the DARMA (Dual Axis Rating and Media Annotation) Method for Continuous Evaluation of Educational Experience

Abstract

DARMA produces discrete equidistant one or two-dimensional data with a user specified sampling frequency (usually every second or two), resulting in a set of time-series data. The basic idea of time-series analysis is to understand the underlying data-generating process through a parameter-parsimonious statistical model. For the purpose of our study, we analyze the dynamic of student responses over time. The basic model is based on a time-series which takes into account the relatively high autocorrelation (correlation from one measure to the next) and the fact that increase and decreases of ratings reflect changes in judgment. Mathematically, this can be expressed succinctly in a stationary ARMA model (AutoRegressive Moving Average). Stationary processes assume that means and variances over the time interval do not change systematically. This is unlikely given the fact that we assume that subjects respond differently when the stimulus (for example, teacher’s clarity) varies throughout the observational period. The stationary process is therefore the null hypothesis of mere random variability in the ratings. The moving average (MA) component can be used to model specifically how long into the process a weak teaching moment affects the ratings, i.e. how long the “shadow-effect” last, or, in the reverse case, how long a teacher can “bask in the glory” of a great moment where she had good rapport with the students. 
The autoregressive (AR) component that can be used to model the time the time series is interrupted through planned and lasting change in the time series. In one of our studies, we fully combined good/bad self-introduction of a teacher with a good/bad lecture. Of the four resulting conditions (G/G, G/B, B/G, B/B) the two change conditions (good intro with bad instruction and vice versa) are considered disruptions of the time series (“shocks”) which will weaken the autocorrelations for a certain amount of time (lags) until they become stationary again. ARMA analysis can estimate the lag that best fit the given data.
Another deviation from a stationary process that can be tested (in the form of rejecting the null hypothesis) are systematic changes in the autocorrelations of various lag-length: For example, if the assessment of the quality of the lecture is cumulative in nature, the autoregression of a given measure (t0) with its immediate predecessor (t-1) should increase as the lecture progresses, i.e. the “inertia” in the judgment should be higher towards the end of a lecture where a general opinion about the quality was formed that is increasingly difficult to change by a singular event. 
Time series analysis allow for very complex models of the data generative process which are unlikely to be useful for the data sets at hand. For example, it is not likely (although an empirical question) that after controlling for the status at (t-1) the prior states (t-2, t-3, t-4,…) will be able to further contribute to the underlying process of teaching quality assessment. Our approach will therefore be to start with a model with few parameters and add parameters that have theoretical meaning. 

Author

• Kai S. Cortina, University of Michigan