Evaluation and Modification of Formulas for Discussion Quality Score

• In Event: AERA Roundtable Session Saturday 11:45 am - Gold 4
  In Roundtable Session: Evaluating and Validating Measures of Student Learning and Performance

Abstract

This study evaluated the effectiveness of the Quality Talk (QT) discussion formula – specifically, whether the discussion quality scores it generated align with QT experts’ evaluations. Correlations between expert ratings and formula-generated scores (produced by applying QT codes to the formula) were analyzed across 30 discussions. A revised formula, which calculated teacher and student contributions separately and combined them into a single score, was also examined. Results show positive correlations between expert scores and the original formula scores, suggesting that the formula reasonably reflects expert judgments. In contrast, the revised formula scores show no clear association with expert scores. Directions for future refinement of the formula were discussed.

Authors

• Chien-Chang Pan, Pennsylvania State University

• P. Karen Murphy, Pennsylvania State University

• Carla M. Firetto, Arizona State University

• Jeff A. Greene, University of North Carolina - Chapel Hill

• Rachel Miriam Vriend Croninger, Pennsylvania State University