The Comparison of Interrater Reliability Estimating Techniques in Assessment of Higher Order Cognitive Skills

• In Event: Poster Session 10
  In Poster Session: Diverse Topics in Psychometrics and Educational Measurement

Abstract

The effect of variance rater number and rubric on the results obtained by the techniques used to estimate the interater reliability is examined. This research’s carries the basical type of descriptive quality. Research’s workgroup consists of 50 students and 10 teachers who rates. Ten teachers, firstly by using analytic rubric, have rated performance tasks of students related to science lesson. Agreement levels among the raters: 2, 5, 10 calculated via using kappa statistic, log-linear analysis, Krippendorff alpha to examine effect of rater number variance. Result of the research when it’s intended to get more detailed results, scores obtained by using analytic rubric that consists of subcategories, can be reached with using loglinear analysis which appropriate for the categorical data analysis.

Authors

• Ozge Bikmaz, Adnan Menderes University

• Nuri Dogan, Hacettepe University