Applying the Many-Faceted Rasch Measurement Model to Explore Reviewer Ratings of AERA Annual Conference Proposals

• In Event: 60.080 - Measurement Issues in Survey Research

Abstract

Conference proposals are submitted each year to The American Educational Research Association (AERA) for their annual meeting. Proposals are rated by researchers that have shared interest and expertise in the area under consideration using a 5-point scale on identified criterion. Given the multiple and varied reviewers, an analysis of psychometric properties like rater severity and consistency are important. The many-faceted Rasch measurement model (MFRM) is applied to a 2009 set of scored proposals. Results indicate some rater leniency and a notable spread of raters. There is also potential multicollinearity as three of the rating criterion are measuring at the same point. In acceptance, the most important rating factor seems to be the ability of findings to generate interest.

Authors

• Kelly D. Bradley, University of Kentucky

• Michael Peabody, American Board of Family Medicine

• Richard Kweku Mensah, University of Kentucky