Paper Summary
Share...

Direct link:

Comparative Analyses of Generalizability Theory and the Many-Facet Rasch Model

Sun, April 15, 12:25 to 1:55pm, Marriott Pinnacle, Floor: Third Level, Pinnacle I

Abstract

Objectives

The purpose of this study is to describe methods for examining the quality of ratings obtained on Advanced Placement (AP) exams. All of these exams include constructed response (CR) tasks that require scoring by raters. Given the importance, visibility, and influence of the AP program, it is essential that the ratings meet a high level of psychometric quality. This study is guided by the following research questions:

1. What do Generalizability Theory and Rasch Measurement Theory reveal about the psychometric quality of two AP exams included in this study?

2. Are the results based on the two different theoretical perspectives complementary or contradictory in their evaluation of the two AP exams?

3. How can the results from these analyses serve to inform and improve rater-mediated assessments within the context of AP exams?

Theoretical Framework

Both Generalizability Theory (implemented with GENOVA) and Many Facets Rasch model (MFRM; implemented with the Facets computer program) are used to assess and compare the rater functioning across tasks on two AP exams. Several earlier studies (Engelhard and Myford, 2003; Wolfe, Myford, Engelhard, and Manalo, 2007) examined features of the AP exam processes with MFRMs. In addition, rater consistency is regularly assessed for AP exams using Generalizability Theory. This study compares and contrasts the results from both approaches, and it provides a rich picture of how to evaluate the quality of the ratings within these two measurement theories. Data from current reader-reliability studies are used to illustrate a multi-methods approach for analyzing ratings on AP exams.

Methods

This study focuses on two AP exams: Statistics (six tasks) and English Language and Composition (three tasks). These exams include different types of rater-mediated tasks, such as mathematical word problems versus essays. The level of rater consistency, thus, would be expected to be quite different across these exams. The data will be analyzed with Generalizability Theory both at an item level, using a person by rater design, and for the entire constructed response section, using the person by raters nested within tasks design.

Results/Conclusions

The Statistics results are complete, and preliminary findings are presented in tables and figures in the uploaded document. See initial MFRM results under the description of the first study. The GENOVA results indicate that rater reliability does vary by item, but is relatively high, even for one rater, as indicated in Table 2. Table 3 shows that reliability across the complete constructed response section, however, is quite low (phi=.524) even though the exam includes six constructed response tasks. The results show that the use of more raters will only minimally improve the reliability, but that the number of items on the exam has a more substantial effect (see Figure 1).

Scientific Significance

This study compares and contrasts the results from two approaches and provides a rich picture of how to evaluate the quality of the ratings within these two measurement theories. This research helps inform best practices in exam development, rater training, and scoring methodologies for large-scale rater-mediated assessments.

Authors