Score and Item Analyses

• In Event: In-Depth Analysis of Eighth-Grade Students Performing Below Basic in the National Assessment of Educational Progress

Abstract

This presentation explains item and subscale score analyses of students with below Basic performance. The first half of the presentation covers a broad overview of score distribution of students with below Basic performance, mathematics sub-scale analysis by achievement level, and item analyses in selected multiple and constructed response items. The second half of the presentation focuses on logistic regression analyses showing patterns with special attention for practitioners and policy makers.

Figure 3 shows the NAEP mathematics score distribution of students with below Basic performance, which cut point is 262, in 2015 and 2013. It is observed that the score distribution pattern is similar between these years. The average scale score of all below Basic students is 238 in both assessment years. Although there are many students just below the cut point, the tail of the score distribution reached towards around 106 points.

Table 6 shows the results of the overall and subscale mathematics scores of 8th grade students who performed below Basic vs. at or above Basic. The overall average score of students with below Basic performance was 238, whereas that of students with at or above Basic was 300. Among students with below Basic performance, the highest subscale score of the five was 243 in Geometry and Algebra, and the lowest was 226 in Measurement. The net overall score difference between students below Basic and at or above Basic was 62 points or 1.68 of a standard deviation. The largest subscale score gap was 74 in Measurement, but the difference expressed in standard deviation units was similar across subscales yet slightly higher in Algebra (1.62).

For the item analysis, only the released items to the public were used in order to maintain the confidentiality. Among 47 math question items in 2013 released on the NAEP Questions Tool for grade 8, we focused only on items which were categorized as easy or medium, since difficult items are clearly beyond the ability of students who performed below Basic and therefore will provide little information of what they can or cannot do. From those, this presentation uses four sample items (see appendix I for more details). Table 7 shows the proportion of 8th-grade students by achievement level who correctly and incorrectly answered three multiple-choice question items and who correctly, partially, and incorrectly answered the one short-constructed-response question item. Figure 4 highlights the differences in the percentage answering correctly or receiving full credit across items and achievement levels.

Lastly, logistic regression results will be presented exploring factors associated with below Basic performance, after controlling for socio-economic status. Based on the descriptive statistics shown in this session combined with the logistic regression analyses, this presentation will contribute to further discussions on the ways in which we can support student groups most at risk of below Basic and students those who clustered just below the cut point, as well as on specific content areas, item types, teacher characteristics, or teaching practices that may be associated with improvement of students performing below Basic.

Author

• Sakiko Ikoma, American Institutes for Research