Adjusting NAEP School Weights for Teacher Nonresponse

• In Event: Feasibility of Teacher-Centric Analyses Using NAEP Data

Abstract

The purpose of this presentation is to explain how our study calculated teacher sampling weights. School sampling weights were used as the basis for calculating the teacher sampling weights, and additional adjustments were then made for teacher nonresponse. The method we employed follows the precedent employed by other NCES data sets. As an initial step we assigned each teacher a base weight equivalent to their corresponding school weight. We then determined the extent of teacher nonresponse and redistributed the weights of the nonresponding teachers among the respondents using an additive approach: summing the probability weights of the nonrespondents, dividing this sum by the number of respondents, and adding this quotient to the weights of the remaining, responding teachers. Additionally, we adjusted the weights uniformly rather than by nonresponse class. Our approach yielded weight adjustments with several positive qualities: minimal distance between the original and adjusted weights (precision), as well as representativeness and unbiasedness.
The purpose of this section is also to determine if teacher nonresponse is likely to introduce bias into analyses of NAEP data by teacher variables. This section addresses three research questions:
1. What is the extent of nonresponse among NAEP fourth grade teachers?
2. Is there substantial bias in the percentage distribution of all NAEP students in comparison to students with teacher match, by select student characteristics?
3. Are there significant differences in the percentage distribution of NAEP students with and without a teacher match, by select student characteristics?
As shown in Table 3, NAEP has an average response rate of 94% among fourth grade teachers. Given this high level of response it is unlikely that teacher nonresponse has introduced much bias into analyses of NAEP data. For example, NCES Statistical Standards only call for a nonresponse bias analysis when the response rate falls below 85%. Nevertheless, we conducted a nonresponse bias analysis as a preparatory step to making nonresponse adjustments to the teacher sampling weight.
Results of the bias analysis are presented in Table 4. All sampled students are compared to students having teacher match by students’ sex, race/ethnicity, eligibility for NSLP, disability status, and English-language learner status. Comparisons showed no significant differences for any variable except disability status. For this variable, 11.9% of all students had a disability, as opposed to only 11.8% of students having teacher match. We suspect that students with disabilities are slightly underrepresented because special education teachers were less likely to respond to the NAEP teacher survey. In this case, both the absolute bias (calculated as the difference between the estimates) and the relative bias (calculated as the absolute bias divided by the first estimate) are quite low. These results show that very little bias has been introduced into the data, even in the case of students with disability.

Author

• Jiao Yu, American Institutes for Research