Nonresponse Bias Analysis for NAEP Grade 12 Using High School Transcript Study Data

• In Event: Research Into High School Coursework and Achievement Using the NAEP High School Transcript Study

Abstract

The presentation provides results from a nonresponse bias analysis (NRBA) study for the 12th grade students selected for the National Assessment of Educational Progress (NAEP) who took the mathematics or science assessment. The analysis is for the 2019 NAEP sample and includes indicators of students’ academic achievement from the High School Transcript Study (HSTS) to evaluate the potential for bias due to nonresponse. The HSTS attempts to collect transcript data for those that graduated from high school that were selected to take the NAEP mathematics or science assessments while in 12th grade.

The NRBA study was conducted in the following steps. We first compared the characteristics of high school graduates and non-graduates to gauge whether the analysis in this report, which focuses on graduates, can be potentially generalized to non-graduates. Next, the mean NAEP assessment scores were compared to the measures of academic achievement from the HSTS to determine how closely they relate among assessed students. A high association provides more confidence in the NRBA results. We then examined whether assessed and nonrespondent students in a variety of demographic groups differ in selected HSTS measures of academic achievement. Lastly, we attempted to predict NAEP assessment scores for nonrespondents so that the amount of bias could be estimated in terms of NAEP assessment scores.

The conclusions from the NAEP NRBA that uses HSTS data are 1) differences exist in characteristics between graduates and non-graduates, and therefore, results from this study should only apply to graduates; 2) there is a strong relationship between NAEP scores and the HSTS measures, which provides confidence that HSTS measures (e.g., GPA) can serve as proxies for NAEP scores for studying the potential for nonresponse bias; 3) there are many significant differences in HSTS measures between assessed and nonrespondent graduates, overall, and by groups defined by region, type of school, race/ethnicity, and gender, however, the indications are that the potential for bias because of these differences is small; 4) nonresponse adjustments reduce the potential for nonresponse bias, but not by much; 5) while differences exist between nonrespondents and assessed students, the estimated potential for bias is small primarily due to the high student response rate; 6) the gap in the difference between assessed and nonrespondent students in GPA, Carnegie credits earned in advanced mathematics, and Carnegie credits earned in advanced science continues to widen, however, this is offset by the steady increase in the assessment rate over time; and 7) between 2009 and 2019, some HSTS measures (e.g., GPA) for excluded graduates surpassed that of assessed graduates for the first time.

The scholarly significance of this work points to the need to continue to strive for high student response rates. If the student response rate decreases, the potential for bias will increase because differences exist between assessed and nonrespondents students.

Authors

• Tom Krenzke, Westat
  Presenting Author

• Victoria Owens, Westat
  Non-Presenting Author