Can School Climate Measures Improve Predictions of Academic Outcomes? Evidence From a Large Urban District

• In Event: Leveraging Research-Practice Partnerships for Insights Into School Climate Experiences and Impacts in Racially Diverse Districts

Abstract

School climate can matter tremendously, as documented by Bryk et al. (2010) and Lee and Smith (1999), among others. This paper uses data from a state school climate survey to examine the degree to create sub-indexes along several dimensions, and then to examine the degree to which adding these measures of climate can improve pre-existing predictive models of various student outcomes.
The California Healthy Kids Survey (CHKS) is administered in grades 7, 9 and 11 throughout California. The district that we study, San Diego Unified School District, has taken the unusual step of treating the survey as a census instead of giving the survey to a small representative sample. We use CHKs census data in 2010-11, 2012-13, 2014-15 and 2015-16. Response rates are high. For instance, in 2013 87% of grade 7, 78% of grade 9, and 84% of grade 11 students participated. Student-level responses are obtained by the district, but are anonymous. Crucially, we do know each student’s school, grade level, gender and race/ethnicity.
In past work at the San Diego Education Research Alliance (SanDERA), we have estimated logit models to forecast a number of academic outcomes, such as proficiency by a given grade in math or English Language Arts, graduating on time, and enrolling in postsecondary schools. The models have proven highly accurate. Nonetheless, statistically significant differences exist across schools in the average degree of over- or under-prediction. We ask whether the school climate measures can materially improve the predictive ability of the existing models. If so the SanDERA models will be improved, and new evidence will have emerged that school climate matters for student outcomes.
The initial parts of the paper develop and validate several distinct measures of school climate. The CHKS data naturally divide into six measures related to school safety, school connectedness, school developmental supports, community development supports, home environment, and measures of drug use and alcohol use.
Overall, 96 questions which were asked consistently over time were used to construct the measures. The paper discusses the principal component analysis (including rotation methods, and reliability analysis including Cronbach’s alpha). This work, specific to the district, builds upon the work of Hanson and Kim (2007) who study the psychometric properties of parts of the CHKS.
Next, the paper correlates the climate measures with mean prediction error on a school by school basis, which in turn prompt estimation of new logit models to predict outcomes such as those mentioned earlier. In a separate analysis, new logit analyses are conducted that used the school climate variables to predict performance on California’s new (Smarter Balanced) state test.
Another section of the paper tests whether separate measures of school climate for individual gender and racial/ethnic groups, which do differ, can improve predictive validity.
The paper concludes with a summary and also discusses work that will be just underway at the time of the AERA meeting to help the host district to use the results to fine-tune its existing supports for schools and students.

Authors

• Julian Betts, University of California - San Diego

• Dina Polichar

• Andrew Zau, University of California - San Diego

• Karen Bachofer, University of California - San Diego

• Jianan Yang