An Empirical Examination of Professional Learning Using the Growth Chart/ers

• In Event: Teaching Professional Practice: A Cross-Program Examination of Innovation

Abstract

Our quantitative and qualitative analyses of findings of early use of the growth chart/ers suggests the following: (a) the tools reflect development and lack of development along the competencies over time; and (b) raters generally use the tool in similar ways, suggesting a shared image of the development of professional skill and practice. In what follows, we outline our research methods and present analyses of results from our first wave of data collection.

Initial results of the program-wide use of the growth charts are the product of a multi-year pilot of the competency tools. (A sample of the growth chart data is included in Appendix A.) In a pilot from January 2014 to May 2015, each intern was rated by multiple evaluators at multiple time points. The ratings were treated as continuous variables. To ensure the models met the assumptions of statistical analysis, two residual diagnostic assessments were performed on factor scores and multilevel models. First, we generated standardized residuals for each case, and the findings revealed only one outlier. Additionally, we plotted standardized residuals. This analysis provided reasonable data to reject the possibility of heteroskedasticity and multicollinearity. We used factor analysis to further investigate the correlation and coalescence of the competency scores over time among the raters for all interns. We examined covariation using zero-order correlations (Pearson’s) and used principal-component factoring to identify latent factors. A Kaiser “eigenvalue” criterion revealed a single factor that explained a total of 87% of the variance. In addition, we interviewed evaluators, and collected observation data of competency training sessions, and analyzed intern course work, using constant comparative analysis to study the qualitative data of the competency pilot.

Findings of the statistical analyses suggest that the growth charters detect differences in interns’ development, or lack thereof, over time, thus demonstrating that the tool is useful for tracking growth. The flat lines of interns whose knowledge, skills, or practices were unchanged reveal that the evaluators took their ratings seriously and did not simply provide ratings to move interns through the program. Furthermore, independent evaluator ratings coalesced. In 25% of the ratings, evaluators’ ratings of interns were identical. 63% of evaluators’ ratings of interns were within one point of each other and 92% of evaluators’ ratings of interns were within three points, or one category, of each other. In addition, evaluators indicated that they also used the ratings over time to guide their instructional interactions with interns.

Another promising result of both quantitative and qualitative analyses is that by the end of the program, interns’ competency ratings aligned with their instructors, varying from their instructors’ scores by 1 to 2 points on average. This increased alignment between interns’ self-ratings and instructors’ ratings together with results of interview data indicate that interns learned over the course of the program to be more reflective and informed about the skills and practices necessary for high quality beginning teaching, suggests that the growth charters not only assess, but also teach, novice professionals.

Author

• Bridget Maher