Using Evaluation Results to Inform the Design and Effectiveness of Professional Development Learning Experiences

• In Event: Formative Intervention Research to Enhance Equitable Mathematics Teaching: Lesson Learned From Multiple Data Sources

Abstract

Purpose. This paper describes how evaluation results from teachers and members of the project’s leadership team in the initial stage of a project can be used to inform the design and implementation of future professional development in remaining years. Specifically, questions for investigation are:
1) What was the nature and quality of the professional development (PD) initiative and participants’ engagement in the professional learning experiences during Years 1 and 2?
2) What were teachers’ reactions to the PD? What was perceived as helpful or not helpful? In what ways did they expect the PD would impact their instruction?
3) What was learned from leadership team members regarding monthly project planning meetings?
4) How were results from the above used to inform and modify future PD experiences?
Theoretical Framework. Two frameworks guided the work. First, it adheres to the evaluation guidelines from NSF and the American Evaluation Association surrounding the levels of formative evaluation (implementation, monitoring, and progress). Secondly, Desimone’s (2009) model of teachers’ learning and instructional change describes a framework for how PD experiences generate change in teachers’ knowledge/beliefs, which enhance teachers’ instructional practices, and subsequently enhance students’ mathematical learning. The PD is the foundation of this model.
Methodology and Data Sources. A mixed methodology was employed, quantitative and qualitative data was collected based on the data source. Surveys were administered to leadership team members (i.e., Principal Investigators in the project) to obtain feedback regarding the focus of monthly meetings and how to best allocate time in future meetings. Teachers also responded to surveys at several times during the annual four-week summer PD sessions.
Other forms of data include: participation tallies during PD sessions, attendance at project meetings/events, document analysis (e.g., PD session handouts, presentations, questions for discussion, exhibits of teachers’ writing/work generated), and videos of sessions.
Results. Teacher survey results for Year 1 sessions indicated that while the focus on mathematics was important, they wanted more time to make connections to their curriculum. Teachers also hoped to learn about equity in the classroom, but said it was lacking in the sessions. Others indicated that readings were difficult, sometimes “pretentious”, and hard to follow. Representatives from the mathematics and school-based components of the project will expand on how the evaluation results were used to inform planning of Year 2 summer PD. Preliminary survey results and observations of sessions indicate the PD was well-received by teachers and their engagement with the content and speakers was at a higher level than the previous year.
Scholarly Significance. This paper describes one of three aspects of the project’s external evaluation. In conjunction with results from the others (change in instructional practice and student achievement), it provides valuable formative feedback to the leadership team and teachers regarding the remaining project years. However, the purposes of sharing results goes beyond determining the effectiveness of the initiative. It contributes to discussions of using multiple sources of evidence to answer questions and demonstrates the importance of attending to the implementation, monitoring, and progress levels of formative evaluation.

Authors

• Carol S. Parke, Duquesne University

• Jayna Bonfini