The Impact of Rigor: Changes in the Evaluation Requirements for Office of English Language Acquisition National Professional Development Grants

• In Event: 63.077 - Office of English Language Acquisition–National Professional Development Grants: Designing Innovative Professional Development Programs for Teachers of English Learners

Abstract

How does the requirement of a rigorous evaluation change the program being evaluated? This paper examines the perceived impact of new evaluation requirements of the Office of English Language Acquisition (OELA) National Professional Development (NPD) grants from two points of view—from the point of view of university faculty who design, administer, and teach that program, and from evaluators tasked with designing and conducting rigorous evaluations of their programs.

In the 2016 request for proposals, OELA NPD grantees are now required to conduct a rigorous evaluation that meets the What Works Clearinghouse (WWC) standards with reservations. This means that in addition to collecting formative evaluation data, and the required Government Performance and Results Act (GPRA) measures, grantees must evaluate the impact of their program through either randomized controlled trial (RCT) or a quasi-experimental design (QED) with sufficient sample to generate the necessary statistical power.

Previous research on English learners (ELs) for WWC have focused on English proficiency, reading, or math assessment outcomes, and have generally shown small or limited effects (Bos, et al., 2012; Deussen, Autio, Roccograndi, & Hanita, 2014; Loeb, Soland, & Fox, 2014). However, OELA NPD grant-funded projects tend to target a broad set of instructional practices and beliefs that are not easily measured through standardized assessments. Many programs emphasize developing teacher understanding of the political and social context for EL students. Programs also emphasize developing an understanding of language acquisition, which teachers report enhances their understanding of their EL students, and their capacity to meet their needs. However, currently there are few measures of how teacher beliefs, expectations, and advocacy impact their instruction or student achievement. Furthermore, estimating the effect of EL-focused professional development on teacher practice is complicated by the fact that well-validated classroom observation protocols were not developed to focus on practices shown to be beneficial for ELs (WWC, 2016). OELA NPD evaluations nonetheless must dedicate extensive resources to measuring outcomes that, while important, may not capture the key issues the programs emphasize.

This paper will explore how university faculty and external evaluators have navigated these tensions to develop an evaluation plan that meets the program needs and evaluation requirements. Specifically, this paper examines the how these new evaluation requirements have affected two NPD-funded programs. We find that university faculty members who design, administer, and teach the NPD-funded program find that the new evaluation requirements may limit attention to aspects of the program that are not easily quantified. For example, requirements around student-level impact do not necessarily facilitate learning about what teachers take from community engagement work, which is a priority under the grant guidelines. Similarly, evaluators conducting a rigorous evaluation of a teacher professional development program find that the needs of the evaluation may run counter to the goals of the program. For example, evaluators may prioritize different teacher candidates to receive funding, excluding teachers who may not be directly linked to measurable student outcomes on state assessments (such as special education teachers).

Authors

• Jason Greenberg Motamedi, Education Northwest

• Mary Martinez-Wenzl, Education Northwest

• Karen D. Thompson, Oregon State University

• Claudia Peralta, Boise State University