Individual Submission Summary
Share...

Direct link:

A Regression Discontinuity Approach to Find a Fluency Benchmark for Second Grade in Spanish in Guatemala

Mon, April 15, 3:15 to 4:45pm, Hyatt Regency, Floor: Bay (Level 1), Bayview A/B Foyers

Proposal

Introduction

The educational accountability system of Guatemala is teacher- and school-centered, low stakes, and based on large-scaled assessments. Accordingly, The Ministry of Education has tested reading ability nationally in benchmark grades (first, third, sixth, ninth and high school) using national standardized tests, including EGRA, and results have been compared yearly ever since. This is known in the literature as achievement or status. However, recent research found that such reporting model does not show the possibility that a school could be improving its results, even though its students don’t achieve the standard due to variables outside the school’s control (Betebenner, 2009). Furthermore, yearly status reports do not show the connection nor the alignment between benchmark grades. Consequently, since 2014, USAID Lifelong Learning has supported the MoE in measuring reading ability of the same cohort of students through consecutive grade levels. This is known in the literature as growth modeling. Shifting from the idea that learning to read is a one-grade shot to the idea that learning to read constitutes a continuum of skills that grow in complexity and within grades, had implications to the approaches chosen to find benchmarks and standards that can communicate to teachers, in a transparent way, a minimum score for expected student progress in reading between grade levels of early elementary school.
Within this context of interpreting growth results, the purpose of this research was to find a fluency benchmark for second grade in Spanish as a second language using a regression discontinuity method. The approach of standard setting taken in this research presupposes that a benchmark established for one skill should impact a more complex skill within the continuum of developing a complex ability such as reading. Thus, in the learning to read trajectory, simpler skills, such as decoding will impact or predict further skills such as fluency. Similarly, fluency would impact reading comprehension. If learning to read is conceived as a trajectory in which students master different skills while progressing through primary school, then it is optimal to have both standards and benchmarks in the continuum to orient teacher practice. Moreover, in the bilingual contexts of Guatemala, transitioning of reading skills from the mother tongue to a second language occurs in second grade. Thus, it is imperative to have benchmarks of fluency in this grade level. Although, fluency cannot be taken as causal of comprehension, it is considered to be a prerequisite of comprehension. In this research we demonstrate that fluency is a prerequisite skill to achieve the educational accountability outcome of reading comprehension and identify how to establish a grade-level standard.
Method
To answer determine the minimum fluency required in Guatemala, a regression discontinuity approach was used. The basic regression equation used in this study was the following:

THETA=β_0+ β_1 (CFLUENCY)+ β_2 (Benchmark)

In this model, theta corresponds to the reading comprehension ability estimations from the results of national reading test for second or third grade. The estimation is done using Item Response Theory and the Rasch model. Furthermore, according to Murnane and Willet (2011), in this regression model, parameters β_1 and β_2 represent the main effects of the fluency and the benchmark established. Thus, the β_2 addresses our research question of trying to estimate the minimum fluency required in Spanish to impact reading comprehension outcome in Second Grade in Guatemala.
Data from an USAID funded longitudinal study conducted from 2015 to 2018 was used. Students participating in this research responded to reading tests at the end of each grade level from first to third grade. Grade level reading tests placed different weight on reading skills. Per the data collection design, students took EGRA at the end of first grade and second grade. In addition, second and third grade students took the national reading comprehension exams. Subsequently, to determine the fluency cut-point, scores from students that had taken both the fluency test in second grade (EGRA) and the second-grade comprehension test were used. The procedure was replicated with those students that had taken the fluency test in second grade (EGRA) and had taken the third-grade comprehension test.
Using a regression discontinuity analysis, different cut-points were tried out until the cut-point that predicted achievement scores of the second-grade comprehension test was statistically significant. To be specific, words per minute score of the EGRA test were contrasted to the ability estimated in the second-grade test.
Results
After regressing different wpm scores, the cutoff point that significantly predicted comprehension in second grade was 80 wpm: β_2 = 0.528, t (1105) = 6.665, p < .000. The model explained theta scores of the second-grade test significantly, R2 = .066, F (2, 1103) = 39.133, p < .000.
After regressing different wpm scores, the cut-point that significantly predicted comprehension in third grade was 70 wpm by the end of second grade. β_2 = 2.119, t (300) = 2.119, p < 0.035. The model explained theta scores of the second-grade test significantly, R2 = .066, F (2, 298) = 39.133, p < .000.
Through a regression discontinuity model, it was possible to determine the standard of reading fluency for second grade. The standard corresponds to 70-80 wpm in second and third grades. As with any standard setting procedure, it is important to evaluate the consistency and validity of the interpretative argument in which the standard established will be used. In terms of consistency with external criteria the standard turns out to be consistent with the standard found in Mexico for Spanish speaking populations (Abadzi, 2012). However, to enhance the consistency and validity of interpretation of results replications of the methodology with various cohorts of students and benchmark confidence intervals of words per minute per grade level will be presented. Finally, results of the inclusion of oral comprehension, another prerequisite variable in bilingual settings, will be explained in the presentation.

Authors