Quantitative Methods and Results From the First Year of Alabama’s Math, Science, and Technology Initiative Study

• In Event: Evaluating Alabama’s Math, Science, and Technology Initiative: Results of a Three-Year, Statewide Randomized Experiment

Abstract

We will present the data collection, analysis methods, and results for the first year of the study. Data sources used to assess the impact of AMSTI on students and teachers included classroom rosters, student achievement and demographic data from the district and state, and web-based surveys of teachers.

State assessment data were collected each year of the study and were used as the measure of the effect of AMSTI on student achievement in mathematics, science, and reading. Four monthly (January- April) web-based surveys were deployed to AMSTI and control teachers during the first and second year of implementation. Survey data were used to measure the effect of AMSTI on classroom instructional practices, teacher content knowledge for teaching math or science, and student engagement. The effect of AMSTI on classroom practices is measured by a composite variable of teacher self-reported time using hands-on instruction, inquiry-based instruction, and instruction promoting student use of higher-order thinking skills. On each survey, teachers reported the number of minutes students spent on each of the three types of instruction during the previous two weeks of instruction. This composite “active learning” score was computed separately for mathematics and science instruction. Demographic and assessment data from the districts and state were used to estimate differential impacts for students based on their pretest level, socioeconomic status, racial/ethnic minority status, and gender.

We will present the analytic models used to estimate average and differential impacts of AMSTI on student achievement in mathematics problem solving, science and reading in grades 4-8. We will discuss the results of these analyses as well as estimates of the impact of AMSTI on teacher outcomes. We will also address briefly issues concerning achieved statistical power for detecting impacts on students.

Authors

• Andrew P. Jaciw, Empirical Education Inc.

• Boya Ma, Empirical Education Inc.