Measuring the Impact of Educative Curriculum Materials on Teacher Knowledge/Efficacy and English Language Learner Student Learning

• In Event: The Role of Educative Curriculum Materials in Supporting Science Teaching Practices With English Language Learners

Abstract

Objective
This paper reports on measures developed to investigate the impact of educative curricular features on teachers’ knowledge of strategies and sense of efficacy working with elementary aged English language learners (ELL). In addition it summarizes the validity and reliability of measures of two key student outcomes for this project.

Theoretical Perspective
To evaluate the effectiveness of the intervention, scores on measures designed for teachers and students must be analyzed for their reliability and validity. This presentation summarizes design strategies and results for this sample of teacher and student measures.

Method and Data Sources
Development of the project’s measures included extensive literature reviews, consultation with content experts, progress variable development, cognitive labs with teacher and student participants as well as pilots of draft versions of the measures. A combination of Classical Test Theory (CTT) and Item Response Theory (IRT) approaches was used to evaluate the functioning of the project’s teacher knowledge and efficacy of teachers (n = 419 teachers) and student knowledge outcomes (n = 501 students). After investigating the measures’ difficulty levels and internal consistency, we investigated the dimensionality of each using factor analytic approaches and consequently selected the most appropriate psychometric models to continue the analysis. IRT approaches were used to examine item difficulty, item fit, and separation reliability for each measure.

Results
Teacher Knowledge Measure: Four subscales make up the measure of teachers’ knowledge of strategies for teaching elementary English language learners. These include knowledge of strategies for developing students’ 1) science content vocabulary, 2) comprehension of informational text, 3) writing in the context of science and 4) developing students’ scientific discourse abilities. The measure is comprised of 24 multiple-choice items. The mean score on the measure was 14.35 items correct out of 24, with a standard deviation of 3.94. Internal reliability of the measure is .69, using Chronbach’s alpha. Person separation reliability and fit statistics for each item fall within acceptable parameters.

Teacher Efficacy Measure: This measure consists of four subscales: 1) planning and developing ELL sensitive instruction, 2) supporting teachers in meeting ELL needs, 3) using primary language as a resource, and 4) openness to new practices for supporting ELL students. Twenty-five items utilized a 5-point Likert-scale ranging from “Strongly Disagree” (1) to “Strongly Agree” (5). The mean response was 3.3 with a standard deviation of .609. Internal reliability of the measure is .93 using Chronbach’s alpha. Person separation reliability and fit statistics for each item fall within acceptable parameters.

Student Measures
Two student measures were of primary interest: vocabulary knowledge and science knowledge. Similar procedures were used to study reliability and validity of these measures. Given the validity analysis, the science knowledge measure consisted of two subscales; the vocabulary measure was uni-dimensional. Alpha reliabilities were: vocabulary (.86); science knowledge (.66 for both subscales). Person separation reliability and fit statistics for each item fall within acceptable parameters.

Scholarly Significance
While this collection of measures is in relatively early phases of diffusion, they stand to contribute to research, development and evaluation efforts of any such program.

Authors

• Seth Corrigan, Lawrence Hall of Science, University of California - Berkeley

• Whitney Alicia Zimmerman, The Pennsylvania State University