Analysis of Student Achievement Growth Using Equating in Item Response Theory: A Case of Malawi

• In Event: Early childhood/primary ed

Proposal

Improving student achievement is a significant issue in most of developing countries. The Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ) III showed that grade 6 student achievement in Malawi is the bottom level among the 15 participating countries. Although the government enhances student achievement in the primary education level, it is still low.
There are a few previous studies on measuring student achievement in Malawi. However, all studies measure student achievement at one particular time. A more significant evaluation is achievement growth. It estimates growth by measuring student achievement at several points in time. It is concerned with trends in school performance over time, patterns of school improvement over time, teacher- and school-level factors that enhance achievement, and teacher qualities that raise achievement after controlling for student-, class-, and school-level factors. Therefore, this study aims to estimate student achievement growth in primary school in Malawi by using equating in item response theory (IRT).
To estimate achievement growth, it is necessary to use item repose theory . It is a theory of testing based on the relationship between individuals’ performances on a test item and the test takers’ levels of performance on a measure of the ability that item was designed to measure. In IRT, there is a process of placing scores from two or more tests into a common score scale. It is called equating. This allows to estimate achievement growth, which we conduct different tests in different time.
We selected 30 public primary schools in the Nkhata Bay District in Malawi as a sample. We used a cluster-sampling method with the clustering as the national examination pass rate for 2010-2012. We conducted the fieldwork in twice: January to March in 2013 and during the same period in 2014. In 2013, we administered the tests to all grades 5 and 7 students in the selected schools. In 2014, we conducted the tests to those who promoted to grades 6 and 8 of the students who wrote the tests in 2013. As a result, we collected data from 659 grade 5 and 507 grade 7 students.
We designed the framework of the tests to measure vocabulary, grammar, conversation and reading comprehension in English, and counting and writing numbers, addition, subtraction, multiplication, division, shapes, patterns, fractions and graphs in mathematics. We developed English and mathematics tests in both 2013 and 2014 with reference to curricula, textbooks, zonal and national examinations, and previous studies that showed results of the Monitoring Achievement in Lower Primary and SACMEQ assessments. The English and mathematics tests for grade 5 each consisted of 35 question items and for grade 7 each consisted of 40 question items. To enable to conduct equating in IRT, we designed the tests in 2013 and 2014 as a particular way. We included the same question items to the two test forms: 16 question items for English and mathematics tests in grades5 and 6, and 19 and 15 question items for English and mathematics tests, respectively, in grades 7 and 8. Then, we estimated item parameters for each test forms separately and item parameters on the common scales were computed from the item parameters in common question items in the two test forms.
In analysis, we conducted the following procedure. First, we checked an undimensionality in each test. Second, we changed values into binary values in each item. Third, we estimated an item parameter in each item. Using common items in the two tests, we conducted equating of item parameters. Fourth, we estimated ability parameter for each student using the equated item parameters and compared achievement between 2013 and 2014. Finally, we compared achievement growth by schools.
As a result, in English, the mean and standard division in grade 5 were 0.397 and 1.250, respectively, while, these in grade 6 were 1.396 and 1.036, respectively. It means that the average of achievement growth from grades 5 to 6 was 0.999 and standard division was decreased at 0.213. In mathematics, the mean and standard division in grade 5 were 0.173 and 1.241, respectively, while, these in grade 6 was 1.628 and 1.197, respectively. The average of achievement growth from grades 5 to 6 was 1.455 and the standard division was decreased at 0.044. On the other hand, in English, the mean and standard division in grade 7 were 0.406 and 1.116, respectively, while these in grade 8 were 1.469 and 1.190, respectively. The average of achievement growth from grades 7 to 8 was 1.063 and standard division was increased at 0.074. In mathematics, the mean and standard division in grade 7 were 0.455 and 1.266, respectively, while these in grade 8 were 1.593 and 1.466, respectively. The average of achievement growth from grades 7 to 9 was 1.138 and standard division was increased at 0.200. In both grades, achievement growth was larger in mathematics than in English. Interestingly, in both subjects, the standard division was decreased from grades 5 to 6, while it was increased from grades 7 to 8.
The averages of achievement growth were largely difference between schools. In grade 5, the achievement growth of English, the highest average was 2.821, while the lowest one was 0.381. The difference was 2.440. Contrary to our expectations, a few schools showed a negative growth. In mathematics, the highest average was 4.050, while the lowest one was 0.213. The difference was 3.837. It was larger in Mathematics than in English. The tendency was similar in grade 7.
Although almost all studies have discussed student achievement measured in one particular time, this study estimated achievement growth in primary school in Malawi by measuring student achievement at two points in time. The results of this study contribute to discuss improvement of student achievement in developing countries. Further study needs to examine factors influencing achievement growth since growth level was largely different between schools. Also, it needs to find out the reasons why there was a negative growth.

Authors

• Kyoko Taniguchi, Nagoya University

• Haruhiko Mitsunaga, Nagoya University, Japan