Conundrums in Measuring Change in Achievement Over Time

• In Event: 75.022 - Dilemmas, Tensions, and Trade-Offs: Some Methodological Conundrums in Education Research

Abstract

Purpose
Large-scale assessment studies are used to examine changes over time. Measuring change over time presumes an underlying construct that may be manifest in performance on particular tasks that may be different at different times. This paper focuses on challenges involved in studying change in achievement in Information and Communication Technology (ICT) literacy where the contexts have changed over time.
Perspectives
Large-scale assessments are intended to measure changes in achievement over time. Item response theory (IRT) is used so that scores in any given cycle are represented on the original scale. IRT posits that any given assessment is a manifestation of an underlying (or latent) trait. Measures of change are concerned with changes in the latent trait. Common-item equating is typically used in the assessment design. Common items that behave in a consistent manner are used to equate scores to the original scale. In PISA approximately one quarter of the items, and in TIMSS just over two-fifths of the items, have been used in previous cycles.
Data and Methods
Data for this paper are drawn from the Australian National Assessment Program ICT – Literacy which monitored changes in the ICT literacy of approximately 10,000 Grade 6 and 10 students over 2005, 2008, and 2011 (ACARA, 2012). The challenge was to assess change consistently in the context of changes in technologies. ICT literacy was assessed with a computer-based test which included multiple tasks related to the creation of digital products. These tasks were embedded in a set of modules based on a narrative theme. Each assessment involved tasks embedded in seven thematic modules and each student was randomly assigned four of these modules. Each of the 2008 and 2011 assessments consisted of three modules from previous cycles and four new modules. New modules included up-to-date software contexts that were not widely used in 2005: e.g. the use of blogs web-based databases, mapping software, video editing, webpage authoring and collaborative workspaces. Common tasks from the trend modules were used to link the 2011 results to those from 2005 and 2008. The surveys also analyzed data about students’ backgrounds and perceptions and use of ICT.
Results
There was a statistically significant increase between 2005 and 2011 in the mean score for Grade 6 students from 400 to 435 scale points which is approximately one-third of standard deviation. For Grade 10 there was no significant change in ICT literacy over the same period of time. In both grades most of the relations between achievement and student background remained the same over time.
Conclusion
The conundrum in this exercise is how to balance measures of performance on particular tasks of current relevance with measures of change in the underlying construct. We argue that it was possible to introduce change in the assessment but still generate measures of change in ICT Literacy. The instruments reflected relevant technological changes and the core concept of ICT literacy. However, the question remains about how far this “core plus” model can be extended as the technological context change.

Authors

• John G. Ainley, Australian Council for Educational Research

• Julian M.S. Fraillon, Australian Council for Educational Research

• Eveline Gebhardt, Australian Council for Educational Research