Practical and Conceptual Issues of Dimensionality in Writing Assessment

• In Event: Issues of Writing in Assessment and Feedback

Abstract

The National Education Monitoring Project in New Zealand assesses the achievement levels of 8- and 12-year old students across New Zealand’s curriculum using a low-stakes, light-sampling, performance-based approach. In the area of writing, students are assessed using a variety of different approaches, including a multiple stage, extended writing activity, short writing tasks, editing and revision tasks. Some tasks are strictly paper and pencil whereas others are administered in a one-to-one format with teachers who are trained to do assessments and individual students.

The purpose of this research was to take a multi-faceted look at the assessment of writing ability of eight and twelve year old students in New Zealand in the context of a new national initiative to have teachers assess the reading, writing and mathematics abilities of children in a programme called National Standards. National Standards, based on the national New Zealand Curriculum, require teachers to report on the progress of students twice each year in terms of achievement against a set of standards of expected achievement levels. Teachers can use whatever measures to assess children that they believe are most appropriate for their setting.

In the first year of the program, teachers are struggling to determine whether students meet the standards or not. The problems come with understanding the specificity of the standard, and with getting assessment information for children that they feel they can rely on.

In this study, we take random samples of 450 students each at Year 4 (grade 3 in the United States) and Year 8 (grade 7 in the United States) and look at eight different writing tasks that the students have been given. These tasks are specifically designed to broadly cover the kinds of writing that children do in school. One of the particularly useful aspects of the National Education Monitoring Project is that the same tasks are given at Year 4 and Year 8 and graded against the same set of standards. Thus, we have directly comparable data across these two years.

We analyzed the data by first conducting a factor analysis on both samples of students to look for unidimensionality in performance. The eigenvalues showed a strong first factor for both samples. We then created total scores from the tasks, and broke the samples out into clusters of students performing around the 10th, 30th, 505h, 70th, and 90th percentiles for both Year 4 and Year 8. Next, we went back to the original tasks, any of which might be used individually by teachers in their assessments, and looked at performance on a student-by-student and task-by-task basis. What we found was that although the data clearly supported the creation of a single scale for these tasks, performance on a task-by-task basis was highly variable from student to student, and the use of any individual task, or small subset of tasks, would leave to substantial inconsistency in assessing students against the standards. Results are discussed in terms of both theoretical and practical considerations.

Authors

• Jeffrey K. Smith, University of Otago

• David A.G. Berg, University of Otago

• Ros Allan, University of Otago

• Alison Gilmour, University of Otago