Feature Analysis of the Basque Country/Spain Assessment Using Eighth-Grade Math Items

• In Event: 64.036 - Feature Analysis Approaches in International Assessments

Abstract

Objectives. This study aims to identify features observed in the Basque Country Diagnostic Assessment and to examine the relationship between identified item features and student performance.

Perspectives. The Basque Country is one of the 17 Spanish autonomous communities, and each has the duty of managing and evaluating the education system in its territory. The Basque Institute for Educational Research and Evaluation (BIERE), funded by the Department of Education of the Basque Government (www.isei-ivei.net) conducts evaluation using Diagnostic Assessment. The main characteristics includes: a census evaluation, carried out annually in the 4th and 8th grades; its focuses on the assessment of basic instrumental skills: Mathematics and Reading Comprehension. Reading comprehension is assessed in both Basque and Spanish; a very low-impact and formative evaluation; the results are disseminated to both families and schools.

Methods. The general framework of rated items features is common to this symposium, although some of the features relating to the contents and concepts of mathematics have been adapted to Spanish context and 8th grade curriculum.

Data. This study analyzed the 59 items used in the 8th grade mathematics test. All items have been used in different assessment in years 2010, 2011, 2013 and 2015, on average, for 18,940 students in 891 classrooms in 329 schools. The tests are computer-administered but are not adaptive tests.

Results. The main results are as follows:
a) 24 content features are distributed very unevenly among the 59 items. Nine content features are not reflected in any item. Eight content features have a very low presence in the items (less than 10%). Six content features have a presence between 10 and 25% and one content area, Order of Operation, appears in 25 items. The average content present in each item is 1.81.
b) As to the types of representation, most prevalent presentation types are text, numbers, drawings and variables. The average number of types of representation present in each item is 3.88.
c) In 93 percent of cases, language is described as complex and academic.
d) Beyond the above descriptive results, regression analysis was conducted of the feature-tagged items with achievement as the outcome variable. The results show that the features of items help explain 40% of the variance of the difficulty parameter (R2 = 0.404). The characteristics that most contributed to this explanation were Math concepts (linear equation and functions), logical problem solving, and problem type including world knowledge, and extraneous information.
Scientific Significance. Feature analysis results will be used to help the Basque Institute for Educational Research and Evaluation to improve the tests, for example by eliminating those items that do not reflect the content and competencies of the specification matrix, and by adjusting overrepresentation. It will also provide a basis for teacher feedback. The use and consistency of findings in feature analysis provides empirical support for this qualitative technique.

Authors

• Maria Castro Morera, Universidad Complutense de Madrid

• Luis Lizasoain