Item Feature Analysis Approach in Panama's Third-Grade Math Assessment

• In Event: 64.036 - Feature Analysis Approaches in International Assessments

Abstract

Objective. This study aims to achieve the following objectives: 1) using feature analysis to obtain a synthetic qualitative description of the structure and composition of Panama’s national assessments; 2) improve the balance of the items regarding their qualitative properties, complexity and types of representation.

Perspectives. In recent years, Panama has promoted an agenda for the improvement of its assessment system focused on the design of evidence-based policies. Consistent with these objectives, the Ministry of Education launched a National Assessment Plan as a central tool to guarantee the rights to an equitable and quality education, based on a pedagogical framework, the Fundamental Rights of Learning (FRL) ,that provides a set of desirable achievements for Panamanian children. Panama then implemented the 'Crecer' standardized assessments, in the domains of Literacy and Numeracy and Science. These assessments are applied annually at the end of the school year and the results of each are given to students, teachers, families and schools, and other stakeholders.

Methods. The common feature analysis framework for all papers in this symposium was used to rate items by assessment experts.

Data. We analyzed the 60 items used to assess Mathematical skills corresponding to children attending the third grade, which are based on three domains: 1) Arithmetic and Algebra, 2) Geometry and Measurement, and 3) Statistics. The evaluation was applied to a population of 62,195 students in 2016 and 57,518 in 2017, from 3,077 educational institutions (82% public and 8% private school).

Results. The general framework of the characteristics of the items corresponds largely with this symposium, however, some content and concepts were adapted by content experts to reflect the national structure Panama. Results summarize as follows:
a) The 60 items are distributed with some under or over-representation among 16 of the 17 concepts and contents of mathematics and 1 of them is not reflected in any item. Nine content areas have a very low presence in the tests (less than 5%, i.e. 3 or less items of the test). Five areas have a presence between 7% to 17% (4 to 10 items), and only three content areas appear in 22 or more items (Ordinal numbers, Basic operations and Numeration). The average content presence in each item is 2.12, with a minimum of 1 and a maximum of 5.
b) Representations: graph, chart, picture, and geometric shape were used with low frequency (5% - 17%), while numbers and text were used in more than a half of test items.
c) Regarding complexity and vocabulary, 95% of the items have an academic mathematical vocabulary of which the average was 1.6 out of 1 to 3 scale.
d) The relationship between item features and item difficulty was consistent with other papers.
Scientific Significance. The findings will be used to revise future assessments and communicate with teachers about potential inferences about their instructional practices.

Author

• Harvey Spencer Sanchez Restrepo, Agencia Latinoamericana de Evaluación y Política Pública