Poster #25 - LLAMA: Development of a Computerized Adaptive Assessment for Preschool Mathematics

• In Event: Poster Session 05

Abstract

Links to Learning: Adaptive Mathematics Assessment (LLAMA) is an innovative assessment linked to instruction in the domains of numbers and operations, shape and space, measurement, and pre-algebra. This poster describes the first three phases of measurement development for LLAMA.
In Phase One we generated construct maps through a systematic literature review. Within each domain we enumerated skill areas and devised a learning progression. Prototype task models and sample items were created and translated into a digital interface for initial validation with teachers, experts, and children. Fourteen preschool teachers reviewed a demonstration version of LLAMA and completed a brief survey. The majority of teachers reported LLAMA would be effective at gathering information about a child’s early mathematical skills and could be used to identify whether a child is having difficulties in mathematics. Additionally, 92.9% of teachers reported LLAMA is likely to be helpful in developing instruction and intervention strategies. Five experts who reviewed LLAMA materials agreed LLAMA content reflects current early math research, although there was some concern about item fairness and the utility of some content. Piloting with a sample of preschoolers suggested children engaged easily with LLAMA, and the mean duration of interactions supports feasibility.
LLAMA seeks to increase child engagement using three ‘adventure’ assessment contexts (i.e., underwater, outerspace, and rainforest) and colorful images. In Phase Two, we explored the following questions: Do children differ in their preference for context? Does context choice differ by child demographics? Does performance differ based on children’s opportunity to choose? Participants were 68 preschoolers from diverse racial, ethnic, linguistic, and ability backgrounds that were matched based on demographics to two conditions: (1) Choice and (2) No Choice. Children in the Choice condition selected the context for assessment while Matched No Choice children received the adventure context that the previous Choice condition child selected. Results indicated the majority of children chose rainforest, with fewer choosing space, and the smallest number choosing ocean with no discernible pattern based on demographics. Additionally, there was no statistically significant difference in performance based on choice.
In Phase 3, we piloted LLAMA with a sample of 150 children from diverse backgrounds to address the following questions: Do assessment items meet a priori criteria for proportion correct, item-total correlation, item/person fit, and differential item functioning? Do item banks provide coverage of the ability scales for mathematical learning progressions? We explored the use of multidimensional Rasch models with simple loading structures (Adams et al., 1997) that provide multiple ability estimates within a learning progression. Item statistics reported include proportion correct, moderate discrimination, Rasch in-fit and out-fit statistics, and differential item functioning (DIF). Rasch models were fit and compared by learning progression, using concurrent calibrations of all forms. Forms, tasks, and progressions are evaluated based on internal consistency, item and test information curves, standard error curves, and overall scale characteristics. Item coverage across the theta scale and overlap in the distributions of items and persons are evaluated using item-person plots.

Authors

• Kristen Missall, University of Washington
  Presenting Author

• Robin Hojnoski

• David James Purpura, Purdue University, West Lafayette

• Anthony Albano, University of California, Davis