Assessing Students’ Growth in Computer Science Knowledge: Applying Diagnostic Classification Modeling

• In Event: Assessing Computational Thinking: Updates and Advances in the Field

Abstract

A key strategy for broadening CS participation in the Chicago Public Schools (CPS) has been the enactment of a high school CS graduation requirement in 2016 (Dettori, et al., 2018). Exploring Computer Science (ECS) is the primary course that students have been taking to fulfill the requirement in CPS. ECS is composed of activities that engage students in CS inquiry around meaningful projects (Margolis et al., 2012). The ECS pedagogy is structured around three interwoven strands: equity, inquiry, and CS concepts. CPS considers the first four units of ECS the minimum for fulfilling the requirement: Human-Computer Interaction, Problem Solving, Web Design, and Programming.

To measure the development of CT practices, CPS uses pretest and posttest assessments developed by SRI International that were aligned to the CT practices in ECS (Snow et al., 2017). The pretest contains six tasks that measure students’ initial understanding of CS concepts and CT practices. The posttest has five tasks, two of which were on the pretest to equate the two forms and allow for measurement of growth from pretest to posttest. Across the pretest and posttest tasks, there are a total of 30 question prompts that are each scored individually, using SRI rubrics. CPS has been using the Rasch-scaled assessment scores since the enactment of the requirement to track overall student growth of CT practices (Authors). However, there is a need to provide formative, diagnostic information to teachers.

The present study analyzed ECS assessment data collected from 3,832 CPS students (Grades 8-12). Longitudinal diagnostic classification models (LDCM; Madison & Bradshaw, 2018) were fitted to the pre-and post-ECS assessment data to trace changes in mastery proportions of four domains of the ECS curriculum: human-computer interaction, problem-solving, web design, and introduction to programming. Logistic regression analyses were subsequently conducted to investigate which demographic variables predict learning growth (i.e., the transition from non-mastery to mastery between pretests and posttest).

Figure 1 shows the ECS curriculum contributed to enhancing students' problem-solving and programming skills. Mastery proportions remained or decreased in human-computer interaction and web design. Increased mastery proportions were found in problem-solving (p < .001; d = .247) and introduction to programming (p < .001; d = .240). Second, with respect to broadening computer science participation, there were two meaningful findings in the results of logistic regressions (Table 1). In race, significant odd ratios were observed for Black students in all four domains. Significant odds ratios were observed among Hispanic students in problem-solving and introduction to programming. These results suggest that compared to white students, underrepresented racial groups in CS benefit more from the ECS curriculum. Students who received special education were less likely to transition to mastery status, particularly in programming (OR = .691, p =.034). To enhance their learning experiences, teachers and stakeholders should prioritize universal design for learning to make CS education environments more inclusive.

These results provide evidence of LDCM fit for the ECS assessments, which can provide ECS teachers with formative data. Teachers can use that data to adapt ECS to meet the needs of their students.

Authors

• Chungsoo Na, Utah State University

• Steven McGee, The Learning Partnership