Instrumentation for Identifying Areas of Growth for Computer Science Teachers

• In Event: Promoting Equitable CS and AI Education: Strategies to Expand Capacity and Access

Abstract

Objectives or Purposes
The CSTA's Standards for CS Teachers [28] outlines five domains for professional practice and embodies the principle that all students can learn CS when taught by educators dedicated to inclusive practices and ongoing professional development. The purpose of this study is to fill the need for ways to identify areas of growth for teachers in teaching CS against the standards.

Perspective(s) or Theoretical Framework
A multiple-choice assessment built on vignettes presents a practical alternative to traditional measures. The use of vignettes is a well-accepted method for assessing pedagogical content knowledge, as they situate questions within authentic classroom stories [29]. Vignettes offer insight into a teacher's decision-making process by requiring them to apply their judgment to a specific, detailed context. They also fill a methodological gap, offering a scalable solution that is more contextualized than self-report surveys but less resource-intensive than direct classroom observations.

Methods and Data Sources
Vignettes have been explored to measure CS pedagogical content knowledge [30]. Given this precedent, we created a K-5 vignettes-based assessment. To start, we chose to assess teacher knowledge against three standards (2, 4, and 5). We implemented the study as mixed-methods by conducting cognitive interviews and then with the final set of vignettes.

We developed and validated an assessment instrument consisting of three vignettes aligned with Standards 2, 4, and 5. The instrument was refined through cognitive interviews with K-5 teachers (n=5) and then pilot-tested with U.S. K-5 teachers (n=111). An analysis based on classical test theory was used to establish the measure's reliability and to determine item difficulty and discrimination values. We also collected evidence of the measure's content and response process validity.

Results
An analysis of the assessment scores showed an approximately normal distribution. Item-level analysis indicated a wide range of item difficulties, from moderately difficult (p=.46) to very easy (p=.95). Item discrimination values ranged from .16 to .48, suggesting some items were more effective than others at differentiating between high- and low-scoring teachers. The measure's internal consistency was modest (α=.66), indicating that reliability could be improved. Scores were positively correlated with teachers' self-reported teaching awards; however, no other independent variables significantly predicted scores.

Scholarly Significance
While this new measure of teacher proficiency shows promising psychometric qualities, further item refinement is required. Once finalized, it can serve as a practical tool for identifying teacher strengths and informing targeted professional development.

Authors

• Joseph C. Tise, Institute for Advancing Computing Education

• Monica McGill, Institute for Advancing Computing Education

• Vicky Sedgwick, CSTA

• Amanda M. Bell, Computer Science Teachers Association