Building Common Measures Through a Networked Improvement Community Among Computer Science Education Researchers

• In Event: Measuring the Efficacy of Large-Scale Computer Science Efforts in K–12 Settings: Strategies and Challenges

Abstract

As CS4All expands rapidly across the U.S., quality evaluations are essential to success. However, there is limited shared knowledge and research for the new field of CS evaluation. This paper will describe our efforts to coordinate and build shared evaluation resources, approaches, and practitioner capacity. Adapting a Networked Improvement Community (NIC) model, we’ve sought to learn: 1) What are the perceived needs of the CS evaluation community? 2) What common goals can be established and how can these be met by an organization that brings together a larger CS evaluation community? 3) How can this community build consensus and advance work within this field? What challenges arise?

Bryk’s Networked Improvement Community model (2011) guided adoption of this NIC’s tenets: (1) focused efforts on improving the measurement of two crucial elements—classroom implementation and student success, (2) integrating extant research-based knowledge in CS evaluation methods with collective analytical and empirical approaches, (3) embracing a “learning by doing” orientation for practicing evaluators, and (4) a view of the NIC bound by norms and structures. Kania and Kramer’s work on collective impact (2011) also guided our approach around the need for centralized infrastructure, dedicated staff, and “and a structured process that leads to a common agenda, shared measurement, continuous communication, and mutually reinforcing activities among all participants.”
For this paper, we draw on multiple sources of data including a needs assessment survey administered in the winter of 2017 to 50 CS evaluators, facilitated working group sessions attended by 37 participants resulting in subgroups, and systematic documentation of common strategies and challenges among participant researchers. The needs analysis (n = 50 of 84 evaluators) revealed that many aspects of quality implementation go unmeasured and that the community would like to increase its capacity to evaluate CS programs, while establishing structures and expectations that support their evaluations.

Results from these data generated four goals that the NIC is actively engaged in: 1) Building an interactive hub for housing CS evaluation information: Features and functionalities of this site are being developed collaboratively by subgroups with representatives from across the community. 2) Gathering and curating a set of CS measures and a shared platform to house actual measures. 3) Building a platform to provide knowledge-building information such as webinars as well as inviting and facilitating ongoing community participation and discussion. 4) Educating policymakers, influencing legislation, and engaging corporate stakeholders (being led by two of the community’s subgroups). Since June 2017, this group has led biweekly meetings coordinating the advancement of these different work strands. Challenges include communicating key and timely information to the large group of active members.

This paper addresses the critical area of collective impact. For the CS evaluation community, the time is ripe for sharing best practices and building evaluative capacity through developing resources and fostering communication. This effort to start, develop, and foster a NIC focused on CS evaluation will greatly contribute towards this end and ultimately towards producing better evidence about the potential for CS education to make a difference for students.

Author

• Kathleen C. Haynie, Haynie Research and Evaluation