Paper Summary
Share...

Direct link:

Literacy Design Collaborative Module Jurying: Innovating for High-Quality Design

Mon, April 7, 2:15 to 3:45pm, Convention Center, Floor: 100 Level, 117

Abstract

How do you ensure that innovative tools are being used by practitioners to develop high quality (valid and rigorous) products?

The theory of action underlying the Literacy Design Collaborative's work is a hypothesis that by providing a set of templates for writing tasks and instructional modules, the quality of teachers' assignments and instruction will improve. Prior research has found the quality of teacher assignments to be predictive of student achievement (see for example, research on the Instructional Quality Assessment (IQA), Matsumura, Garnier, Pascal, & Valdés, 2002; and research on the scales used by the Center for Authentic Intellectual Work, Iowa Dept of Education, 2012, and King, Schroeder, & Chawszczweski, 2001). The ultimate goal of LDC is to improve student learning, achievement, and readiness for college and careers.

While the LDC teaching task and module templates were designed to "hard-wire" the Common Core Literacy Standards into the module, it became apparent after a year of piloting the LDC templates across the country that the templates did not guarantee quality or rigor. While deceptively simple to use, the templates require that module designers make strategic decisions about what content, texts, questions, and instructional strategies to insert into the "fill in the blank" templates. In addition, there were few models that could be presented to practitioners as exemplars to emulate. LDC identified a clear need to develop a set of quality criteria and module exemplars that could be used to set the standard for what "high quality" look like.

Beginning in the fall of 2011, the Literacy Design Collaborative (LDC) and the Stanford Center for Assessment, Learning, and Equity (SCALE) began a collaboration to develop criteria to evaluate the quality of the LDC modules that were being designed by teachers and other users, including their alignment to the Common Core Literacy Standards (CCLS). Another purpose that shaped the design of this tool was the goal of providing fine-grained, constructive feedback that task designers could use to improve their designs. This paper describes the development process and rationale for the design of the "Jurying Rubric" that emerged from that process.

In addition to the rubric, SCALE was commissioned to develop a training protocol and anchor modules to train and calibrate LDC jurors. This training protocol underwent several rounds of pilot testing and refinement. The jurying process that was settled upon is a socially moderated process in which jurors rely on each other's expertise. In this approach, each module is scored independently by at least two jurors who then compare notes and arrive at consensus on the scores. This paper discusses the rationale for the design decisions that were made about the jurying rubric and jurying process, including the decision to rely on distributed expertise. An electronic scoring and annotation platform designed for the project opens the door for remotely located jurors to work together on rating modules and for a national community to share the responsibility of holding itself accountable to a common standard of quality.

Authors