Piloting an Assessment System With Locally Scored and Externally Scored Components for a Potential AP Cambridge | Capstone Program and Credential

• In Event: Technical Issues in Performance Scoring

Abstract

Purpose
As part of considering a potential AP® Cambridge | Capstone Program and Credential®, the AP program piloted an interdisciplinary investigations and critical reasoning seminar course in the 2012-2013 academic year (College Board, 2013). Over 650 students from 15 high schools completed the Seminar Course. The assessment components of the Seminar Course consist of a team project, an individual presentation and a secure timed written exam. The four assessment objectives are deconstruction, reconstruction, reflection, and communication and collaboration. The purpose of this paper is to explore scoring issues encountered during the pilot.

Theoretical Perspective
The team project and individual presentation are internally assessed – graded by the classroom teacher – and externally moderated by Cambridge International Examinations (CIE). While localized scoring with external moderation is routine and accepted in large scale assessment programs internationally (e.g., Queensland Study Authority (2013) and Cambridge International Examinations (2013)) it is not typical for US assessment programs. The written exam is externally scored using experienced reading leadership, trained readers and monitoring of reading for acceptable quality standards.
 
Methods
This paper will review scoring and cutscore setting procedures used and challenges encountered during the pilot. The procedures for introducing a new assessment with components that are locally scored and externally moderated will be compared with established procedures for externally scored assessments. With an eye toward future development, issues that arose and acceptable directions will be the focus of the discussion.

Preliminary Results
The team project is on a topic of local relevance and global importance and requires a 3,000-word group report and a 2,000-word individual report. Thirty points may be earned for the project ¬– six for the group work and 24 for the individual work. Points are awarded based on holistic rubrics for each of the elements of reconstruction, reflection, communication and collaboration. The individual presentation is based on provided material about a global issue seen through a variety of perspectives. Up to 40 points are awarded based on holistic rubrics for each of the elements of deconstruction, reconstruction, reflection, and communication.

The 90-minute skills-based written exam is based on issues of global significance. Students answer required structured questions based on two or more sources provided. Questions require both short and longer responses that demonstrate critical and comparative analyses of the sources material. Responses are not assessed on students’ knowledge and understanding of the specific issues represented in the source material. Up to 30 points are awarded based on holistic rubrics for each response addressing skills of deconstruction and reconstruction. The possible raw score points of 30, 40, and 30 for the team, individual and written assessment components respectively are weighted to contribute 20, 30, and 50 percent toward the final course grade.

Significance
Projects have long been used in K12 educational contexts and higher education settings to promote learning and support assessment. Projects offer the opportunity to assess student competencies that have traditionally been challenging to measure. Innovative scoring approaches are needed to bring these methods to scale.

Authors

• Rosemary A. Reshetar, The College Board

• Thomas Matts, The College Board