Survey Development for Measuring Math Formative Assessment Use in Schools

• In Event: Exploring the Current Status of Formative Assessments in California’s P-12 Schools as Compared With Schools Nationwide: A National Science Foundation-Funded Project

Abstract

This poster explores the process of survey development for a statewide study on the use of mathematics formative assessments in schools. Research confirms that high-stakes standardized assessments are only successful in contributing to the development of effective schools when they are combined with frequent formative assessments that are used for learning (Black & Wiliam, 1998; Chappuis, Stiggins, Arter & Chappuis, 2004; Stiggins, 1999; Wiggins, 1999). Given the strong relationship among formative assessment, professional development, and student learning cited in the literature, a statewide survey of principals was developed to assist in defining the forms in which formative assessment is utilized in schools.

The survey development process included a variety of steps to ensure the creation of a robust instrument free of bias. While paying specific attention to constructing a survey instrument to generate the data required by the research questions and reliably measure the information across testing situations, we conducting a literature review to develop survey content. From the literature review and analysis of focus group transcripts, the survey questions were developed. These initial drafts were tested in individual think-aloud sessions with school principals. The draft was then revised based on the suggestions offered by think-aloud participants and presented it to an advisory group of education and assessment professionals who then revised the survey in an all-day group meeting. The edited draft was passed on to mathematics and assessment experts for final review and comment. This draft was tested by placing the survey online and recruiting a sample of principals for test-retest reliability. After final revision of the survey items based on reliability results, the survey was launched for data collection.

This process of drafting and revision, backed by expert opinion and reliability testing was utilized in order for the research team to establish the validity and reliability of the survey for collecting the data sought by the research questions. In the test-retest procedure, we computed Pearson correlations and percent agreement to determine the reliability of the survey instrument. Prior to the final revisions, Pearson correlations on Likert-scale items ranged between 0.26 and 0.87. On items where percent agreement was calculated, reliability ranged between 48.3% and 97.0%. Item reliability was found to be variable based on the school size.

Understanding the process of survey instrument development and testing allows researchers to better evaluate data supplied by survey instruments and to understand how better to design survey instruments in their own work.

References:

Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5(1), 7-74.

Chappuis, S., Stiggins, R., Arter, J., & Chappuis, J. (2004). Assessment for learning: An action guide for school leaders. Portland, OR: Assessment Training Institute.

Stiggins, R. (1999). Assessment, student confidence, and school success. Phi Delta Kappa, 81, 191-198.

Wiggins, G. P. (1999). Assessing student performance. San Francisco, CA: Jossey Bass Publishers.

Author

• Nicole Blalock, University of California - Davis