Validity Evidence Based on Internal Structure for a Higher Education Written Communication Assessment

• In Event: Developing and Evaluating the Psychometric Quality of Next-Generation Student Learning Outcomes Assessments

Abstract

Theoretical Framework and Purpose

The ability to write effectively using standard written English is particularly important in higher education, where proficiency with written communication is considered a critical student learning outcome (SLO; Sparks, Song, Brantley, & Liu, 2014). To this end, the Educational Testing Service has developed a new next-generation written communication assessment. The purpose of this presentation is to: (a) introduce the process of addressing the limitations of existing assessments in the current assessment, and (b) provide validity evidence based on internal structure to support the use of the new assessment in higher education institutions.

Methods and Preliminary Results

Based on a review of existing frameworks, empirical research, as well as input from higher education institutions, the newly-developed written communication assessment was developed to capture three components of writing knowledge: knowledge of social and rhetorical situations, domain knowledge and conceptual strategies, as well as knowledge of language use and conventions (for operational definitions and example items refer to Sparks, Song, Brantley, & Liu, 2014). To support score-based inferences from this conceptualization of written communication, reliability and dimensionality analyses were conducted based on pilot-test data from 33 institutions (N=3,464).

Before conducting the psychometric analyses, any examinee that responded to less than 75% of test items or possessed an average response time less than 3 seconds was deemed unmotivated and was removed from further analyses. This criteria resulted in the removal of 16 (.005%) examinees from the total sample. Table 1 shows the 7 forms administered with a breakdown of sample size, item type proportions, mean difficulty (based on proportion correct), as well as mean discrimination (based on item-total correlations). As the interest of this assessment is to report scores at the institution-level, aggregate-level internal consistency reliability was next evaluated. Results demonstrated adequate reliability for the total score (α ranged between .87 and .99) and all three subdomain scores (α ranged between .74 and .98). To evaluate the theoretical written communication model proposed, dimensionality assessments were conducted using confirmatory factor analysis. The theoretical three-factor structure was found to provide adequate model fit for only forms 5 and 7; however, the inter-factor correlations were quite high for these two forms (correlations ranged between .88 and .97 for both forms; see Table 2). As a result, a unidimensional model (i.e., all items measure one construct) was evaluated and was found to provide adequate model fit for all forms.

Study Importance

This study provides evidence that a three-factor model of written communication was not supported in a university sample. Such dimensionality results would suggest that institutions that use this assessment may not be able to provide subdomain scores of written communication. Furthermore, these results may be useful for theoreticians and test developers interested in developing new written communication assessments.

Authors

• Joseph A. Rios, Educational Testing Service

• Liyang Mao, IXL Learning