Creating a Measure of Discussion Quality in After-School Settings for Linguistically Diverse Youth

• In Event: Bilingualism in the United States: Exploring Linguistic and Social Ecologies Across Home, School, and Out-of-School Contexts

Abstract

Objective:
The Discussion Observation Tool (DOT) was developed to measure the quality of academic discussion in an urban afterschool setting. This study presents the reliability and validity of this new instrument among English Language Learners (ELLs) and their peers.

Theoretical framework:
High-quality discussions are important for the advancement of school-related outcomes for all students (e.g., Cazden, 2001) and especially essential for ELLs (e.g., Baker et al., 2014). Recent research suggests the promise of integrating social-developmental, academic, and linguistic perspectives on discussion (Elizabeth, Anderson, Snow, & Selman, 2012), but limited evidence exists on how to reliably and validly measure high-quality discussions. In particular, prior research has not examined discussions in afterschool, a setting that uniquely activates peer relational and discourse processes to advance academic and social outcomes, including for ELLs. While existing surveys and classroom quality observations capture afterschool proximal processes (Vandell et al., 2007), no tool focuses on discussion processes in these settings.

Data sources:
Multi-method data were collected as part of a larger university-community partnership. Study participants included staff (n = 19) and youth (n = 370) from five afterschool programs (grades 3-8) in one low-income urban area. Sites served primarily Latino (75%) and African-American (21%) students; 36% are ELLs. The proportion of ELLs varied substantially across classrooms (5% to 75%). Afterschool staff matched the youth’s ethnic and linguistic backgrounds.

Informed by prior work (Elizabeth et al. 2012), the DOT was developed to observe how discussion processes advance social and academic skills. The DOT comprises three constructs (academic discussion, analysis and reasoning skills, and advanced language) and focuses on both how the group leader and students exhibited these skills. Six items are rated on a 4-point scale (1 = highly uncharacteristic; 4 = highly characteristic).

Method:
Observations were rated in three contiguous 15-minute cycles by four raters across three time points (fall, winter, spring). In addition to descriptive statistics, generalizability theory (Brennan, 2011; Ho & Kane, 2013) was used to assess the reliability, validity, and sources of variability in this new discussion instrument.

Results:
Within fall observations (winter and spring analyses are forthcoming), Advanced Language used by students has the highest mean score, while group leader facilitated Analysis and Reasoning has the lowest score, demonstrating the tool’s ability to differentiate among various facets of classroom discussion. Preliminary analyses indicate that DOT scores are lower in classrooms with higher proportions (40%+) of ELLs.

The DOT demonstrates high reliability and the ability to capture between and within classroom processes as generalizability coefficients were high (74% when time is random and 96% when time is fixed). Variance decomposition reveals that the classroom component explained the largest proportion of variance on the DOT (56.5%), which is conceptually appropriate as this is the true score.

Significance:
The Discussion Observation Tool systematically and reliability captures proximal processes and discourse quality for linguistically diverse students in the afterschool setting. These findings support the use of the tool to investigate differences in opportunities for high-quality discussion for ELLs and to ultimately improve afterschool programs.

Authors

• Sophia Hwang, New York University

• Michael J. Kieffer, New York University

• Elise Cappella, New York University