Emerging Validity Evidence for a Survey Measure of Connected Learning

• In Event: Connected Learning: Evidence in Support of an Emerging Model for Learning Across Settings Among Nondominant Youth

Abstract

Purpose
The objective of this study is to develop and validate a survey instrument to measure youth experience of connected learning.

Theoretical Framework
The theoretical framework of connected learning and its features is described as part of the session overview. In order to be measurable, these features need to be conceptualized in such a way that individual differences between persons can be (a) defined, and (b) connected to observable outcomes of a measurement procedure (in this case, survey responses). The “construct mapping” framework (Wilson, 2005) was applied to articulate these theories.

Method
The survey development process began with the articulation of construct definitions for features of connected learning, identification and development of items, and pilot testing (Dadey, 2013). We then conducted a field test with a large sample of youth; the multidimensional Rasch model was used to examine the extent to which response patterns conformed to theoretical expectations.

Sources of Data
Data come from a survey conducted in spring 2013. The study sample was comprised of 497 youth aged 12-17 from 19 different program sites that provide opportunities for youth to pursue activities that reflect many of the features of the current model of connected learning. Both the average and modal age of youth in the survey is 15. Just over half (50.3%) were female. The sample is ethnically diverse: 32% identified as Mexican, Mexican American, or Chicano; 26% as African American; 18% as White; 17% as other Hispanic or Latino; 10% as Asian or Asian American; and 5% as Native American.

Results
The Rasch model misfit the data in a variety of informative ways. On the most negative end, there is no evidence from the current survey that the “academically oriented” feature of connected learning is measurable at all. The five other features fared better, though in some cases the empirical results suggested revisions to construct maps, which will be treated as hypotheses to be tested in future rounds of data collection. The reliability estimates for these features ranged from .59 to .79. The (disattenuated) correlations between the features ranged from .07 to .61. There was thus no indication that any of the features were so closely related as to be considered redundant.

The measured features varied significantly by participation patterns of students (both frequency and mean duration of sessions). For many features the mean difference between sparsely attending and frequently attending students was greater than .50 standard deviations. The causal model underlying this result bears further investigation. A hidden third variable, such as an individual’s well-developed interest in a topic (Hidi & Renninger, 2006) could account for both frequent participation and perceptions of connected learning.

Scholarly Significance
This study makes conceptual, practical, and substantive contributions to understanding connected learning. Conceptually, the study helps to refine feature definitions. Practically, it contributes validity and reliability evidence for an instrument that could be utilized in future research. Substantively, we have presented evidence concerning the structure of the dimensions of youths’ experiences of connected learning.

Authors

• Andrew Maul, University of California, Santa Barbara

• Lawrence P. Gallagher, SRI International

• William R. Penuel, University of Colorado - Boulder

• Timothy Podkul, SRI International