Using Network Analysis to Study Civic Participation in Schools: Application and Insights from ICCS Data

• In Event: Beyond Borders: Exploring Civic Engagement through Comprehensive Data

Proposal

Preparing students to participate in political life is usually considered one of the main goals of education systems across the world. To effectively assume their citizenship duties, students need to nurture capacities that supersede literacy and mathematical skills (Westheimer & Kahne, 2004). Among these capacities, the literature highlights civic knowledge, civic skills, and civic values (Allen, 2016). One alternative to advance student learning in these domains is the provision of participatory activities in schools, following the intuition that the best way to learn how to engage in political life is by garnering experience in activities that resemble it (Rebell, 2018). Considering the use and variation of civic participation activities in schools, what is the best way to measure students’ experiences with it?

In this paper, we propose an alternative approach to score and measure students’ reports of participation in civic engagement activities in schools, using data from 21 countries in the 2016 cycle of the International Civics and Citizenship Study (ICCS). More specifically, we use the tools of Social Network Analysis (SNA) to generate networks of student co-reported participation. This approach allows us to map the interconnections between student reports for each of the six participation items in the ICCS student questionnaire. We use these results to generate an SNA-score for each student, representing their average position within these six networks at the school-grade level (i.e., a measure of centrality). By factoring in the co-reported participation of students’ peers in the scoring procedure, we aim to better capture the construct of interest. Critically, we use the same items the ICCS uses to generate measures of student participation in their own reports. However, we use the raw scores in the dataset in an alternative way.

To better understand what our measure uncovers, we use the raw data of the civic participation items in the ICCS to generate comparison participation scores for each student by fitting an Item Response Theory (IRT) Graded Response Model (GRM). This set of GRM-scores represents the traditional approach to scoring which conceptualizes civic participation in schools as an individual trait. We explore the correlation between SNA- and GRM-scores for students in each participating country, and visually compare score distributions. Pearson correlation coefficients for the SNA and GRM scores range between 0.72 and 0.94 for the 21 countries in our dataset. As expected, both measures are highly yet not perfectly correlated. We further explored the association between both scores with the use of two-way scatterplots. Visual inspection of the graphs shows that, across the 21 participating countries, there is significant variation in student SNA-scores for identical values of GRM-scores (Figure 1). Substantively, this indicates that students reporting the same levels of participation across the six survey items, are nevertheless differently positioned in the networks we can construct from the data (Figure 2 & Figure 3).

To further test the validity of the use and interpretation of the SNA-scores as measures of student participation in civic engagement activities in schools, we perform Classical Test Theory (CTT) analysis on our data and conduct a Factor Analysis (FA). Cronbach’s alpha values with our SNA approach range from 0.49 to 0.79 across countries, being relatively smaller than the ones obtained for the raw scores in the participation scale. Factor analysis results suggest a unique underlying dimension to the network-generated participation scores. Factor loadings for each of the six measures of degree are above the conventional threshold of 0.3. Goodness of fit statistics tend to show adequate model fit (RMSEA ≤0.08;CFI ≥0.9;SRMR ≤0.08).

In a final step we use the SNA and GRM-scores to predict students’ civic test score performance. We fit a series of random intercepts multilevel models and observe that SNA-scores tend to be positively associated with civic test scores, above and beyond what is predicted from the GRM-scores (Table 1). The inclusion of two highly correlated predictors in our models renders the interpretation of the size of the coefficients difficult and collinearity becomes a source of concern. However, we believe the consistency of the direction and statistical significance of the results broadly suggests that, for students reporting the same level of individual participation in-school activities, being better connected in the co-reported participation networks is positively associated with civic test scores (Table 2).

Overall, our work shows that for all the technical progress being made in the measurement field, thinking carefully about the characteristics of the construct we are interested in measuring remains a worthwhile endeavor. We find preliminary evidence that conventional measurement approaches to civic participation experiences in schools might be masking some features such as the importance of participating together with others. Far from conclusive, results from this study can nevertheless inspire future research on the social and relational dimensions of students’ learning experiences. We also hope they can be thought-provoking for those researchers in the field interested in leveraging conventional assessment and survey data in innovative ways.

Author

• Nicolas Riveros Medelius, Harvard Graduate School of Educatio