Adaptation, piloting, and validation of SEL measures in Nigeria

• In Event: Measuring the Difficult to Measure: School Culture and Social and Emotional Learning

Proposal

The Education Crisis Response (ECR) is a USAID-funded initiative implemented by Creative Associates International in collaboration with the International Rescue Committee (IRC), Florida State University and two local partners. The project supports five states to expand access to quality, relevant non-formal education and alternative education opportunities for internally displaced and host community out-of-school children ages 6 to 17 in areas heavily affected by conflict. To better capture the effects of the intervention, we identified, adapted and validated a set of SEL measures for use in Nigeria through the following process:

Identification of Relevant Dimensions and SEL Measures
We conducted school visits in the Adamawa region of Nigeria to talk to stakeholders and better identify the SEL dimensions that are relevant for measurement purposes. Through focus groups and interviews, we identified the following relevant dimensions for measurement: mental health, prosocial skills and school engagement. We identified instruments from other contexts to adjust to Nigeria.

Translation, adaptation and validation of the instruments
Participants
The sample included 800 six to seventeen year old students (365 girls and 475 boys) from 28 learning centers.
Instruments
Social Competencies Questionnaire: 1) Pro-social and communication behaviors, 2) Emotional self-regulation and 3) Academic skills.
Children’s Stories:
Hostile attribution bias (HAB):
Orientations toward conflict resolution
Mood and feelings questionnaire (Angold et al., 1995): Stress
School engagement vs. disaffection: school engagement and school disaffection
Process
In order to adapt the battery of instruments that we identified we proceeded in the following way:
Translation into Hausa and back into English
Cognitive pre-testing and contextualization
Piloting
Validation
Analytic strategy
To collect evidence of reliability we used Cronbach’s Alpha and factor determinacy of scales and latent constructs. We used Confirmatory Factor Analysis (CFA) to collect evidence of convergent and divergent validity and structural equation models to find evidence of criterion-related validity.

Results

The measures had varying degrees of reliability and validity. During the presentation, Autumn will explore the results of each measure, the constraints and recommendations for future use.

The Social Competence Scale
SCS is designed to measure three dimensions, and dimensionality analysis confirms expected structure.
CFA shows satisfactory fit indexes (CFI=0.92, RMSEA=0.02) and provide evidence of convergent and divergent validity.
Three subscales exhibit acceptable reliability (α > 0.70)
We find evidence of predictive validity as the three social competences measured by this scale exhibit negative and statistically significant relationships with HAB and AIS and positive statistically significant relationships with social strengths.

The Moods and Feelings Questionnaire
The MFQ is designed to measure a single dimension, and dimensionality analysis of confirms expected structure.
CFA shows satisfactory fit indexes (CFI=0.98, RMSEA=0,02), which provide evidence of convergent validity.
Scale exhibits good reliability (α = 0.91).
Predictive validity: Depression has an expected positive relationship with school disaffection and negative relationship with school engagement.

The School Engagement vs. Disaffection Questionnaire
The questionnaire is designed to measure two dimensions: Engagement and Disaffection. Dimensionality analysis confirms expected structure.
CFA suggest that some items should be dropped from the Disaffection scale. After suggested changes, we obtain satisfactory fit indexes (CFI=0.90, RMSEA=0.03), which provide evidence of convergent and divergent validity.
After applying changes, the 'Engagement' dimension displays reliability (α = 0.73) and the 'Disaffection' dimension of (α = 0.90).
Predictive validity: School engagement shows positive and significant associations with problem solving, prosocial skills, academic skills and emotional regulation and negative, statistically significant relationship with school disaffection.

Children’s Stories
CFA cannot be applied in a traditional way because responses depend on each other. Thus, each sub-scales responses were fitted in a model alongside the HAB sub-scale as an alternative approach. The observed fit for these models is acceptable for two models, with the exemption of the subscale of Disengagement, which did not converge.
Reliability analysis displays low reliabilities for all mean scores (alpha <.5).
Predictive validity: HAB shows a positive relationship with AIS, and a negative relationship with problem solving, disengagement, prosocial skills, academic skills, emotional regulation. AIS shows negative and statistically significant relationship with problem solving and disengagement, prosocial skills, academic skills, emotional regulation.

Conclusion
We make recommendations for how to adjust scale to better captures students’ SEL outcomes in Nigeria.

Author

• Autumn Brown, International Rescue Committee