Session Submission Summary

Beyond tool adaptation: how culture and context should inform the development, use and interpretation of SEL measures

Sun, February 19, 4:45 to 6:15pm EST (4:45 to 6:15pm EST), Grand Hyatt Washington, Floor: Constitution Level (3B), Renwick

Group Submission Type: Formal Panel Session

Proposal

School closures in response to the COVID-19 pandemic, along with the other stressors of the pandemic, left deep marks on children’s socioemotional wellbeing. In response to the negative implications, social emotional learning (SEL) programs have been explicitly prioritized across the world (e.g., see World Bank, 2022). Even before the pandemic, educational programs in international development increasingly focused on improving the socioemotional skills of beneficiaries. Socioemotional skills refer to the knowledge, skills, and attitudes that one might expect to see in people as they develop successfully (Blythe et al, 2018). These programs focus on competencies such as self-confidence and communication, largely due to their positive relationship to academic achievement and life outcomes as seen primarily in Western contexts (Heckman and Urzua, 2006; Grant et al. 2017, Taylor et al., 2017). With the rise of SEL programs, came a rise in corresponding terminology, frameworks, and assessment tools, or measures (Jones, et al, 2020).
Assessments of SEL programs typically correspond to a framework that organizes the skills and attitudes that one expects from the participants of the program. In theory, SEL frameworks should be concrete, clear, developmentally appropriate and context relevant (Casillas, Roberts, & Jones, 2023). Contextually relevant frameworks are those which are appropriate for the situations that individuals experience in their lives and capture the way in which the target skills manifest in relevant situations (Casillas, Roberts, & Jones, 2023). For example, if a framework is concerned with the way in which students relate to other people, it should account for the specificities about the way children in that society relate to others at school, in their homes, in their communities and other relevant contexts.
SEL frameworks also need to be culturally relevant. Culture corresponds to systems of values, norms, beliefs, and customs that shape the way people make meaning and behave, though culture itself is an ill-defined, nebulous concept (Gay, 2000; Hecht & Shin, 2015). Assessment frameworks, including those for SEL, must be responsive to the specific cultural parameters of the learners being assessed. SEL frameworks need to be particularly culturally relevant to enhance their value, meaning and efficacy among the diverse populations they serve and respond to the different ways in which social and emotional skills are demonstrated. In countries like the United States, culturally relevant SEL frameworks are typically concerned with diverse and underserved learners and how they relate to social justice in their educational systems (e.g., see Casillas et al. 2023). In the context of international education, thinking about culturally relevant SEL frameworks also means questioning the adoption and proliferation of Western constructs in non-Western contexts.
SEL frameworks guide the production or selection of measures that are used to evaluate programs and advance knowledge. Few studies have focused on the local understanding of target SEL skills, before engaging in the development and assessment of related measures (Jukes et al, 2018; Giacomazzi et al., 2022). Researchers and practitioners in international education are encouraged to adapt measurement tools designed for existing frameworks to the local context (e.g., RTI International, 2015; Jones, et al, 2020). Adaptation is meant to ensure that tools and measures are adequate for new settings by “removing construct irrelevant variance” or minimizing cultural and linguistic differences that are considered irrelevant to the skills being measured. In reality, adaptation typically consists of translation and item validation by locally relevant stakeholders, and is primarily concerned with issues like measurement invariance and reduction of statistical bias (e.g., see ITC, 2015). Based on these practices, the extent to which tools can be modified is limited, as adaptations start from “well researched” frameworks and pre-defined sets of items, primarily based on frameworks designed in different (Western) contexts. Moreover, in practice, many researchers and evaluators have a predetermined set of outcomes that they are interested in measuring, meaning that adaptation of tools may be superficial at best. While these practices are largely consistent with international guidelines on testing (e.g., ITC 2015), they do not go far enough to fully account for cultural and contextual variation. Indeed, the mainstream psychometric field has started to move away from traditional norms and guidelines that limit the transformational power of tests and have drawn from critical race theory and similar frameworks to challenge notions such as construct equivalence and standardization (e.g., see Randall, 2012; Sireci, 2020). Similar efforts are necessary to confront the misrepresentation, mismeasurement, and potential harm that can be introduced by culturally and contextually irrelevant SEL frameworks and measures.
This panel seeks to gain insight into alternative ways to account for culture and context in SEL frameworks and tools. The panel will explore ways in which to incorporate local viewpoints at the development stage across several countries in the Global South and for different socioemotional and life-skills. The first paper will draw on qualitative methodologies to gain deep understanding into the ways in which local stakeholders and communities frame the construct of Problem Solving, and the implications that such framing has for evaluation practice. The second paper will use mixed methods to explore the power of developing SEL frameworks and tools that are meaningful for the participants and will share insights into adequate ways of approaching such task. Together, these two papers will describe why contextually and culturally relevant SEL frameworks and tools should be developed from the bottom up, particularly in non-Western populations. In turn, the third paper will use quantitative methods to highlight how even after thorough adaptation, SEL tools may fall short to produce equivalent measures across countries. In addition, using interviews, the study will explore the potential causes of why “adopting and adapting” SEL measures may not work as intended. Overall, the panel is designed to use the authors’ experience to foster an evidence-based critical dialogue around ways in which to think about SEL frameworks and tools in the Global South.

Sub Unit

Chair

Individual Presentations

Discussant