The Importance of Situated Assessment Frameworks in the Evaluation of Life Skills Programs

• In Event: Implementing and Measuring Evidence-based Social and Emotional Learning & Life Skills Education interventions in Complex Environments

Proposal

Educational programs are increasingly focusing on socioemotional and/or life skills. Life skills refer to skills that enable people to deal with the needs of their daily lives (AGI, 2013). Evaluating the success of life skills programs involves examining the changes in life skills among participants, but also their ability to use those skills in their daily decisions. Therefore, appropriate assessment frameworks for life skills programs need to be situated in the lives of the participants. In this presentation we argue about the importance of contextually relevant assessment frameworks in life skills evaluations, and we share our insights about how to develop frameworks in participatory manners. In particular, we will describe how we developed our own assessment framework, which consists of a (a) revised blueprint, (b) participatory construct maps, a (c) data collection tool and a (d) validation plan to be used alongside.

The presentation will describe how validity and validation should inform the development and maintenance of assessment frameworks. Validating a tool to be used in program evaluation is a tedious and complex process. Validity is understood as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests“ (AERA, APA, & NCME 2014, p.11). Important distinctions emerge from this definition: validity is a property of the score interpretations (not of a tool), validity is dependent upon the intended uses of the instruments, and validation relies on both theory and evidence.

A popular validation framework that is consistent with this notion of validity is what Kane (2013) refers to as argument-based approach to validity (ABA Validity). In a nutshell, argument-based validation consists of a) specifying interpretations and uses of the scores, b) identifying claims/logical paths that would support the intended interpretations and uses, c) gathering evidence to evaluate the plausibility of the claims, and d) making a judgment about each of the proposed scores’ interpretations and uses are justified. The similarities between ABA validity and theories of change, makes it particularly suitable for program evaluation settings.

An argument-based approach is as good as the theory and evidence it is based on. This presentation will discuss the relevance of different sources of evidence for life skills programs in international education. Traditionally, the types of evidence used in validation efforts included evidence based on content, on relationship to other variables, on internal structure, on response processes and on consequences. More recently, researchers have proposed expanding the sources of evidence to acknowledge the sociocultural and contextual characteristics that cannot and should not be ignored in the validation process (e.g. Randall, 2021; Mislevy, 2018). Drawing from these recent advancements, we will discuss the ways to ensure that assessment frameworks in life skills programs are adequate, ensure meaningful learning and promote justice.

Authors

• Fernanda Gándara, Room to Read

• Daniel G Oliver, Room to Read