Modeling Impact: The Program Context Evaluators and Analysts Miss without Latent Variable Modeling

• In Event: Survey Data Analysis, Issues in Measurement Error, and Latent Variable Modeling

Abstract

Introduction: Program evaluators and policy analysts are often interested in collecting systematic feedback about personal experiences to describe the program context (Alkin, 2004; Whiteman, 1997). For example, multisite programs may collect survey data from participants to self-evaluate implementation and improve program services. However, issues of measurement error arise when using survey measures across multiple participants, program sites, or states and can undermine valid inference about program context. The need to statistically describe program context while considering measurement error inspires this project providing applied examples of latent class analysis (LCA) and latent transition analysis (LTA) models to provide meaningful inference about program context.
Method: This paper uses a national example from a local example from a multisite evaluation of a college access program and the High School Longitudinal Study of 2009 (HSLS:2009) from the National Center for Educational Statistics. For both examples, the presentation will briefly discuss class enumeration, measurement components, and how to incorporate observed demographic variables (Nylund et al., 2007). LCA is a useful tool because it describes distinct patterns in population behavior that effectively describe program context, and how longitudinal changes in population behavior influences outcomes of interest to policy makers.
Results: One example focuses on the application of LCA to study the program context for a regional multisite college access program serving students at nine Title I schools. First, each program site distributed surveys asking participants about their needs related to college applications, financial aid, and enrolling in college. We then used the LCA model to identify four measurement profiles (Cognizant Explorers, Cost-Conscious Negotiators, Deliberate Navigators, and Unburdened Prospectors) through the class enumeration process. The four classes described distinct patterns of needs and significantly predicted whether student submitted their free application for federal student aid.The next example uses data from the HSLS:2009 study measuring math and science self-efficacy using questions such as “I believe I can do well at math tests,” in 9th and 12th grades. Using LTA to study qualitative changes in student self-efficacy, the results described subpopulations of students, which can be used to develop targeted intervention plans. For example, the results showed that 55% of the population held high self-efficacy towards math and science in 9th grade, while only 52% of the population reported the same pattern. Moreover, 69% of these students were likely to maintain strong self-efficacy, 5% of students transitioned to having low self-efficacy in both math and science. The results suggested that students who maintained math and science self-efficacy also earned higher math scores compared to students who lost overall self-efficacy.
Conclusion: These empirical examples highlight how finite mixture modeling techniques describe population heterogeneity and how policymakers can use results to develop targeted interventions. While the primary utility of mixture modeling lies in its ability as a measurement model, this paper contributes to policy analysis methods and tools by illustrating how evaluators and policy analysts can create meaningful insight about program context. Future research should consider how mixture modeling applies to the evaluation of targeted interventions for identified subpopulations.

Author

• Travis Rodolfo Candieas, University of California, Santa Barbara
  Presenting Author