U.S. History through Young People’s Eyes: a Randomized Controlled Trial of Mission US

• In Event: Evaluating Student Learning Through Innovative Digital Curricula

Abstract

Objective. Mission US is a set of web-based, interactive role-playing history games and curricular materials that address a critical problem: students lack fundamental knowledge of U.S. history. According to the most recent National Assessment of Educational Progress (NAEP, 2018), only 13% of Grade 8 students were proficient in U.S. history. We conducted a randomized controlled trial to evaluate the impact of two Mission US games and associated curricula on Grade 8-10 students' history knowledge and skills.



Methods. Participants included a national sample of 65 Grade 8-10 social studies teachers in 60 public, private, and charter schools and their 1,305 students. Teachers were randomized prior to online baseline student assessments. Teachers assigned to the intervention group attended a 1-hour professional development session and used Mission US with a focal class for approximately 12 hours during the 3-month implementation period. Teachers in the control group used an analogous website and were provided background materials to ensure that equivalent content was taught across all classrooms. Assessments were again administered to all students post-implementation.



Data Sources. To measure students’ historical content knowledge, two alternate assessments were drawn from the Grade 8 and 12 U.S. History NAEP question bank (Cronbach’s alpha=.76–.81; Hedges’ g=0.15). The Individual Interest Questionnaire (Rotgans, 2015) was used to measure students’ interest in history (Cronbach’s alpha=.86–.88; Hedges’ g=0.005) and the NAEP Interest Index to measure motivation (Cronbach’s alpha=.80–.85; Hedges’ g=0.005). A researcher-developed instrument, the Historical Empathy Measure, evaluated students' historical empathy (Cronbach’s alpha=.78; Hedges’ g=0.029). Researchers established baseline equivalence for all measures. Each outcome was analyzed using a student-nested-within-school multilevel regression model, where the outcome was regressed on treatment indicator, baseline score, and group enrollment.



Results. We found no significant differences in the average treatment effects on any student outcomes between schools that were assigned Mission US and those using only their existing curriculum. However, there were important differential effects based on school profiles: there was a positive impact of Mission US on students’ history knowledge (as measured by the NAEP) in schools that were low poverty, located in urban and suburban areas, and whose student demographics were majority White. Positive impact was also associated with teachers who had higher fidelity. Conversely, in schools that were high poverty, had a smaller teacher to student ratio, and whose schools had more Black and Hispanic students, students in the treatment group gained significantly less than the control group from pre to post.



Significance. Our inability to detect an overall impact was likely a result of a smaller and more heterogenous sample than planned, due to recruitment challenges. Our subgroup analyses do suggest that Mission US has a positive impact for some students, while others gained significantly less. As such, given that Mission US is free of charge, easy to implement, and requires very little time, more research with a bigger sample is needed to tease out for whom Mission US might be most effective.

Authors

• Hai Lun Tan, Education Development Center, Inc.

• Joy Lorenzo Kennedy, Education Development Center, Inc.