Assessing a Games-Based Approach to Systems Thinking Through the Development of 21st-Century Skills at Quest to Learn

• In Event: Seeing the Bigger Picture: Supporting Systems Thinking Through Designing Digital Systems

Abstract

The Quest to Learn School (Q2L; see Shute & Torres, 2011) is intended to enable all students, regardless of their academic or personal challenges, to contribute to the design and innovation necessary to meet the demands of a global society. As with any new idea, Q2L needs to be systematically evaluated, and this presentation focuses on students' development in relation to three competencies – systems thinking, teamwork, and time management – across an 18-month period (i.e., Sept. 2009, Mar. 2010, Nov. 2010, and May 2011). Looking across this span of time is essential to capture current levels of competencies and monitor their development over time.
Our exploratory research with Q2L began when the school opened (September 2009). We have just collected the fourth and final set of assessment data relative to three competencies and their constituent factors: (a) systems thinking (i.e., dynamic thinking, closed-loop thinking, and transfer of model), (b) teamwork (i.e., cooperating, negotiating, and influencing others), and (c) time management (i.e., making plans and meeting deadlines) over the course of the 1.5 year project. The alpha reliabilities of the assessments were good across all four administrations. For example, Time 4 reliabilities were as follows: systems thinking = .84; teamwork = .76; time management = .86. The following analyses report data from students who completed the first and fourth administrations to illustrate growth over time, focusing only on systems thinking skills.
Systems thinking (ST): Students who began Q2L in the 6th grade and who are finishing the 7th grade significantly improved on their overall ST skills from Time 1 to Time 4 (t (35) = 3.72; p < .001). When the three ST factors were analyzed separately, we found significant gains for dynamic thinking and transfer the model (as well as a non-significant gain for closed-loop thinking). The second cohort of students (i.e., who started 6th grade this academic year) also showed significant gains on their ST skills from Time 1 to Time 2 (t (21) = 4.20; p < .001).
We also examined Q2L students' state math and English (i.e., reading) test scores relative to two matched schools. From data obtained last year (June 2010), students at Q2L scored significantly lower on their standardized math test scores relative to a demographically-matched pair of middle schools (F (1, 475) = 12.44, p < .01) while Q2L students scored significantly higher than their matched counterparts on standardized reading scores (F (1, 464) = 4.91, p < .03). Thus improvement is needed in math skills for Q2L students. Note, however, that our three competencies (ST, TW, and TM) correlate with state test scores: ST significantly correlates with math (.48) and reading (.40) test scores; TW significantly correlates with math (.29); and TM significantly correlates with reading (.27). This suggests that these three competencies may play an important role in improving state test scores, thus affirming the importance of assessing and supporting such skills in students at Q2L.

Author

• Valerie J. Shute, Florida State University