Knowledge Integration Guidance and Student Revision of Essays in Inquiry Science

• In Event: Knowledge Integration: Trajectories, Opportunities, and Future Directions

Abstract

Objectives & Theoretical Framework. Prior research demonstrated that automated, adaptive knowledge integration (KI) guidance was more effective than other forms of guidance (Gerard, Ryoo, Liu, McElhaney, Rafferty & Linn, 2015). Even with this guidance, especially students with low prior knowledge, struggled to make effective revisions. Effective revision involves integrating ideas by finding and incorporating new evidence and resolving inconsistencies (Fitzgerald, 1987). To model the process of KI revision, we developed an annotator tool. We analyze contributions of the annotator and of adaptive guidance assigned based on natural language processing.
Methods & Data sources. We designed guidance for two essays in WISE Plate Tectonics (http://wise.berkeley.edu/previewproject.html?projectId=18449). We integrated c-raterML™, a natural language processing tool developed by the Educational Testing Service, to score the essays. c-raterML™ scoring models demonstrated satisfactory human-machine agreement using KI rubrics (k= .75, k=.81). To assess the impact of the annotator, students (n=510, 6th - 7th graders) were randomly assigned to two conditions:
• Annotator condition (Fig1): students write their essay, use the annotator to critique a fictional peer’s essay, receive one round of adaptive KI guidance on their short essay, and revise.
• Automated condition: students receive two rounds of adaptive, KI guidance on their short essay and two opportunities to revise.
Results.
Embedded Essays. Students significantly improved their short essays in both conditions [Annotator t(80)=5.22, p<.001; Automated t(92)=3.93, p<.001]. The Annotator was significantly more effective than Automated in helping students use guidance to improve [F(1, 172)=6.94, p<.01).
Revision Strategies. We coded revision strategies using a rubric where making new connections was the most complex level of revision and adding disconnected ideas was the lowest. Students with low prior knowledge were significantly more likely to integrate new ideas when revising in the Annotator condition as compared to the Automated condition [X2(1)=5.57, p<0.05].
Pre/post essay. Students in both conditions made significant gains pre to posttest [t(126)=10.43, p<.001), with no effect of condition. To measure students’ ability to use guidance, one pre/post item prompted students to write a short essay, receive one round of automated guidance, and revise. We scored students’ initial essay and revision, and created an improvement score. Students used guidance to significantly improve their short essay, with no difference between conditions [Fig2]. Low prior knowledge students in the Annotator condition made significantly greater improvement than their counterparts in the Automated condition.
Significance. The KI framework gave us insights into how students struggled to connect ideas when revising, and informed our design of the annotator to model how students could make integrated, as opposed to additive revisions. We find that modeling the revision process with the annotator enabled students to make greater improvement when using automated guidance on the embedded assessments, than providing additional rounds of guidance. Our KI revision analysis suggests this is because the annotator helped students, particularly those with low prior knowledge, use the guidance to make integrated revisions. We plan future studies to test the annotator with additional units. We are exploring ways to personalize the annotator to model the KI process of distinguishing among ideas.

Authors

• Libby F. Gerard, University of California - Berkeley

• Ou Lydia Liu, ETS