Development of a Next-Generation Concept Inventory With Artificial Intelligence–Based Evaluation for College Environmental Programs (Poster 7)

• In Event: Artificial Intelligence–Augmented Assessment Tools to Support Instruction in STEM

Abstract

Objectives
We are developing an assessment for undergraduates’ foundational knowledge in interdisciplinary environmental programs (EPs) and their ability to use complex systems-level thinking, starting with Food-Energy-Water (FEW) systems. The Next Generation Concept Inventory (NGCI) requires students to construct their own answers, which better reveal student thinking about complex topics and phenomena. Once developed, machine learning (ML) models will evaluate students’ text responses to the NGCI.
Perspectives
EPs are increasingly popular in U.S. universities. However, EPs lack consensus about core concepts and learning outcomes. Currently, there is no concept inventory for EP content. We follow an evidentiary validity framework for designing concept inventories. Further, we apply a design framework for developing and evaluating constructed responses (CR) and automated text scoring for short responses in science.
Methods and Data Sources
We performed a content analysis on EP course materials to identify shared objectives across courses and programs. We conducted ~100 semi-structured interviews with undergraduates which we analyzed thematically. Course materials were collected from a representative sample of EPs across the US, including baccalaureate, master’s, and doctoral universities, which we also used to solicit students for interviews. We surveyed eight EP instructors about the assessment items and collected responses from over 700 EP undergraduates across seven institutions. We used open-coding of student responses to develop rubrics.
Results
From our course materials and interview analyses, we identified four key activities for assessment prompts: Explaining connections between FEW, Identifying sources of FEW, Cause & Effect of FEW usage and Tradeoffs. These are aligned with Next Generation Science Standards cross-cutting concepts. We developed three sets of CR items to these four activities using different phenomena and disciplinary ideas as contexts. We developed a series of analytic coding rubrics to identify students’ scientific and informal ideas in CRs and how students connect these ideas. To detect complex systems-level thinking, analytic rubrics are combined into a holistic score representing the complexity of systems thinking. Such an approach has been used to improve ML accuracy and may be applicable to more complex responses, like those to NGCI. The series of rubrics identify predictions of possible outcomes and differentiate between the listing of FEW components and explaining connections. Human raters have demonstrated good levels of agreement (0.72-0.85) using these rubrics.
Scholarly Significance
Our findings from qualitative analysis of CRs suggest undergraduates understand and discuss environmental injustices committed against historically marginalized peoples to acquire FEW resources. We also found students often conflate forms of energy needed to produce food with nutritional forms of energy. So far, we have identified key objectives across college EPs, common student preconceptions about environmental phenomena, and a set of assessment targets pertinent for a range of EP courses. Further findings will address a challenge to AI-based scoring, since systems-level thinking relies on connections between components and defining system boundaries.

Authors

• Kevin Haudek, Michigan State University
  Presenting Author

• Chelsie Romulo, University of Northern Colorado
  Presenting Author

• Steven Anderson, University of Northern Colorado
  Presenting Author

• Lydia Horne, Rowan University
  Presenting Author

• Ennea Fairchild-Grant, Pacific Northwest National Laboratories
  Presenting Author

• Shirley Vincent, Vincent Evaluation Consulting
  Presenting Author

• Amanda Manzanares, University of Northern Colorado
  Presenting Author

• Emily Royse, University of Northern Colorado
  Presenting Author

• Sol Adams
  Non-Presenting Author

• Heqiao Wang, Michigan State University
  Non-Presenting Author

• Caterina B. Azzarello, University of Northern Colorado
  Non-Presenting Author