Quantified Qualitative Coding of Constructed Responses: Rubric Development (Poster 1)

• In Event: Leveraging Constructed Responses to Understand Text Comprehension

Abstract

Theme 1, Poster 1: Methods to Understand, Describe, and Analyze Constructed Responses

1. Objectives. Expert judgments (coding) of open-ended responses is relatively common across fields. It is critical to develop a rubric that effectively captures the phenomena of interest as is establishing and maintaining inter-rater reliability. However, there is often little transparency in the actual coding process or discussion of the tension between validity and reliability. This poster describes how the research team co-constructed and negotiated meaning while developing and refining a rubric that captures the comprehension processes that occur in students’ open-ended CRs (e.g., think-alouds, self-explanations).

2. Perspectives. The larger project relies on cognitive theories of text comprehension which suggest that readers enact comprehension processes and strategies to establish and maintain a coherent mental representation of the text content.

3. Method. We implemented a design-based research-inspired framework to engage in collaborative and iterative revision of our rubric.

4. Data Sources. We examined archives of (1) copies of each iteration of the rubric with tracked changes and comments from version-to-version, (2) spreadsheets from each rater that included response-by-response codes as well as notes and questions for specific responses (3) combined spreadsheets in which we compared rater scores and logged discussion notes (4) rater and researcher notes from weekly meetings and email threads of discussions around specific examples.

5. Results. Analysis of and reflection on this process revealed that relatively fine-grained analysis of comprehension processes could be reliably detected by multiple teams of raters. The broader analysis of our process revealed differences across raters in the extent to which they implicitly instantiated criteria that privileged operationalization over natural variability in the focal processes or vice-versa. This work also highlights that individual CRs exist within multiple contexts (e.g. one response within a participant’s set of responses, one response within all other responses to that sentence-prompt).

6. Significance. Our first design cycle of the rubric was completed on one set of data (blind). We have continued to use this approach to code the remaining four data sets as well as adapted the rubric toward use in multiple-document contexts. The development of a flexible rubric affords analysis of a variety of types of CRs (think-alouds, self-explanations, explanations to others) and allows for easier comparison across data sets to examine both specificity and generalizability of text comprehension processes.

Authors

• Kathryn McCarthy, Georgia State University
  Presenting Author

• Natalie Newton, Arizona State University
  Non-Presenting Author