Knowledge Specification: A Priori Requirement for Effective Embedded Assessment

• In Event: New Measurement Paradigms: Psychometric Methods for Technology-Based Assessments

Abstract

1. Objectives/Purposes. Computer-based environments afford automated collection of vast amounts of data quantifying student interactions during authentic performance tasks. These tasks can be engineered as embedded assessments that measure learning and provide formative and summative feedback. However, data are useless unless 1) data contain information, 2) data translate into information, and 3) data reports effectively communicate that information. This paper argues embedded assessment requires a priori alignment of assessment and to-be-learned content through formal knowledge specification. When authentic assessment is embedded within computer-based instruction, knowledge specification must align both instruction and the assessment with to-be-learned content. Alignment enables a priori specification of an expert model for comparison to learner activity. Computer-based, interactive assessment designs that begin with knowledge specification and apply it to measure learning are rule-based approaches.
2. Perspective(s)/theoretical framework: Formal knowledge specification is the preliminary and fundamental design step for sciences of human learning in cognitive psychology (Anderson & Lebiere, 1998; Anderson & Schunn, 2000; Gentner, 1980, 1983) and instructional design (Gagné, Briggs, & Wager, 1992; Smith & Ragan, 2005). In learning science, domain specification may model how learners “represent knowledge and develop competence in a targeted domain” (Pellegrino, Chudowsky, & Glaser, 2001, p. 44). Cognitive tutors are designed from formal task analyses (Anderson & Schunn, 2000; Koedinger & Corbett, 2006). Instructional games can be designed upon formal knowledge specification mapped from target domain to game world (Reese, 2009). Simulations can be designed to model how students represent and develop proficiency (Quellmalz, Silberglitt, & Timms, 2011). In each case, viable assessment requires that assessment observations and the interpretative process align with the knowledge specification (Pellegrino, et al., 2001).
3. Methods/techniques/modes of inquiry. A summary representing research programs using cognitive tutors, simulations, and videogames illustrates application of rule-based assessment.
4. Data sources/evidence/objects/materials. Knowledge specification approaches, computer environments, and assessment applications are presented for cognitive tutors (Anderson & Schunn, 2000; Gobert & Koedinger, 2011; Koedinger & Corbett, 2006), simulations (Quellmalz, et al., 2011), and videogames (Reese et al., in press; Reese & Tabachnick, 2010).
5. Results and/or substantiated conclusions or warrants for arguments/point of view. Copious data cannot advance teaching and learning unless data carry information. And data containing information lacks utility if analysis systems cannot mine that information. A priori domain specification aids and informs assessment design and analysis. Diverse but complimentary approaches to knowledge specification afford designers, researchers, and developers ways to construct assessments that measure targeted learning within cognitive tutors, simulations, and videogames. Conceptual understanding can be conceptualized and operationalized as ability to flexibly apply and integrate relevant systems of declarative and procedural knowledge (Anderson & Schunn, 2000).
6. Scientific/scholarly significance of the study or work. Today, federal, industry, and private investments seek to advance teaching, learning, and life-long achievement through cyberlearning technologies with embedded assessment. Like cognitive tutors (Anderson & Schunn, 2000), effective videogames and simulations must be constructed after and over an accurate knowledge specification. Knowledge specification is time intensive, domain specific, and required. Advancement in cyberlearning requires vetted, appropriate, domain-specific knowledge specification.

Author

• Debbie Denise Denise Reese, Wheeling Jesuit University - Center for Educational Technologies