Network-Based Tools for the Visualization and Analysis of Domain Models

• In Event: From Fixed Test to Endless Data Systems: Analyzing and Communicating Ongoing Formative Assessment

Abstract

Domain modeling is an important stage in educational assessment design (Mislevy, & Riconscente, 2006). At this stage, information collected from domain analyses, which are typically conducted by subject matter experts (SMEs), is used to structure relationships among the important knowledge and skills that students are expected to learn in the subject domain. A visual display of the domain model, such as graphs or maps, has the great advantage of forcing people into “recognizing patterns and interpreting spatial relationships” (Mislevy et al., 2010, p. 7). As learning support systems become more complex and the assumptions upon which they are based become more hidden, there is an increased need for tools to explore, analyze and communicate these foundational structures. In these more complex arrangements communication of networks of dependencies requires the use of network-based graphics as well as corresponding algebraic expressions of network structure. This paper illustrates the use of these techniques and their algebraic analogs and discusses how key aspects of network structure (and logical dependence) can be identified.

In many contexts the direct graph of dependency relationships is a key structure for both human understanding as well as computational data structures. These dependencies which can be expressed in matrix as well as geometric form, spell out the idea that a knowledge element must be acquired before learning another. Following the logic of exploratory data analysis and data visualization (Behrens et al., 2012), insights into the structure of a complex set of dependencies can be communicated both to analysts and end-users using network visualization tools. Consider, for example, the dependency graph illustrated on the right which reflects a series of dependency relationships between objectives in an automated learning support system. As reflected in the graphic, it is easy to discern a visible “choke point” in which one skill becomes a gating skill for future advancement. Such a relationship is easily observed in these graphics though may be, as in this case, easily obscured in a long list of object-to-objective dependency tables.

A second goal of the paper is to illustrate the application of network metrics in the quantification of network structure. In the network literature there are number of common node-level metrics including in-degree and out-degree as well as betweenness and centrality. While originally devised to represent social importance, these metrics have corresponding dependency interpretations as well. Nodes with high in-degree have a higher number of immediately dependent objectives, while those with high out-degree have influence over a large number of sub-ordinate elements. Betweenness represents objectives at boundary locations (as illustrated above) and centrality represents the opposite. In this paper we illustrate how these metrics can be used to diagnose and explore domain model structure for both design and communicative purposes.

A number of software packages can be used for visualizing and analyzing network relationships, such as NodeXL (Hansen, Shneiderman, & Smith, 2010) and R.

Author

• Hua Wei, Pearson