How Do We Measure Technological Pedagogical Content Knowledge? Let Me Count the Ways

• In Event: Education Technology, Teacher Knowledge, and Classroom Impact: Frameworks and Approaches to Research

Abstract

Objectives

We present a review of various techniques of measuring TPACK, specifically addressing the following two questions: (1) What kinds of measures are used in the TPACK literature? (2) Are those measures reliable and valid?

Theoretical Framework(s)

The TPACK framework functions as a “conceptual lens” through which one views education technology by drawing attention to specific aspects of the phenomena, highlighting relevant issues, and ignoring irrelevant ones. In this view, the framework functions as a classification scheme providing insight into the nature and relationships of the objects (and ideas and actions) under scrutiny.

Method(s)

This study uses a systematic review of literature. We conducted two levels of analysis: study- and measurement-levels. The study level analysis examined the characteristics of each study (e.g., number of TPACK measures). The measurement level focused on individual TPACK measures rather than on the studies (e.g., target population of the measure; evidence of reliability and validity).

Data Sources

Coding was done by one of the authors who had experience in meta-analysis. Ambiguous cases, were resolved by consensus of all authors. A total of 66 studies were identified for inclusion. A random sample of 19 studies were double coded (84 to 100% depending on the category being coded.

Results

Study Level

A majority of the studies we investigated were published in journals and conference proceedings (a total of 62 out of 66 studies). The other four were unpublished dissertations and conference presentations. Forty one studies used more than two different types of TPACK instruments.

Measurement Level

The sample contained 141 measures of TPACK. We found five types of measures. Self-report measures (N=31) and performance assessments (N=31) were used most frequently; semi-structured interviews (N=30) and observations (N=29) were also used. Open-ended questionnaires (N=20) were the least frequently used TPACK measures.

Self report. A total of 12 studies (39% of 31) presented evidence of reliability. One study reported Raykov’s reliability rho; the other eleven studies reported Cronbach’s Alpha. Eleven studies (35%) presented evidence that they addressed the issue of validity, most often conducting either an exploratory or confirmatory factor analysis.

Performance assessment. Only six studies (19% of 31) presented evidence of reliability (e.g., inter-rater or test-retest reliability). Only one study (3%) explicitly addressed the issue of validity.

Open-ended questionnaire. Only three studies (15% of 20) presented evidence of reliability (e.g., inter-rater reliability). Only one study (5%) explicitly addressed the issue of validity.

Interview. None of the thirty studies reported concrete evidence that established the reliability of their interview measures. Five studies reported that the interviews were coded by multiple coders but did not present any reliability index. None addressed the issue of validity explicitly.

Observation. Only three studies reported a reliability index (inter-rater reliability). For validity, none presented any concrete validity evidence other than reporting that they were based on the TPACK framework.

Significance

This study analyzes the contribution of a variety of research methodologies for measuring TPACK and allows future researchers to make better-informed decisions on what types of assessment suit their research questions.

Authors

• Matthew J. Koehler, Michigan State University

• Punya Mishra, Michigan State University

• Tae Seob Shin, Hanyang University